DataFlow pipeline stuck on initializing tempLocation? - java-8

Question

I'm new to DataFlow and try to set up a streaming pipeline reading CSV files from Google Cloud Storage into BigQuery. The pipeline is successfully created and CSV files are being read and parsed. The whole pipeline isn't properly initialized though. So no data is loaded into BigQuery.
I use Java 8 and Apache Beam 2.5.0.
When I traverse the DataFlow execution graph I see that the block Write to bigquery/BatchLoads/TempFilePrefixView/Combine.GloballyAsSingletonView/View.CreatePCollectionView/ParDo(StreamingPCollectionViewWriter) gets input but never spits out any output. Because of that the following step Write to bigquery/BatchLoads/TempFilePrefixView/Combine.GloballyAsSingletonView/View.CreatePCollectionView/CreateDataflowView is never executed.
I'm "inspired" by code from for example
https://github.com/asaharland/beam-pipeline-examples/blob/master/src/main/java/com/harland/example/streaming/StreamingFilePipeline.java
public class MyStreamPipeline {
private static final Logger LOG = LoggerFactory.getLogger(MyStreamPipeline.class);
private static final int WINDOW_SIZE_SECONDS = 120;
public interface MyOptions extends PipelineOptions, GcpOptions {
#Description("BigQuery Table Spec project_id:dataset_id.table_id")
ValueProvider<String> getBigQueryTableSpec();
void setBigQueryTableSpec(ValueProvider<String> value);
#Description("Google Cloud Storage Bucket Name")
ValueProvider<String> getBucketUrl();
void setBucketUrl(ValueProvider<String> value);
}
public static void main(String[] args) throws IOException {
MyOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(MyOptions.class);
Pipeline pipeline = Pipeline.create(options);
List<TableFieldSchema> tableFields = new ArrayList<>();
tableFields.add(new TableFieldSchema().setName("FIELD_NAME").setType("INTEGER"));
// ... more fields here ...
TableSchema schema = new TableSchema().setFields(tableFields);
pipeline
.apply("Read CSV as string from Google Cloud Storage",
TextIO
.read()
.from(options.getBucketUrl() + "/**")
.watchForNewFiles(
// Check for new files every 1 minute(s)
Duration.standardMinutes(1),
// Never stop checking for new files
Watch.Growth.never())
)
.apply(String.format("Window Into %d Second Windows", WINDOW_SIZE_SECONDS),
Window.into(FixedWindows.of(Duration.standardSeconds(WINDOW_SIZE_SECONDS))))
.apply("Convert CSV string to Record",
ParDo.of(new CsvToRecordFn()))
.apply("Record to TableRow",
ParDo.of(new DoFn<Record, TableRow>() {
#ProcessElement
public void processElement(ProcessContext c) {
Record record = c.element();
TableRow tr = record.getTableRow();
c.output(tr);
return;}}))
.apply("Write to bigquery",
BigQueryIO
.writeTableRows()
.to(options.getBigQueryTableSpec())
.withSchema(schema)
.withTimePartitioning(new TimePartitioning().setField("PARTITION_FIELD_NAME").setType("DAY"))
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND));
pipeline.run();
}
}
I execute the pipeline using a Maven command like this one:
mvn -f pom_2-5-0.xml clean compile exec:java \
-Dexec.mainClass=com.organization.processor.gcs.myproject.MyStreamPipeline \
-Dexec.args=" \
--project=$PROJECT_ID \
--stagingLocation=gs://$PROJECT_ID-processor/$VERSION/staging \
--tempLocation=gs://$PROJECT_ID-processor/$VERSION/temp/ \
--gcpTempLocation=gs://$PROJECT_ID-processor/$VERSION/gcptemp/ \
--runner=DataflowRunner \
--zone=$DF_ZONE \
--region=$DF_REGION \
--numWorkers=$DF_NUM_WORKERS \
--maxNumWorkers=$DF_MAX_NUM_WORKERS \
--diskSizeGb=$DF_DISK_SIZE_GB \
--workerMachineType=$DF_WORKER_MACHINE_TYPE \
--bucketUrl=$GCS_BUCKET_URL \
--bigQueryTableSpec=$PROJECT_ID:$BQ_TABLE_SPEC \
--streaming"
I really don't understand why the pipeline isn't initialized properly and why no data is loaded into BigQuery.
Any help appreciated!