Why task processing serial in flink - parallel-processing

Question

//抓包
DataStream<byte[]> logSource = env.addSource(new PacpParalleSourceFunction()).setParallelism(CAPTURE_PACKAGE_PARALLELISM);
//解析
XmlLogParse xmlLogParse = new XmlLogParse();
DataStream<JSONObject> objectDataStream = logSource.map(new Log2ObjFunction(xmlLogParse)).setParallelism(PARSE_LOG_PARALLELISM);
//写csv The parallelism of non parallel operator must be 1
LogStorage logStorage = new WriteFileStorage();
objectDataStream.countWindowAll(ConstantField.BATCH_DEAL_COUNT).apply(
new AllWindowFunction<JSONObject, Object, GlobalWindow>() {
#Override
public void apply(GlobalWindow globalWindow, Iterable<JSONObject> iterable, Collector<Object> collector) throws Exception {
ArrayList<JSONObject> list = Lists.newArrayList(iterable);
if (list.size() > 0) {
long start = System.currentTimeMillis();
logStorage.doStorage(list);
logger.info("写入文件 " + list.size() + " 耗时" + (System.currentTimeMillis() - start) + "ms " + Thread.currentThread().getName());
}
}
}
).name("out put to csv").setParallelism(1);
//写kafka
Properties props = new Properties();
props.put("bootstrap.servers", KAFKA_HOST);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("topic", KAFKA_DST_TOPIC_NAME);
DataStreamSink<JSONObject> obj2kafka = objectDataStream.addSink(new KafkaSinkFunction(props)).setParallelism(WRITE_KAFKA_PARALLELISM);
obj2kafka.name("output2kafka");
// execute application
env.execute("elksoc-log4p");
How can I make these two tasks parallel?
How can I make these two tasks parallel?
How can I make these two tasks parallel?
How can I make these two tasks parallel?
How can I make these two tasks parallel?
How can I make these two tasks parallel?
How can I make these two tasks parallel?

Answer 1

Perhaps you should use Side Outputs, for more information see https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/side_output/