Passing parameters between Flink Jobs - java-8

Question

I have a job that reads data from Cassandra and store the data as List ( the method fillOnceGeoFencesFromDB() attached below ) and than I create StreamExecutionEnvironment and consume data from the Kafka queue.
During transformation of DataStream I try to reference recently filled static ArrayList , but it's empty.
What is a best practice to pass previously filled List into the next Job ?
Any Idea will be appreciated.
private static ArrayList<GeoFences> allGeoFences = new ArrayList<>();
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
env.enableCheckpointing(5000); // checkpoint every 5000 msecs
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
Properties kafkaProps = new Properties();
kafkaProps.setProperty("zookeeper.connect", LOCAL_ZOOKEEPER_HOST);
kafkaProps.setProperty("bootstrap.servers", LOCAL_KAFKA_BROKER);
kafkaProps.setProperty("group.id", KAFKA_GROUP);
kafkaProps.setProperty("auto.offset.reset", "earliest");
fillOnceGeoFencesFromDB(); // populate data in ArrayList<GeoFences> allGeoFences
DataStream <Tuple6<UUID, String, String, String, String, Timestamp>> stream_parsed_with_timestamps = env
.addSource(new FlinkKafkaConsumer010<>(KAFKA_SUBSCRIBE_TOPIC, new SimpleStringSchema(), kafkaProps))
.rebalance().map(new MapFunction<String, Tuple4<UUID, String, String, Timestamp>>() {
private static final long serialVersionUID = 1L;
#Override
public Tuple4<UUID, String, String, Timestamp> map(String value) throws Exception {
return mapToTuple4(value);
}})
.
.
.
.
.
.

Answer 1

Please keep in mind that whatever happens in the map function will take place on the task managers while all your code in the main is only used to define your job.
Pass your parameter explicit to the MapFunction (That will make the code easier to read).
private static class GeoFenceMapper implements MapFunction<String, Tuple4<UUID, String, String, Timestamp>> {
private ArrayList<GeoFences> allGeoFences;
public GeoFenceMapper(ArrayList<GeoFences> allGeoFences) {
this.allGeoFences = allGeoFences;
}
#Override
public Tuple4<UUID, String, String, Timestamp> map(String value) throws Exception {
return mapToTuple4(value);
}})
}
and than use this new mapper:
DataStream <Tuple6<UUID, String, String, String, String, Timestamp>> stream_parsed_with_timestamps = env
.addSource(new FlinkKafkaConsumer010<>(KAFKA_SUBSCRIBE_TOPIC, new SimpleStringSchema(), kafkaProps))
.rebalance().map(new GeoFenceMapper(fillOnceGeoFencesFromDB()))
Hope this helps!