Configuring flume to not generate .tmp files when sinking data to hdfs - hadoop

Question

I am using flume to stream data into hdfs from server logs. But while data is getting streamed into the hdfs it is first creating .tmp file. Is there a way in the configuration where .tmp files can be hidden or there name can be changed by appending a . in front. My collection agent file look like-
## TARGET AGENT ##
## configuration file location: /etc/flume/conf
## START Agent: flume-ng agent -c conf -f /etc/flume/conf/flume-trg-agent.conf -n collector
#http://flume.apache.org/FlumeUserGuide.html#avro-source
collector.sources = AvroIn
collector.sources.AvroIn.type = avro
collector.sources.AvroIn.bind = 0.0.0.0
collector.sources.AvroIn.port = 4545
collector.sources.AvroIn.channels = mc1 mc2
## Channels ##
## Source writes to 2 channels, one for each sink
collector.channels = mc1 mc2
#http://flume.apache.org/FlumeUserGuide.html#memory-channel
collector.channels.mc1.type = memory
collector.channels.mc1.capacity = 100
collector.channels.mc2.type = memory
collector.channels.mc2.capacity = 100
## Sinks ##
collector.sinks = LocalOut HadoopOut
## Write copy to Local Filesystem
#http://flume.apache.org/FlumeUserGuide.html#file-roll-sink
#collector.sinks.LocalOut.type = file_roll
#collector.sinks.LocalOut.sink.directory = /var/log/flume
#collector.sinks.LocalOut.sink.rollInterval = 0
#collector.sinks.LocalOut.channel = mc1
## Write to HDFS
#http://flume.apache.org/FlumeUserGuide.html#hdfs-sink
collector.sinks.HadoopOut.type = hdfs
collector.sinks.HadoopOut.channel = mc2
collector.sinks.HadoopOut.hdfs.path = /user/root/flume-channel/%{log_type}
collector.sinks.k1.hdfs.filePrefix = events-
collector.sinks.HadoopOut.hdfs.fileType = DataStream
collector.sinks.HadoopOut.hdfs.writeFormat = Text
collector.sinks.HadoopOut.hdfs.rollSize = 1000000
Any help will be appreciated.

Answer 1

All files in Flume which are opened for writing can have .tmp extension by default. You can alter this with another extension. But we cannot avoid this extension. Moreover it is required to differentiate with closed files.
So it is better to some suffix like "." for open files. Flume HDFS Sink offers several parameters:
hdfs.inUsePrefix – Prefix that is used for temporal files that flume actively writes into
hdfs.inUseSuffix .tmp Suffix that is used for temporal files that flume actively writes into
hdfs.inUsePrefix = .
collector.sinks.HadoopOut.hdfs.inUsePrefix = .
hdfs.inUseSuffix = if it is blank it uses .tmp otherwise it uses specified suffix.

Answer 2

set hdfs.idleTimeout=x where x is a positive number