In my Storm based application I need to query oracle table periodically So I thought to use Tick tuple of storm. But it's not giving correct result and tick tuple is not producing.
My storm version is 1.0.1.2.5.3.0-37
I tried as below,
Added getComponentConfiguration method in bolt as http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/ link but tick tuple is not generating.
So I changed the code and used Config from topology for generating tick tuple.I refer https://www.safaribooksonline.com/blog/2014/01/06/multi-threading-storm/ link but here I got tick tuple only once.
Below is my code of tick tuple with bolt,
public class TickTupleBolt implements IRichBolt{
private OutputCollector collector = null;
private static final long serialVersionUID = 1L;
private static final Logger LOG = LoggerFactory.getLogger(TickTupleBolt.class);
public void prepare(Map stormConf, TopologyContext context,OutputCollector collector) {
this.collector = collector;
}
public void execute(Tuple tuple) {
LOG.info("Start of TickTupleBolt.execute");
try {
if (isTickTuple(tuple)) {
//if(tuple.getSourceStreamId().equals("__tick")){
LOG.info("**got tick tuple");
}else{
LOG.info("not got tick tuple");
}
} catch (Exception e) {
LOG.error("Bolt execute error: {}", e);
collector.reportError(e);
}
LOG.info("End of TickTupleBolt.execute");
}
public void cleanup() {
// TODO Auto-generated method stub
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
// TODO Auto-generated method stub
}
public Map<String, Object> getComponentConfiguration() {
// configure how often a tick tuple will be sent to our bolt
Map<String, Object> conf = new HashMap<String, Object>();
conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 1);
return conf;
}
protected boolean isTickTuple(Tuple tuple) {
return tuple.getSourceComponent().equals(Constants.SYSTEM_COMPONENT_ID)
&& tuple.getSourceStreamId().equals(Constants.SYSTEM_TICK_STREAM_ID);
}
}
I got one link Tick Tuple not functioning in apache storm 0.9.4 but there is no answer.
So can any body please let me know,
How to implement tick tuple in Storm
Is there any other way (apart from tick tuple) to do periodic job in storm
UPDATE - Topology Code
My Topology builder,
public class Topology {
private static final Logger LOG = LoggerFactory.getLogger(Topology.class);
public static StormTopology buildTopology() {
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("tickspout", new TickTupleSpout());
builder.setBolt("tickbolt", new TickTupleBolt()).shuffleGrouping("tickspout");
return builder.createTopology();
}
public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException, AuthorizationException {
Config conf = new Config();
//conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 5);//tried it also
conf.setDebug(true);
//conf.setNumWorkers(2);
StormSubmitter.submitTopology(args[0], conf, buildTopology());
}
}
UPDATE - Spout Code
public class TickTupleSpout extends BaseRichSpout{
private static final Logger LOG = LoggerFactory.getLogger(TickTupleSpout.class);
private static final long serialVersionUID = 1L;
private SpoutOutputCollector collector;
public TickTupleSpout() {
}
public void open(Map conf, TopologyContext context,
SpoutOutputCollector collector) {
// TODO Auto-generated method stub
LOG.info("Start of TickTupleSpout.Open");
this.collector = collector;
LOG.info("End of TickTupleSpout.Open");
}
public void nextTuple() {
LOG.info("Start of TickTupleSpout.nextTuple");
this.collector.emit(new Values("0");//just send dummy value
LOG.info("End of TickTupleSpout.nextTuple");
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("breachdata"));
}
//public Map<String, Object> getComponentConfiguration() {
//Config conf = new Config();
//int tickFrequencyInSeconds = 5;
//conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, tickFrequencyInSeconds);
//return conf;
//}
}
Thanks.
Related
i started a project where i implement appache kafka.
I already have a working producer that writes data into the queue. So far so good. Now i wanted to program an consumer that reads out all the data in the queue.
That is the corresponding code:
try {
consumer.subscribe(Collections.singletonList("names"));
if (startingPoint != null){
consumer.
consumer.poll(Duration.ofMillis(0));
consumer.seekToBeginning(consumer.assignment());
}
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> record : records) {
keyValuePairs.add(new String[]{record.key(),record.value()});
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}
} catch (Exception e) {
e.printStackTrace();
} finally {
consumer.close();
}
That code doesnt work right now like it is supposed to do. Only new records are consumed.
I was able to find out that
seekToBeginning() isn´t working because no partition is assigned to the consumer in that moment.
If i increase the duration of the poll it works. If i just pause the thread on the other hand it doesn´t.
Could someone please try to explain me why that is the case. I tried to find out by myself and already read something about a Kafka heartbeat. But i still haven´t fully understood what happens exactly.
The assignment takes time; polling for 0 will generally mean the poll will exit before it occurs.
You should add a ConsumerRebalanceListener callback to the subscribe() method and perform the seek in onPartitionsAssigned().
EDIT
#SpringBootApplication
public class So69121558Application {
public static void main(String[] args) {
SpringApplication.run(So69121558Application.class, args);
}
#Bean
public ApplicationRunner runner(ConsumerFactory<String, String> cf, KafkaTemplate<String, String> template) {
return args -> {
template.send("so69121558", "test");
Consumer<String, String> consumer = cf.createConsumer("group", "");
consumer.subscribe(Collections.singletonList("so69121558"), new ConsumerRebalanceListener() {
#Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
}
#Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
consumer.seekToBeginning(partitions);
}
});
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
records.forEach(System.out::println);
Thread.sleep(5000);
consumer.close();
};
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so69121558").partitions(1).replicas(1).build();
}
}
Here are a couple of examples of doing it the Spring way - just add one of these (or both) to the above class.
#KafkaListener(id = "so69121558", topics = "so69121558")
void listen(ConsumerRecord<?, ?> rec) {
System.out.println(rec);
}
#KafkaListener(id = "so69121558-1", topics = "so69121558")
void pojoListen(String in) {
System.out.println(in);
}
The seeks are done a bit differently too; here's the complete example:
#SpringBootApplication
public class So69121558Application extends AbstractConsumerSeekAware {
public static void main(String[] args) {
SpringApplication.run(So69121558Application.class, args);
}
#KafkaListener(id = "so69121558", topics = "so69121558")
void listen(ConsumerRecord<?, ?> rec) {
System.out.println(rec);
}
#KafkaListener(id = "so69121558-1", topics = "so69121558")
void pojoListen(String in) {
System.out.println(in);
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so69121558").partitions(1).replicas(1).build();
}
#Override
public void onPartitionsAssigned(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) {
callback.seekToBeginning(assignments.keySet());
}
}
Using single SFTP channel I need to process two remote directories lowpriority and highprioiry but lowpriority files pick after the highpriority .
please let know how handle multiple directories in SFTP inbound adapter with single channel ?
We can do using https://docs.spring.io/spring-integration/reference/html/sftp.html#sftp-rotating-server-advice Rotation Service advice in Spring 5.1.2 Release but what about 4.3.12 Release.?
It is not available in 4.3.x; the feature was added in 5.0.7.
It needs infrastructure changes so it will be hard to replicate with custom code in 4.3.x.
You could use two adapters and stop/start them as necessary.
EDIT
Here is one solution; the advice on the primary flow starts the secondary flow when no new files are found. The secondary flow runs just once, then restarts the primary flow; and the cycle continues...
#SpringBootApplication
public class So54329898Application {
public static void main(String[] args) {
SpringApplication.run(So54329898Application.class, args);
}
#Bean
public IntegrationFlow primary(SessionFactory<LsEntry> sessionFactory) {
return IntegrationFlows.from(Sftp.inboundAdapter(sessionFactory)
.localDirectory(new File("/tmp/foo"))
.remoteDirectory("foo/foo"), e -> e
.poller(Pollers.fixedDelay(5_000, 5_000)
.advice(startSecondaryAdvice())))
.channel("channel")
.get();
}
#Bean
public IntegrationFlow secondary(SessionFactory<LsEntry> sessionFactory) {
return IntegrationFlows.from(Sftp.inboundAdapter(sessionFactory)
.localDirectory(new File("/tmp/foo"))
.remoteDirectory("foo/bar"), e -> e
.poller(Pollers.trigger(oneShotTrigger(sessionFactory)))
.autoStartup(false))
.channel("channel")
.get();
}
#Bean
public IntegrationFlow main() {
return IntegrationFlows.from("channel")
.handle(System.out::println)
.get();
}
#Bean
public Advice startSecondaryAdvice() {
return new StartSecondaryWhenPrimaryIdle();
}
#Bean
public FireOnceTrigger oneShotTrigger(SessionFactory<LsEntry> sessionFactory) {
return new FireOnceTrigger((Lifecycle) primary(sessionFactory));
}
public static class StartSecondaryWhenPrimaryIdle extends AbstractMessageSourceAdvice
implements ApplicationContextAware {
private ApplicationContext applicationContext;
#Override
public boolean beforeReceive(MessageSource<?> source) {
return true;
}
#Override
public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
this.applicationContext = applicationContext;
}
#Override
public Message<?> afterReceive(Message<?> result, MessageSource<?> source) {
if (result == null) {
System.out.println("No more files on primary; starting single shot on secondary");
this.applicationContext.getBean("primary", Lifecycle.class).stop();
this.applicationContext.getBean("secondary", Lifecycle.class).stop();
this.applicationContext.getBean(FireOnceTrigger.class).reset();
this.applicationContext.getBean("secondary", Lifecycle.class).start();
}
return result;
}
}
public static class FireOnceTrigger implements Trigger {
private final Lifecycle primary;
private volatile boolean done;
public FireOnceTrigger(Lifecycle primary) {
this.primary = primary;
}
#Override
public Date nextExecutionTime(TriggerContext triggerContext) {
if (done) {
System.out.println("One shot on secondary complete; restarting primary");
this.primary.start();
return null;
}
done = true;
return new Date();
}
public void reset() {
done = false;
}
}
}
making my previous question more readable, following is my code which works fine for single server single client connection, but i want my client to connect 2 or more servers dynamically,
public class ClientCall {
public static void main(String[] args) {
#SuppressWarnings("resource")
ApplicationContext ctx = new AnnotationConfigApplicationContext(GatewayConfig.class);
GatewayService gatewayService = ctx.getBean(GatewayService.class);
//int i=0;
Message message = new Message();
/*while(i<4)
{*/
message.setPayload("It's working");
gatewayService.sendMessage(message);
/* i++;
}*/
}
}
public class Message {
private String payload;
// getter setter
}
#EnableIntegration
#IntegrationComponentScan
#Configuration
#ComponentScan(basePackages = "com.gateway.service")
public class GatewayConfig {
// #Value("${listen.port:6788}")
private int port = 6785;
#Autowired
private GatewayService<Message> gatewayService;
#MessagingGateway(defaultRequestChannel = "sendMessageChannel")
public interface Gateway {
void viaTcp(String payload);
}
#Bean
public AbstractClientConnectionFactory clientCF() {
TcpNetClientConnectionFactory clientConnectionFactory = new TcpNetClientConnectionFactory("localhost",
this.port);
clientConnectionFactory.setSingleUse(false);
return clientConnectionFactory;
}
#Bean
#ServiceActivator(inputChannel = "sendMessageChannel")
public MessageHandler tcpOutGateway(AbstractClientConnectionFactory connectionFactory) {
TcpOutboundGateway outGateway = new TcpOutboundGateway();
outGateway.setConnectionFactory(connectionFactory);
// outGateway.setAsync(true);
outGateway.setOutputChannel(receiveMessageChannel());
outGateway.setRequiresReply(true);
outGateway.setReplyChannel(receiveMessageChannel());
return outGateway;
}
#Bean
public MessageChannel sendMessageChannel() {
DirectChannel channel = new DirectChannel();
return channel;
}
#Bean
public MessageChannel receiveMessageChannel() {
DirectChannel channel = new DirectChannel();
return channel;
}
#Transformer(inputChannel = "receiveMessageChannel", outputChannel = "processMessageChannel")
public String convert(byte[] bytes) {
return new String(bytes);
}
#ServiceActivator(inputChannel = "processMessageChannel")
public void upCase(String response) {
gatewayService.receiveMessage(response);
}
#Transformer(inputChannel = "errorChannel", outputChannel = "processMessageChannel")
public void convertError(byte[] bytes) {
String str = new String(bytes);
System.out.println("Error: " + str);
}
}
public interface GatewayService<T> {
public void sendMessage(final T payload);
public void receiveMessage(String response);
}
#Service
public class GatewayServiceImpl implements GatewayService<Message> {
#Autowired
private Gateway gateway;
#Autowired
private GatewayContextManger<String, Object> gatewayContextManger;
#Override
public void sendMessage(final Message message) {
new Thread(new Runnable() {
#Override
public void run() {
gateway.viaTcp(message.getPayload());
}
}).start();
}
#Override
public void receiveMessage(final String response) {
new Thread(new Runnable() {
#Override
public void run() {
Message message = new Message();
message.setPayload(response);
Object obj = gatewayContextManger.get(message.getPayload());
synchronized (obj) {
obj.notify();
}
}
}).start();
}
}
and below is sever code similar there is another server with different port and ip then how to make connections to these servers?
class TCPServer
{
public static void main(String argv[]) throws Exception
{
String clientSentence;
String capitalizedSentence;
ServerSocket welcomeSocket = new ServerSocket(6785);
while(true)
{
Socket connectionSocket = welcomeSocket.accept();
BufferedReader inFromClient =
new BufferedReader(new InputStreamReader(connectionSocket.getInputStream()));
DataOutputStream outToClient = new DataOutputStream(connectionSocket.getOutputStream());
clientSentence = inFromClient.readLine();
System.out.println("Received: " + clientSentence);
capitalizedSentence = clientSentence + "\r\n";
outToClient.writeBytes(capitalizedSentence);
}
}
}
A few comments.
Instead of starting a thread to send the message, simply make sendMessageChannel an ExecutorChannel, using a ThreadPoolTaskExecutor - it will be more efficient and gets you out of the business of managing threads.
If you only have 2 servers to connect to, rather than coming up with a dynamic scheme, simply define 2 TCP adapters and add a #Router after sendMessageChannel.
You can tell the router which server to send it to by setting a header.
#MessagingGateway(defaultRequestChannel = "sendMessageChannel")
public interface Gateway {
void viaTcp(String payload #Header("which") String target);
}
Use a HeaderValueRouter to route on header which.
See Message Routing in the reference manual.
I am new with storm, and I am running a toplogy
public class FakeCallLogReaderSpout implements IRichSpout {
//Create instance for SpoutOutputCollector which passes tuples to bolt.
private SpoutOutputCollector collector;
private boolean completed = false;
//Create instance for TopologyContext which contains topology data.
private TopologyContext context;
//Create instance for Random class.
private Random randomGenerator = new Random();
private Integer idx = 0;
#Override
public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
this.context = context;
this.collector = collector;
}
#Override
public void nextTuple() {
if(this.idx <= 1000) {
List<String> mobileNumbers = new ArrayList<String>();
mobileNumbers.add("1234123401");
mobileNumbers.add("1234123402");
mobileNumbers.add("1234123403");
mobileNumbers.add("1234123404");
Integer localIdx = 0;
while(localIdx++ < 100 && this.idx++ < 1000) {
String fromMobileNumber = mobileNumbers.get(randomGenerator.nextInt(4));
String toMobileNumber = mobileNumbers.get(randomGenerator.nextInt(4));
while(fromMobileNumber == toMobileNumber) {
toMobileNumber = mobileNumbers.get(randomGenerator.nextInt(4));
}
Integer duration = randomGenerator.nextInt(60);
this.collector.emit(new Values(fromMobileNumber, toMobileNumber, duration),duration);
}
}
}
#Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("from", "to", "duration"));
}
//Override all the interface methods
#Override
public void close() {}
public boolean isDistributed() {
return false;
}
#Override
public void activate() {}
#Override
public void deactivate() {}
#Override
public void ack(Object msgId) {
System.out.println(msgId);
}
#Override
public void fail(Object msgId) {}
#Override
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
//Create a class CallLogCreatorBolt which implement IRichBolt interface
public class CallLogCreatorBolt implements IRichBolt {
//Create instance for OutputCollector which collects and emits tuples to produce output
private OutputCollector collector;
#Override
public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
this.collector = collector;
}
#Override
public void execute(Tuple tuple) {
String from = tuple.getString(0);
String to = tuple.getString(1);
Integer duration = tuple.getInteger(2);
collector.emit(new Values(from + " - " + to, duration));
}
#Override
public void cleanup() {}
#Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("call", "duration"));
}
#Override
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
public class CallLogCounterBolt implements IRichBolt {
Map<String, Integer> counterMap;
private OutputCollector collector;
#Override
public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
this.counterMap = new HashMap<String, Integer>();
this.collector = collector;
}
#Override
public void execute(Tuple tuple) {
String call = tuple.getString(0);
Integer duration = tuple.getInteger(1);
if(!counterMap.containsKey(call)){
counterMap.put(call, 1);
}else{
Integer c = counterMap.get(call) + 1;
counterMap.put(call, c);
}
collector.ack(tuple);
}
#Override
public void cleanup() {
for(Map.Entry<String, Integer> entry:counterMap.entrySet()){
System.out.println(entry.getKey()+" : " + entry.getValue());
}
}
#Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("call"));
}
#Override
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
I am calling collector.ack(tuple) from CallLogCounterBolt, the but ack is not getting called.
Does anybody have any idea why it is not getting called?
Also if drop the ack altogether from the code, what impact will it have?
You need to anchor and ack tuple in each bolt. I guess your topology is like FakeCallLogReaderSpout => CallLogCreatorBolt=> CallLogCounterBolt. Thus, in CallLogCreatorBolt you should do
collector.emit(tuple, new Values(from + " - " + to, duration));
collector.ack();
Read https://storm.apache.org/releases/0.10.0/Guaranteeing-message-processing.html for more details.
Why does the itemReader method is always sending the exact same file name to be processed in CustomItemProcessor?
As far as I understand, since I settup reader as #Scope and I set more than 1 in chunk, I was expecting the "return s" to move forward to next value from String array.
Let me clarify my question with a debug example in reader method:
1 - the variable stringArray is filled in with 3 file names (f1.txt, f2.txt and f3.txt)
2 - "return s" is evoked with s = f1.txt
3 - "return s" evoked again before evoked customItemProcessor method (perfect untill here since chunk = 2)
4 - looking at s it contains f1.txt again (different from what I expected. I expected f2.txt)
5 and 6 - runs processor with same name f1.tx (it should work correctly if the second turn of "return s" would contain f2.txt)
7 - writer method works as expected (processedFiles contain twice the two names processed in customItemProcessor f1.txt and f1.txt again since same name was processed twice)
CustomItemReader
public class CustomItemReader implements ItemReader<String> {
#Override
public String read() throws Exception, UnexpectedInputException,
ParseException, NonTransientResourceException {
String[] stringArray;
try (Stream<Path> stream = Files.list(Paths.get(env
.getProperty("my.path")))) {
stringArray = stream.map(String::valueOf)
.filter(path -> path.endsWith("out"))
.toArray(size -> new String[size]);
}
//*** the problem is here
//every turn s variable receives the first file name from the stringArray
if (stringArray.length > 0) {
for (String s : stringArray) {
return s;
}
} else {
log.info("read method - no file found");
return null;
}
return null;
}
CustomItemProcessor
public class CustomItemProcessor implements ItemProcessor<String , String> {
#Override
public String process(String singleFileToProcess) throws Exception {
log.info("process method: " + singleFileToProcess);
return singleFileToProcess;
}
}
CustomItemWriter
public class CustomItemWriter implements ItemWriter<String> {
private static final Logger log = LoggerFactory
.getLogger(CustomItemWriter.class);
#Override
public void write(List<? extends String> processedFiles) throws Exception {
processedFiles.stream().forEach(
processedFile -> log.info("**** write method"
+ processedFile.toString()));
FileSystem fs = FileSystems.getDefault();
for (String s : processedFiles) {
Files.deleteIfExists(fs.getPath(s));
}
}
Configuration
#Configuration
#ComponentScan(...
#EnableBatchProcessing
#EnableScheduling
#PropertySource(...
public class BatchConfig {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private JobRepository jobRepository;
#Bean
public TaskExecutor getTaskExecutor() {
return new TaskExecutor() {
#Override
public void execute(Runnable task) {
}
};
}
//I can see the number in chunk reflects how many time customReader is triggered before triggers customProcesser
#Bean
public Step step1(ItemReader<String> reader,
ItemProcessor<String, String> processor, ItemWriter<String> writer) {
return stepBuilderFactory.get("step1").<String, String> chunk(2)
.reader(reader).processor(processor).writer(writer)
.allowStartIfComplete(true).build();
}
#Bean
#Scope
public ItemReader<String> reader() {
return new CustomItemReader();
}
#Bean
public ItemProcessor<String, String> processor() {
return new CustomItemProcessor();
}
#Bean
public ItemWriter<String> writer() {
return new CustomItemWriter();
}
#Bean
public Job job(Step step1) throws Exception {
return jobBuilderFactory.get("job1").incrementer(new RunIdIncrementer()).start(step1).build();
}
Scheduler
#Component
public class QueueScheduler {
private static final Logger log = LoggerFactory
.getLogger(QueueScheduler.class);
private Job job;
private JobLauncher jobLauncher;
#Autowired
public QueueScheduler(JobLauncher jobLauncher, #Qualifier("job") Job job){
this.job = job;
this.jobLauncher = jobLauncher;
}
#Scheduled(fixedRate=60000)
public void runJob(){
try{
jobLauncher.run(job, new JobParameters());
}catch(Exception ex){
log.info(ex.getMessage());
}
}
}
Your issue is that you are relying on an internal loop to iterate over the items instead of letting Spring Batch do it for you by calling ItemReader#read multiple times.
What I'd recommend is changing your reader to the something like the following:
public class JimsItemReader implements ItemStreamReader {
private String[] items;
private int curIndex = -1;
#Override
public void open(ExecutionContext ec) {
curIndex = ec.getInt("curIndex", -1);
String[] stringArray;
try (Stream<Path> stream = Files.list(Paths.get(env.getProperty("my.path")))) {
stringArray = stream.map(String::valueOf)
.filter(path -> path.endsWith("out"))
.toArray(size -> new String[size]);
}
}
#Override
public void update(ExecutionContext ec) {
ec.putInt("curIndex", curIndex);
}
#Override
public String read() {
if (curIndex < items.length) {
curIndex++;
return items[curIndex];
} else {
return null;
}
}
}
The above example should loop through the items of your array as they are read. It also should be restartable in that we're storing the index in the ExecutionContext so if the job is restarted after a failure, you'll restart where you left off.