Apache storm Trident - creating topologies dynamically - apache-storm

Question

Is there any way to create topologies dynamically in trident ?
Could anyone please provide examples ?

Answer 1

First of all, you might also know that creating topologies is not part of Trident. Trident is just an API for microbatching.
And creating new topologies is dynamic by definition. This is what the TopologyBuilder class is doing.
So to answer your question, yes it is possible to create new topologies from Trident, or from simple Storm spouts and bolts. The only thing you need is that your topology creating logic should have access to the Storm cluster (classes and other resources) which is again by definition satisfied if you run your logic in Storm.
The last thing you will need is to find a way to submit the newly created topology, and this is what the StormSubmitter class was made for, which is again (!surprise :) ) by definitions satisfied to be on your classpath when you run your logic inside a Trident or ordinary spout/bolt.
Out of curiosity, why are you planning to do this? What are your requirements?
Example:
import java.util.Map;
import org.apache.storm.Config;
import org.apache.storm.StormSubmitter;
import org.apache.storm.generated.StormTopology;
import org.apache.storm.spout.SpoutOutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.TopologyBuilder;
import org.apache.storm.topology.base.BaseRichSpout;
import org.apache.storm.trident.operation.TridentCollector;
import org.apache.storm.trident.spout.IBatchSpout;
import org.apache.storm.tuple.Fields;
public class DynamicTopologySpout implements IBatchSpout {
private static final long serialVersionUID = -3269435263455830842L;
#Override
#SuppressWarnings("rawtypes")
public void open(Map conf, TopologyContext context) {}
#Override
public void emitBatch(long batchId, TridentCollector collector) {
if (newTopologyNeeded()) {
TopologyBuilder builder = new TopologyBuilder();
builder
.setSpout("spout", new BaseRichSpout() {
private static final long serialVersionUID = 1L;
#Override public void declareOutputFields(OutputFieldsDeclarer declarer) {}
#Override #SuppressWarnings("rawtypes") public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {}
#Override public void nextTuple() {}
}, 1)
.setMaxSpoutPending(15)
.setNumTasks(1);
StormTopology topology = builder.createTopology();
Config config = new Config();
try {
StormSubmitter.submitTopology("dynamic-topology", config, topology);
} catch (Exception e) {
e.printStackTrace();
collector.reportError(e);
}
}
}
private boolean newTopologyNeeded() {
// Check if topology needed ...
return false;
}
#Override
public void ack(long batchId) {}
#Override
public void close() {}
#Override
public Map<String, Object> getComponentConfiguration() { return null; }
#Override
public Fields getOutputFields() { return null; }
}