Does anyone have a good example of writing files to hdfs2 via camel?
I tried the following code :
import org.apache.camel.CamelContext;
import org.apache.camel.ProducerTemplate;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
public final class Main {
private Main() {
}
public static void main(String args[]) throws Exception {
CamelContext context = new DefaultCamelContext();
context.addRoutes(new RouteBuilder() {
public void configure() {
from("file:C:\\FILES\\SRC\\2015-03-31_16-58-56.png?noop=true")
.to("hdfs2://xxxx:9000/testCamel/D2/qwe.png");
//.to("file:C:\\FILES\\OUT");
}
});
//ProducerTemplate template = context.createProducerTemplate();
context.start();
context.stop();
}
}
The files are created in HDFS but all of them are empty (0 bytes).
make sure noop=false when you consume from a file to HDFS. hdfs component uses chunk to consume, so if noop is true, camel will think that it already consumed it.
Related
I am new to apache camel and open tracing with Jaeger, I trying to get the traces for apache camel in Jaeger UI but its not getting captured through Open Tracing.
import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
import org.springframework.stereotype.Component;
#Component
public class RouteBuilderClient {
public void test() throws Exception {
DefaultCamelContext camelContext = new DefaultCamelContext();
camelContext.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
from("timer:first-timer")
.to("log:first-timer");
}
});
camelContext.start();
camelContext.stop();
}
}
However if if I extend the RouteBuilder class(Below is the sample code) and then I override the configure method then traces are getting generated for Apache camel. Is there any way without extending the routeBuilder I can get the traces?
import org.apache.camel.builder.RouteBuilder;
import org.springframework.stereotype.Component;
#Component
public class RouteBuilderClient extends RouteBuilder {
#Override
public void configure() throws Exception {
from("timer:first-timer")
.to("log:first-timer");
}
}
My controller class:
import org.apache.camel.opentracing.starter.CamelOpenTracing;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
#SpringBootApplication
#CamelOpenTracing
public class JaegarClientApplication {
public static void main(String[] args) {
SpringApplication.run(JaegarClientApplication.class, args);
}
}
Try adding this in your setup() method, before starting the Camel context:
OpenTracingTracer ottracer = new OpenTracingTracer();
ottracer.init(camelContext);
camelContext.start();
Below is my code that Worked successfully.
import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
import org.apache.camel.opentracing.OpenTracingTracer;
public class RouteBuilderClient3 {
public void test() throws Exception {
RouteBuilder builder = new RouteBuilder() {
public void configure() {
from("timer:first-timer3").transform().constant("Rajesh client 3")
.to("log:first-timer3");
}
};
CamelContext myCamelContext = new DefaultCamelContext();
myCamelContext.addRoutes(builder);
OpenTracingTracer ottracer = new OpenTracingTracer();
ottracer.init(myCamelContext);
myCamelContext.start();
// myCamelContext.stop();
}
}
I want to use separate configuration files for different bundles instead of using a single configuration/config.ini file. I am following this doc https://www.eclipse.org/equinox/documents/quickstart-framework.php .
I am not able to figure out how to achieve this. Any suggestion would be very helpful.
Thanks in advance.
Following links solve the problem I was facing-
https://www.eclipse.org/forums/index.php/t/1072082/
https://git.eclipse.org/c/om2m/org.eclipse.om2m.git/tree/org.eclipse.om2m.site.mn-cse/configurations/services/lifx.basedriver.properties
https://git.eclipse.org/c/om2m/org.eclipse.om2m.git/tree/org.eclipse.om2m.sdt/org.eclipse.om2m.sdt.home.lifx/src/main/java/org/eclipse/om2m/sdt/home/lifx/impl/Activator.java
Steps I followed to achieve this -
Create configurations/services/<config_file_name>.properties having configurations in key-value pair. For example - property1 property1_value
Import package org.osgi.service.cm for your bundle
Use the following code
package <package_name>;
import java.util.Dictionary;
import java.util.Hashtable;
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
import org.osgi.framework.Constants;
import org.osgi.framework.ServiceRegistration;
import org.osgi.service.cm.ConfigurationException;
import org.osgi.service.cm.ManagedService;
public class Activator implements BundleActivator, ManagedService {
private static BundleContext context;
private ServiceRegistration managedServiceServiceRegistration;
static BundleContext getContext() {
return context;
}
public void start(BundleContext bundleContext) throws Exception {
Activator.context = bundleContext;
Dictionary properties = new Hashtable<>();
properties.put(Constants.SERVICE_PID, "config_file_name");
managedServiceServiceRegistration =
bundleContext.registerService(ManagedService.class.getName(), this, properties);
}
public void stop(BundleContext bundleContext) throws Exception {
Activator.context = null;
}
#Override
public void updated(Dictionary properties) throws ConfigurationException {
try {
System.out.println(properties.get("property1"));
} catch (Exception e) {
e.printStackTrace();
}
}
}
Out of curiosity I'd like to build a simple websockets tyrus server chat project with javac and libraries in -classpath.
Yes, I know that this is not the standard way (mvn is), but I'm doing it as a proof of concept. I manage to get project built but it throws a NullPointerException
Code is:
ChatMain.java:
package chat;
import java.io.IOException;
import java.util.Collections;
import java.util.Map;
import javax.websocket.DeploymentException;
import org.glassfish.tyrus.server.Server;
public class ChatMain {
public static void main(String[] args) throws DeploymentException,
IOException {
Map<String, Object> properties = Collections.emptyMap();
Server server = new Server("localhost", 8080, "/ws", properties,
ChatEndPoint.class);
System.out.println(server);
try {
server.start();
System.in.read();
} finally {
server.stop();
}
}
}
ChatEndPoint.java:
package chat;
import java.io.IOException;
import java.util.Map;
import java.util.HashMap;
import java.util.Collections;
import javax.websocket.OnClose;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.ServerEndpoint;
import org.json.*;
#ServerEndpoint("/chat")
public class ChatEndPoint {
private static Map<String, Session> peers = Collections.synchronizedMap(new HashMap<String, Session>());
#OnOpen
public void onOpen(Session peer) {
System.out.println("onOpen");
}
#OnMessage
public void onMessage(String message, Session peer) throws IOException {
System.out.println("onMessage");
JSONObject json = new JSONObject(message);
String nick = (String) json.get("nick");
boolean newClient = json.has("action") && ((String) json.get("action")).equals("add");
synchronized(peers) {
// Iterate over the connected sessions
// and broadcast the received message
for (Map.Entry<String, Session> entry : peers.entrySet()) {
if (newClient) {
JSONObject json1 = new JSONObject(json, new String[] { "action" }).put("nick", entry.getKey());
peer.getBasicRemote().sendText(json1.toString());
}
entry.getValue().getBasicRemote().sendText(message);
}
}
if (newClient)
peers.put(nick, peer);
}
#OnClose
public void onClose(Session peer) {
System.out.println("onClose");
peers.values().remove(peer);
}
}
javac line:
javac -cp chat/javax.websocket-api-1.1.jar:chat/tyrus-server-1.12.jar:chat/tyrus-spi-1.12.jar:chat/tyrus-core-1.12.jar:chat/grizzly-framework-2.3.22.jar:chat/grizzly-http-server-2.3.22.jar:chat/tyrus-container-grizzly-server-1.12.jar:chat/json.jar: chat/ChatMain.java
java line:
java -cp chat/javax.websocket-api-1.1.jar:chat/tyrus-server-1.12.jar:chat/tyrus-spi-1.12.jar:chat/tyrus-core-1.12.jar:chat/grizzly-framework-2.3.22.jar:chat/grizzly-http-server-2.3.22.jar:chat/tyrus-container-grizzly-server-1.12.jar:chat/json.jar: chat.ChatMain
runtime error:
Exception in thread "main" java.lang.NullPointerException
at org.glassfish.tyrus.container.grizzly.server.GrizzlyServerContainer$1.stop(GrizzlyServerContainer.java:228)
at org.glassfish.tyrus.server.Server.stop(Server.java:231)
at chat.ChatMain.main(ChatMain.java:23)
I understand that this can be done
Interestingly, more libraries are needed. If I include all libraries in websocket-ri-archive-1.12.zip bundle, it works well. Definitely, a builder is the way to go
Is there any way to create topologies dynamically in trident ?
Could anyone please provide examples ?
First of all, you might also know that creating topologies is not part of Trident. Trident is just an API for microbatching.
And creating new topologies is dynamic by definition. This is what the TopologyBuilder class is doing.
So to answer your question, yes it is possible to create new topologies from Trident, or from simple Storm spouts and bolts. The only thing you need is that your topology creating logic should have access to the Storm cluster (classes and other resources) which is again by definition satisfied if you run your logic in Storm.
The last thing you will need is to find a way to submit the newly created topology, and this is what the StormSubmitter class was made for, which is again (!surprise :) ) by definitions satisfied to be on your classpath when you run your logic inside a Trident or ordinary spout/bolt.
Out of curiosity, why are you planning to do this? What are your requirements?
Example:
import java.util.Map;
import org.apache.storm.Config;
import org.apache.storm.StormSubmitter;
import org.apache.storm.generated.StormTopology;
import org.apache.storm.spout.SpoutOutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.TopologyBuilder;
import org.apache.storm.topology.base.BaseRichSpout;
import org.apache.storm.trident.operation.TridentCollector;
import org.apache.storm.trident.spout.IBatchSpout;
import org.apache.storm.tuple.Fields;
public class DynamicTopologySpout implements IBatchSpout {
private static final long serialVersionUID = -3269435263455830842L;
#Override
#SuppressWarnings("rawtypes")
public void open(Map conf, TopologyContext context) {}
#Override
public void emitBatch(long batchId, TridentCollector collector) {
if (newTopologyNeeded()) {
TopologyBuilder builder = new TopologyBuilder();
builder
.setSpout("spout", new BaseRichSpout() {
private static final long serialVersionUID = 1L;
#Override public void declareOutputFields(OutputFieldsDeclarer declarer) {}
#Override #SuppressWarnings("rawtypes") public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {}
#Override public void nextTuple() {}
}, 1)
.setMaxSpoutPending(15)
.setNumTasks(1);
StormTopology topology = builder.createTopology();
Config config = new Config();
try {
StormSubmitter.submitTopology("dynamic-topology", config, topology);
} catch (Exception e) {
e.printStackTrace();
collector.reportError(e);
}
}
}
private boolean newTopologyNeeded() {
// Check if topology needed ...
return false;
}
#Override
public void ack(long batchId) {}
#Override
public void close() {}
#Override
public Map<String, Object> getComponentConfiguration() { return null; }
#Override
public Fields getOutputFields() { return null; }
}
I am relatively new to the flume interceptors concept and facing an issue where before applying the interceptor the file sinked is normal text file and after applying the interceptor everything turns really bad.
My interceptor code as below -
package com.flume;
import org.apache.flume.*;
import org.apache.flume.interceptor.*;
import java.util.List;
import java.util.Map;
import java.util.ArrayList;
import java.io.UnsupportedEncodingException;
import java.net.InetAddress;
import java.net.UnknownHostException;
public class CustomHostInterceptor implements Interceptor {
private String hostValue;
private String hostHeader;
public CustomHostInterceptor(String hostHeader){
this.hostHeader = hostHeader;
}
#Override
public void initialize() {
// At interceptor start up
try {
hostValue =
InetAddress.getLocalHost().getHostName();
} catch (UnknownHostException e) {
throw new FlumeException("Cannot get Hostname", e);
}
}
#Override
public Event intercept(Event event) {
// This is the event's body
String body = new String(event.getBody());
if(body.toLowerCase().contains("text")){
try {
event.setBody("hadoop".getBytes("UTF-8"));
} catch (UnsupportedEncodingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
// These are the event's headers
Map<String, String> headers = event.getHeaders();
// Enrich header with hostname
headers.put(hostHeader, hostValue);
// Let the enriched event go
return event;
}
#Override
public List<Event> intercept(List<Event> events) {
List<Event> interceptedEvents =
new ArrayList<Event>(events.size());
for (Event event : events) {
// Intercept any event
Event interceptedEvent = intercept(event);
interceptedEvents.add(interceptedEvent);
}
return interceptedEvents;
}
#Override
public void close() {
// At interceptor shutdown
}
public static class Builder
implements Interceptor.Builder {
private String hostHeader;
#Override
public void configure(Context context) {
// Retrieve property from flume conf
hostHeader = context.getString("hostHeader");
}
#Override
public Interceptor build() {
return new CustomHostInterceptor(hostHeader);
}
}
}
Flume conf is -
agent.sources=exec-source
agent.sinks=hdfs-sink
agent.channels=ch1
agent.sources.exec-source.type=exec
agent.sources.exec-source.command=tail -F /home/cloudera/Desktop/app.log
agent.sources.exec-source.interceptors = i1
agent.sources.exec-source.interceptors.i1.type = com.flume.CustomHostInterceptor$Builder
agent.sources.exec-source.interceptors.i1.hostHeader = hostname
agent.sinks.hdfs-sink.type=hdfs
agent.sinks.hdfs-sink.hdfs.path= hdfs://localhost:8020/bosch/flume/applogs
agent.sinks.hdfs-sink.hdfs.filePrefix=logs
agent.sinks.hdfs-sink.hdfs.rollInterval=60
agent.sinks.hdfs-sink.hdfs.rollSize=0
agent.channels.ch1.type=memory
agent.channels.ch1.capacity=1000
agent.sources.exec-source.channels=ch1
agent.sinks.hdfs-sink.channel=ch1
on doing a cat on the file created in HDFS -
SEQ!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable���*q�CJv�/ESmP�ź
some textP�żc
some more textP���K
textP��ߌangels and deamonsP��%�
text bla blaP��1�angels and deamonsP��1�
testP��1�hmmmP��1�anything
Any suggestions?
Thanks
Looks like nothing Wrong with Interceptor.
In your Flume Agent config.
You are not specifying this property (hdfs.fileType) so it is taking this as a default SequenceFile
Try adding this line to your HDFS SINK and let me know if this works.
agent.sinks.hdfs-sink.hdfs.fileType=DataStream