The goal is to read data emitted over the network.
On the data generation side I have an app which spews to stdout. The content of this data is a JSON string.
Here's what I'm doing (on Linux Mint 17, using an BSD flavored netcat):
data generation:
my_app_which_outputs_json | netcat localhost 9999
In SpringXD: (with xd-singlenode)
xd:>stream create --name tcptest --definition "tcp --decoder=LF --port=9999 | file " --deploy
Created and deployed new stream 'tcptest'
Output:
/tmp/xd/output$ cat tcptest.out
82,117,110, ... (etc, lots more bytes)
I'm sure this is user error, but not sure what to change to make it right.
I should note that if I do this, it works as expected:
my_app_which_outputs_json > /tmp/somefile.txt
...
xd:>stream create --name filetest --definition "tail --name=/tmp/somefile.txt | file" --deploy
Following up to your own answer, the converters are currently not configured to understand that a byte[] can be converted to a String with content type application/json.
Of course, even if we configured it to do so, it wouldn't be able to determine if the content really is JSON.
I added the following to my stream defn and it is now doing what I expected. I found this in the "Type Conversion" section of the documentation.
stream create --name tcptest \
--definition "tcp --decoder=LF --port=9999 --outputType=text/plain \
| file " --deploy
(The newlines and backwhacks are not in my actual code, but are used for readability.)
I also tried ... --outputType=application/json... which did not work. Not entirely sure why.
text/plain with a byte[] payload activates a ByteArrayToStringMessageConverter. application/json does not currently. I have created https://jira.spring.io/browse/XD-2512 to address this
Related
I have a elasticsearch cluster running on Azure AKS. I want to connect to a different es cluster running on seperate AKS for which I need to export certificate from one cluster and add it to the other cluster. I am following the official documentation from here
.
However I am not able to export the certificate and getting error on executing the following command:
kubectl get secret europecluster-es-transport-certs-public -o
go-template='{{index .data "ca.crt"}}'
Error I am getting is:
error: error parsing template {{index .data ca.crt}}, template: output:1: function "ca" not defined
I am novice in elastic and kubernetes space, and not able to find solution for this on the internet.
If you are okay to manually extract the ca.crt value and decode it then you can try following:
Extract ca.crt value without quotes [copy to clipboard]
kubectl get secret europecluster-es-transport-certs-public | grep ca.crt
perform a base64decode and redirect it to a file
echo -n <paste clipboard content> | base64 -d -w 0 > remote.ca.crt
Above procedure performs same operation as go template is doing in your command.
Example:
kubectl get secret default-token-h8w57 -o json | grep -i ca.crt
"ca.crt": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01UQXlPVEV4TkRVeU9Gb1hEVE13TVRBeU56RXhORFV5T0Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTEl4CmpjMCttcGVXWm5TL3NnenloZ1Ftd1ZaN3A1b2hoUktqY0gvNkVIbDBRbzljOTkyZVBTcmEzNEU0cHpmZXRBUE8Kdm1Ia0Q2Z0dCb1FyVUI3NHFMOFZpaUs4c0hZQXcyWElxMERTZHhHb3VqcUVabUM4SnpSK3gxVE1CaUZ2YUR4dQpaZVpTT3JTc1R2dGN6TjNnMG5XK0xPY1Q2UCtGQlRLbzh1RXBjbXY5cll1ZytOR25xZ0l3L0VNRXlQenM4RGk1CkhzYVJma0FwSmloeERUdTBTY1Z5MkpaakxZZ2RBMUlaSkRScjV6Unc1U3RlWlltTm5rVTY5cEtVVlNlQ2lQWnUKMFdlY3ZaTXE1NDhKWWtmUStWY3pFMjFtUTBJMSs4NXpOUUFvQmZ4aG5tYjNhcW5yL2hEdUZETm9PelIrdCtUSApteTU2ajRWTUtzY3RvNUxkOFFFQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZNZVlQcGVuYmV3RUg4bFFKdDlxaUs4bG5QWmFNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFDbFpIZGQrZDlWWElobTdsdhskhdjshdjsahdjkasdhkasdhasXOUhQNC9HMXRScTVLUWtZSlJjVHdreGZWNUlhMS8zNW1vRwpyeU5SOVZzYnRZeDF6aFNsRy91NWRGOWFYYjI3M2J4bWNEOVY0UUQvamNXMWRsdnJ6NlFWMGg3dEcwcUd6UG1xClUveC9saXJaTWMrTmVKSXJXZGo5ZjM5dXFuR2VCZnF6ZWN4QXBoRG5xY1dUNWZTVjlSVjdqaE5sNnhSZUVlRGMKUmZQMnFlb3g4d0xyYXBiVDVOSG9PK1FjS3NoUHhPL0FTNXhVVE9yOTZ2YTZkSFhzZFdsQWdaTUtva1lldlN1SApBdjVrYml3ODJBVzlaOHZrS0QrQXdFSWFwdzNNQnEvOUFxQjZBZm93RTJCckZVcTdwVzk3ZHUvRC81NWxQbTN5CllmVFo3ZVZnQUF4Yk1lTDRDdlhSZ1FJWHB5NmROTFN0SGJCSAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
echo -n LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01UQXlPVEV4TkRVeU9Gb1hEVE13TVRBeU56RXhORFV5T0Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTEl4CmpjMCttcGVXWm5TL3NnenloZ1Ftd1ZaN3A1b2hoUktqY0gvNkVIbDBRbzljOTkyZVBTcmEzNEU0cHpmZXRBUE8Kdm1Ia0Q2Z0dCb1FyVUI3NHFMOFZpaUs4c0hZQXcyWElxMERTZHhHb3VqcUVabUM4SnpSK3gxVE1CaUZ2YUR4dQpaZVpTT3JTc1R2dGN6TjNnMG5XK0xPY1Q2UCtGQlRLbzh1RXBjbXY5cll1ZytOR25xZ0l3L0VNRXlQenM4RGk1CkhzYVJma0FwSmloeERUdTBTY1Z5MkpaakxZZ2RBMUlaSkRScjV6Unc1U3RlWlltTm5rVTY5cEtVVlNlQ2lQWnUKMFdlY3ZaTXE1NDhKWWtmUStWY3pFMjFtUTBJMSs4NXpOUUFvQmZ4aG5tYjNhcW5yL2hEdUZETm9PelIrdCtUSApteTU2ajRWTUtzY3RvNUxkOFFFQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZNZVlQcGVuYmV3RUg4bFFKdDlxaUs4bG5QWmFNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFDbFpIZGQrZDlWWElobTdsdhskhdjshdjsahdjkasdhkasdhasXOUhQNC9HMXRScTVLUWtZSlJjVHdreGZWNUlhMS8zNW1vRwpyeU5SOVZzYnRZeDF6aFNsRy91NWRGOWFYYjI3M2J4bWNEOVY0UUQvamNXMWRsdnJ6NlFWMGg3dEcwcUd6UG1xClUveC9saXJaTWMrTmVKSXJXZGo5ZjM5dXFuR2VCZnF6ZWN4QXBoRG5xY1dUNWZTVjlSVjdqaE5sNnhSZUVlRGMKUmZQMnFlb3g4d0xyYXBiVDVOSG9PK1FjS3NoUHhPL0FTNXhVVE9yOTZ2YTZkSFhzZFdsQWdaTUtva1lldlN1SApBdjVrYml3ODJBVzlaOHZrS0QrQXdFSWFwdzNNQnEvOUFxQjZBZm93RTJCckZVcTdwVzk3ZHUvRC81NWxQbTN5CllmVFo3ZVZnQUF4Yk1lTDRDdlhSZ1FJWHB5NmROTFN0SGJCSAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== | base64 -d -w 0 > remote.ca.crt
I am trying to feed some netflow data into kafka. I have some netflow.pcap files which I read like
tcpdump -r netflow.pcap and get such an output:
14:48:40.823468 IP abts-kk-static-242.4.166.122.airtelbroadband.in.35467 > abts-kk-static-126.96.166.122.airtelbroadband.in.9500: UDP, length 1416
14:48:40.824216 IP abts-kk-static-242.4.166.122.airtelbroadband.in.35467 > abts-kk-static-126.96.166.122.airtelbroadband.in.9500: UDP, length 1416
.
.
.
.
In the official docs they mention the traditional way of starting a kafka producer, starting a kafka consumer and in the terminal input some data on producer which will be shown in the consumer. Good. Working.
Here they show how to input a file to kafka producer. Mind you, just one single file, not multiple files.
Question is:
How can I feed the output of a shell script into kakfa broker?
For example, the shell script is:
#!/bin/bash
FILES=/path/to/*
for f in $FILES
do
tcpdump -r netflow.pcap
done
I can't find any documentation or article where they mention how to do this. Any idea? Thanks!
Well, based on the link you gave on how to use the shell kafka producer with an input file, you can do the same with your output. You can redirect the output to a file and then use the producer.
Pay attention that I used >> in order to append to the file and not to overwrite it.
For example:
#!/bin/bash
FILES=/path/to/*
for f in $FILES
do
tcpdump -r netflow.pcap >> /tmp/tcpdump_output.txt
done
kafka-console-produce.sh --broker-list localhost:9092 --topic my_topic
--new-producer < /tmp/tcpdump_output.txt
I created a twitter stream to extract a tweet info that I created, the stream is below:
xd:>stream create --name twitter2 --definition "twittersearch --consumerSecret=xxx --consumerKey=xxx --query='#SpringXDisSunnyThisEvening' | file" --deploy
The stream worked fine, I ended up with a file.out "JSON" which contains my tweet info.
I tried to add a filer to my stream like the below:
xd:>stream create --name twitter2 --definition "twittersearch --consumerSecret=xxx --consumerKey=xxx --query='#SpringXDisSunnyThisEvening' | filter --expression=#jsonPath(payload,'$.name').contains('moha') | file" --deploy
It did not work, no output file.
Log:
015-07-04T16:32:48-0500 1.2.0.RELEASE ERROR inbound.twitter2.0-redis:queue-inbound-channel-adapter1 redis.RedisMessageBus$1 - Failed to deliver message; retries exhausted; message sent to queue 'ERRORS:twitter2.0' org.springframework.messaging.MessageHandlingException: Expression evaluation failed: #jsonPath(payload,'$.name').contains('mohammad'); nested exception is java.lang.reflect.InvocationTargetException
I think you need to use this expression
--expression=#jsonPath(payload,'$.user.screen_name').contains('something')
Note it is user.screen_name instead of just name from the twitter response.
I have a stream "myStream" of text records of different types, say "A", "B", "C", and so on...Each type of records needs to be processed differently. I want to create multiple taps on "myStream" for each type: tapA, tapB, tapC, etc. But it seems tap doesn't have any filtering capability. Is that true or am I missing something? Is there a better way to accomplish this?
You can have filter on a tap stream. You can see tap as a regular stream that has source channel (tap channel) set and this source channel can be combined with multiple processors and a single sink.
See this example:
xd:>stream create test --definition "http | log" --deploy
xd:>http post --data test
xd:>stream create tap-test --definition "tap:stream:test.http > filter --expression=payload.contains('a') | log" --deploy
xd:>http post --data test
xd:>http post --data apple
and in the container log you would see something like this:
2015-05-19 11:48:36,276 1.2.0.SNAP INFO pool-16-thread-11 sink.test - test
2015-05-19 11:48:41,445 1.2.0.SNAP INFO pool-16-thread-17 sink.test - apple
2015-05-19 11:48:41,445 1.2.0.SNAP INFO xd.localbus-2 sink.tap-test - apple
Note the payload 'apple' comes out of regular and tap stream.
I have a question is how to create stream with multi processor. For ex:
stream create --name multiModuleTest --definition "tcp | processor1 | processor2 |file" --deploy
Or
stream create --name multiModuleTest --definition "(tcp, http) | processor1 | processor2 | (file,log)" --deploy
Please tell me how to do that? Thanks !
Fire up multiple containers and use the deployment manifest to set the module count.
If you want to multiprocess within a processor instance, change the input channel to an ExecectorChannel. Docs here and here.