I have a question is how to create stream with multi processor. For ex:
stream create --name multiModuleTest --definition "tcp | processor1 | processor2 |file" --deploy
Or
stream create --name multiModuleTest --definition "(tcp, http) | processor1 | processor2 | (file,log)" --deploy
Please tell me how to do that? Thanks !
Fire up multiple containers and use the deployment manifest to set the module count.
If you want to multiprocess within a processor instance, change the input channel to an ExecectorChannel. Docs here and here.
Related
I created a stream named vfs-od-fe-hb，where the source is vfs, the processors are od and fe, and the sink is hb. I want to create a tap in the postion of fe and store the result into hdfs.
My shell command is:
stream create --name vfs-od-fe-hb-tap-ib-hdfs --definition "tap:stream:vfs-od-fe-hb.fe > ib > hdfs" --deploy
ib is my processor, it can handle the messages from fe successfully, but now the wrong messages:
Command failed org.springframework.xd.rest.client.impl.SpringXDException: XD133E:(pos 34): Expected channel prefix of 'queue' or 'topic' but found 'hdfs'
If ib is a processor, the syntax should be
tap:stream:vfs-od-fe-hb.fe > ib | hdfs
i.e. the tap (named channel) is ib's input channel and we pipe its output as normal.
The parser is reacting to the > hdfs because > can only be between a named channel and processor or sink.
(or after a processor when sending to a named channel - queue:..., topic:...).
I created a twitter stream to extract a tweet info that I created, the stream is below:
xd:>stream create --name twitter2 --definition "twittersearch --consumerSecret=xxx --consumerKey=xxx --query='#SpringXDisSunnyThisEvening' | file" --deploy
The stream worked fine, I ended up with a file.out "JSON" which contains my tweet info.
I tried to add a filer to my stream like the below:
xd:>stream create --name twitter2 --definition "twittersearch --consumerSecret=xxx --consumerKey=xxx --query='#SpringXDisSunnyThisEvening' | filter --expression=#jsonPath(payload,'$.name').contains('moha') | file" --deploy
It did not work, no output file.
Log:
015-07-04T16:32:48-0500 1.2.0.RELEASE ERROR inbound.twitter2.0-redis:queue-inbound-channel-adapter1 redis.RedisMessageBus$1 - Failed to deliver message; retries exhausted; message sent to queue 'ERRORS:twitter2.0' org.springframework.messaging.MessageHandlingException: Expression evaluation failed: #jsonPath(payload,'$.name').contains('mohammad'); nested exception is java.lang.reflect.InvocationTargetException
I think you need to use this expression
--expression=#jsonPath(payload,'$.user.screen_name').contains('something')
Note it is user.screen_name instead of just name from the twitter response.
I have a stream "myStream" of text records of different types, say "A", "B", "C", and so on...Each type of records needs to be processed differently. I want to create multiple taps on "myStream" for each type: tapA, tapB, tapC, etc. But it seems tap doesn't have any filtering capability. Is that true or am I missing something? Is there a better way to accomplish this?
You can have filter on a tap stream. You can see tap as a regular stream that has source channel (tap channel) set and this source channel can be combined with multiple processors and a single sink.
See this example:
xd:>stream create test --definition "http | log" --deploy
xd:>http post --data test
xd:>stream create tap-test --definition "tap:stream:test.http > filter --expression=payload.contains('a') | log" --deploy
xd:>http post --data test
xd:>http post --data apple
and in the container log you would see something like this:
2015-05-19 11:48:36,276 1.2.0.SNAP INFO pool-16-thread-11 sink.test - test
2015-05-19 11:48:41,445 1.2.0.SNAP INFO pool-16-thread-17 sink.test - apple
2015-05-19 11:48:41,445 1.2.0.SNAP INFO xd.localbus-2 sink.tap-test - apple
Note the payload 'apple' comes out of regular and tap stream.
The goal is to read data emitted over the network.
On the data generation side I have an app which spews to stdout. The content of this data is a JSON string.
Here's what I'm doing (on Linux Mint 17, using an BSD flavored netcat):
data generation:
my_app_which_outputs_json | netcat localhost 9999
In SpringXD: (with xd-singlenode)
xd:>stream create --name tcptest --definition "tcp --decoder=LF --port=9999 | file " --deploy
Created and deployed new stream 'tcptest'
Output:
/tmp/xd/output$ cat tcptest.out
82,117,110, ... (etc, lots more bytes)
I'm sure this is user error, but not sure what to change to make it right.
I should note that if I do this, it works as expected:
my_app_which_outputs_json > /tmp/somefile.txt
...
xd:>stream create --name filetest --definition "tail --name=/tmp/somefile.txt | file" --deploy
Following up to your own answer, the converters are currently not configured to understand that a byte[] can be converted to a String with content type application/json.
Of course, even if we configured it to do so, it wouldn't be able to determine if the content really is JSON.
I added the following to my stream defn and it is now doing what I expected. I found this in the "Type Conversion" section of the documentation.
stream create --name tcptest \
--definition "tcp --decoder=LF --port=9999 --outputType=text/plain \
| file " --deploy
(The newlines and backwhacks are not in my actual code, but are used for readability.)
I also tried ... --outputType=application/json... which did not work. Not entirely sure why.
text/plain with a byte[] payload activates a ByteArrayToStringMessageConverter. application/json does not currently. I have created https://jira.spring.io/browse/XD-2512 to address this
For example, I'd like to define a stream in Spring-XD as, 'source1 | sink1'
and the 'source1' module has a option: 'option1'.
What I like to do is to configure 'source1' to have multiple instances (this is doable in Spring-XD), and at the same time, I like to let each instance of 'source1' to have different value on 'option1'.
Is it possible with currently-available DSL design in Spring-xd?
You can't currently do that but you can use named channels...
stream create --name foo --definition "source1 --option=x > queue:xx"
stream create --name bar --definition "source1 --option=y > queue:xx"
stream create --name baz --definition "queue:xx > sink1"