I created a stream named vfs-od-fe-hb，where the source is vfs, the processors are od and fe, and the sink is hb. I want to create a tap in the postion of fe and store the result into hdfs.
My shell command is:
stream create --name vfs-od-fe-hb-tap-ib-hdfs --definition "tap:stream:vfs-od-fe-hb.fe > ib > hdfs" --deploy
ib is my processor, it can handle the messages from fe successfully, but now the wrong messages:
Command failed org.springframework.xd.rest.client.impl.SpringXDException: XD133E:(pos 34): Expected channel prefix of 'queue' or 'topic' but found 'hdfs'
If ib is a processor, the syntax should be
tap:stream:vfs-od-fe-hb.fe > ib | hdfs
i.e. the tap (named channel) is ib's input channel and we pipe its output as normal.
The parser is reacting to the > hdfs because > can only be between a named channel and processor or sink.
(or after a processor when sending to a named channel - queue:..., topic:...).
Related
I created a twitter stream to extract a tweet info that I created, the stream is below:
xd:>stream create --name twitter2 --definition "twittersearch --consumerSecret=xxx --consumerKey=xxx --query='#SpringXDisSunnyThisEvening' | file" --deploy
The stream worked fine, I ended up with a file.out "JSON" which contains my tweet info.
I tried to add a filer to my stream like the below:
xd:>stream create --name twitter2 --definition "twittersearch --consumerSecret=xxx --consumerKey=xxx --query='#SpringXDisSunnyThisEvening' | filter --expression=#jsonPath(payload,'$.name').contains('moha') | file" --deploy
It did not work, no output file.
Log:
015-07-04T16:32:48-0500 1.2.0.RELEASE ERROR inbound.twitter2.0-redis:queue-inbound-channel-adapter1 redis.RedisMessageBus$1 - Failed to deliver message; retries exhausted; message sent to queue 'ERRORS:twitter2.0' org.springframework.messaging.MessageHandlingException: Expression evaluation failed: #jsonPath(payload,'$.name').contains('mohammad'); nested exception is java.lang.reflect.InvocationTargetException
I think you need to use this expression
--expression=#jsonPath(payload,'$.user.screen_name').contains('something')
Note it is user.screen_name instead of just name from the twitter response.
I have a stream "myStream" of text records of different types, say "A", "B", "C", and so on...Each type of records needs to be processed differently. I want to create multiple taps on "myStream" for each type: tapA, tapB, tapC, etc. But it seems tap doesn't have any filtering capability. Is that true or am I missing something? Is there a better way to accomplish this?
You can have filter on a tap stream. You can see tap as a regular stream that has source channel (tap channel) set and this source channel can be combined with multiple processors and a single sink.
See this example:
xd:>stream create test --definition "http | log" --deploy
xd:>http post --data test
xd:>stream create tap-test --definition "tap:stream:test.http > filter --expression=payload.contains('a') | log" --deploy
xd:>http post --data test
xd:>http post --data apple
and in the container log you would see something like this:
2015-05-19 11:48:36,276 1.2.0.SNAP INFO pool-16-thread-11 sink.test - test
2015-05-19 11:48:41,445 1.2.0.SNAP INFO pool-16-thread-17 sink.test - apple
2015-05-19 11:48:41,445 1.2.0.SNAP INFO xd.localbus-2 sink.tap-test - apple
Note the payload 'apple' comes out of regular and tap stream.
The goal is to read data emitted over the network.
On the data generation side I have an app which spews to stdout. The content of this data is a JSON string.
Here's what I'm doing (on Linux Mint 17, using an BSD flavored netcat):
data generation:
my_app_which_outputs_json | netcat localhost 9999
In SpringXD: (with xd-singlenode)
xd:>stream create --name tcptest --definition "tcp --decoder=LF --port=9999 | file " --deploy
Created and deployed new stream 'tcptest'
Output:
/tmp/xd/output$ cat tcptest.out
82,117,110, ... (etc, lots more bytes)
I'm sure this is user error, but not sure what to change to make it right.
I should note that if I do this, it works as expected:
my_app_which_outputs_json > /tmp/somefile.txt
...
xd:>stream create --name filetest --definition "tail --name=/tmp/somefile.txt | file" --deploy
Following up to your own answer, the converters are currently not configured to understand that a byte[] can be converted to a String with content type application/json.
Of course, even if we configured it to do so, it wouldn't be able to determine if the content really is JSON.
I added the following to my stream defn and it is now doing what I expected. I found this in the "Type Conversion" section of the documentation.
stream create --name tcptest \
--definition "tcp --decoder=LF --port=9999 --outputType=text/plain \
| file " --deploy
(The newlines and backwhacks are not in my actual code, but are used for readability.)
I also tried ... --outputType=application/json... which did not work. Not entirely sure why.
text/plain with a byte[] payload activates a ByteArrayToStringMessageConverter. application/json does not currently. I have created https://jira.spring.io/browse/XD-2512 to address this
For example, I'd like to define a stream in Spring-XD as, 'source1 | sink1'
and the 'source1' module has a option: 'option1'.
What I like to do is to configure 'source1' to have multiple instances (this is doable in Spring-XD), and at the same time, I like to let each instance of 'source1' to have different value on 'option1'.
Is it possible with currently-available DSL design in Spring-xd?
You can't currently do that but you can use named channels...
stream create --name foo --definition "source1 --option=x > queue:xx"
stream create --name bar --definition "source1 --option=y > queue:xx"
stream create --name baz --definition "queue:xx > sink1"
I am trying to transfer a 60mb file to queue, but Websphere MQ fte stall the transfer and keep recovering. I am running WebSphere MQ FTE on default configuration.
I have tested following scenario with different results according to configuration changes I made.
These commands were issued to create monitor:
fteCreateTransfer -sa AGENT1 -sm TQM.FTE -da AGENT2 -dm QM.FTE -dq FTE.TEST.Q -p QM.FTE -de overwrite -sd delete -gt /var/IBM/WMQFTE/config/TQM.FTE/TEST_TRANSFER.xml D:\\rvs\\tstusrdat\\ALZtoSIP\\INC\\*.zip
fteCreateMonitor -ma AGENT1 -mn TEST_MONITOR -md D:\\rvs\\tstusrdat\\ALZtoSIP\\INC -mt /var/IBM/WMQFTE/config/TQM.FTE/TEST_TRANSFER.xml -tr match,*.zip
Test was performed on files: 53MB and 30MB
Default configuration (just enableQueueInputOutput=true added to AGENT2.properties)
1) all default
no success, transfer status: "recovering"
for both files
2) added maxInputOutputMessageLength=60000000, destination queue max message length changed to 103809024
result transfer status: "failed" with following exception PM71138: BFGIO0178E: A QUEUE WRITE FAILED DUE TO A WMQAPIEXCEPTION WITH MESSAGE TEXT CC=2 RC=2142 MQRC_HEADER_ERROR
for both files
After reading this: http://pic.dhe.ibm.com/infocenter/wmqfte/v7r0/topic/com.ibm.wmqfte.doc/message_length.htm I came with working settings:
3) maxInputOutputMessageLength=34603008 (its maximum value), destination queue max message length still to 103809024
result for file with size 30MB: succcess
result for file with size 53MB: "failed" with following exception PM71138: BFGIO0178E: A QUEUE WRITE FAILED DUE TO A WMQAPIEXCEPTION WITH MESSAGE TEXT CC=2 RC=2142 MQRC_HEADER_ERROR
So according to this I am afraid one can't transfer larger then 34603008 bytes.
If you are transferring file to queue you definitely can't use the default settings. You have to add "enableQueueInputOutput=true" to agent.properties for agent thet uses queue as source or destination.