Consumed messages from Kafak is printed as numbers in spring-xd - spring
I am trying to make a simple spring-xd experiment.
Spring-xd is downloaded and running in the single node mode. We already have a Kafka cluster and we have a spark streaming job that read tweets from twitter API and push it into Kafka.
I am trying to read from Kafka using spring-xd and write to file so I created the following stream
xd:>stream create tweet --definition "kafka --zkconnect=zkserver:2181 --topic=tweets | file" --deploy
The result in the output file will be something like what appears below
123,34,105,100,34,58,34,116,97,103,58,115,101,97,114,99,104,46,116,119,105,116,116,101,114,46,99,111,109,44,50,48,48,53,58,55,54,54,48,55,55,56,57,52,50,51,55,57,55,48,52,51,50,34,44,34,111,98,106,101,99,116,84,121,112,101,34,58,34,97,99,116,105,118,105,116,121,34,44,34,97,99,116,111,114,34,58,123,34,111,98,106,101,99,116,84,121,112,101,34,58,34,112,101,114,115,111,110,34,44,34,105,100,34,58,34,105,100,58,116,119,105,116,116,101,114,46,99,111,109,58,50,56,49,54,51,54,52,57,52,55,34,44,34,108,105,110,107,34,58,34,104,116,116,112,58,47,47,119,119,119,46,116,119,105,116,116,101,114,46,99,111,109,47,67,82,82,111,103,101,114,115,53,34,44,34,100,105,115,112,108,97,121,78,97,109,101,34,58,34,71,50,32,67,104,97,114,108,105,101,34,44,34,112,111,115,116,101,100,84,105,109,101,34,58,34,50,48,49,52,45,49,48,45,48,56,84,50,49,58,52,54,58,50,56,46,48,48,48,90,34,44,34,105,109,97,103,101,34,58,34,104,116,116,112,115,58,47,47,112,98,115,46,116,119,105,109,103,46,99,111,109,47,112,114,111,102,105,108,101,95,105,109,97,103,101,115,47,55,51,49,50,55,53,53,55,50,56,49,53,53,51,50,48,51,50,47,89,84,66,109,89,113,78,111,95,110,111,114,109,97,108,46,106,112,103,34,44,34,115,117,109,109,97,114,121,34,58,110,117,108,108,44,34,108,105,110,107,115,34,58,91,123,34,104,114,101,102,34,58,110,117,108,108,44,34,114,101,108,34,58,34,109,101,34,125,93,44,34,102,114,105,101,110,100,115,67,111,117,110,116,34,58,55,49,50,44,34,102,111,108,108,111,119,101,114,115,67,111,117,110,116,34,58,52,57,52,44,34,108,105,115,116,101,100,67,111,117,110,116,34,58,54,48,44,34,115,116,97,116,117,115,101,115,67,111,117,110,116,34,58,51,57,53,50,53,44,34,116,119,105,116,116,101,114,84,105,109,101,90,111,110,101,34,58,34,69,97,115,116,101,114,110,32,84,105,109,101,32,40,85,83,32,38,32,67,97,110,97,100,97,41,34,44,34,118,101,114,105,102,105,101,100,34,58,102,97,108,115,101,44,34,117,116,99,79,102,102,115,101,116,34,58,34,45,49,52,52,48,48,34,44,34,112,114,101,102,101,114,114,101,100,85,115,101,114,110,97,109,101,34,58,34,67,82,82,111,103,101,114,115,53,34,44,34,108,97,110,103,117,97,103,101,115,34,58,91,34,101,110,34,93,44,34,108,111,99,97,116,105,111,110,34,58,123,34,111,98,106,101,99,116,84,121,112,101,34,58,34,112,108,97,99,101,34,44,34,100,105,115,112,108,97,121,78,97,109,101,34,58,34,67,84,34,125,44,34,102,97,118,111,114,105,116,101,115,67,111,117,110,116,34,58,51,53,57,125,44,34,118,101,114,98,34,58,34,115,104,97,114,101,34,44,34,112,111,115,116,101,100,84,105,109,101,34,58,34,50,48,49,54,45,48,56,45,49,56,84,48,52,58,48,50,58,53,55,46,48,48,48,90,34,44,34,103,101,110,101,114,97,116,111,114,34,58,123,34,100,105,115,112,108,97,121,78,97,109,101,34,58,34,84,119,105,116,116,101,114,32,102,111,114,32,105,80,104,111,110,101,34,44,34,108,105,110,107,34,58,34,104,116,116,112,58,47,47,116,119,105,116,116,101,114,46,99,111,109,47,100,111,119,110,108,111,97,100,47,105,112,104,111,110,101,34,125,44,34,112,114,111,118,105,100,101,114,34,58,123,34,111,98,106,101,99,116,84,121,112,101,34,58,34,115,101,114,118,105,99,101,34,44,34,100,105,115,112,108,97,121,78,97,109,101,34,58,34,84,119,105,116,116,101,114,34,44,34,108,105,110,107,34,58,34,104,116,116,112,58,47,47,119,119,119,46,116,119,105,116,116,101,114,46,99,111,109,34,125,44,34,108,105,110,107,34,58,34,104,116,116,112,58,47,47,116,119,105,116,116,101,114,46,99,111,109,47,67,82,82,111,103,101,114,115,53,47,115,116,97,116,117,115,101,115,47,55,54,54,48,55,55,56,57,52,50,51,55,57,55,48,52,51,50,34,44,34,98,111,100,121,34,58,34,82,84,32,64,115,116,101,112,104,57,51,48,54,53,58,32,78,101,119,116,58,32,84,114,117,109,112,32,73,83,73,83,32,83,112,101,101,99,104,32,39,72,105,115,116,111,114,105,99,39,32,45,32,39,77,111,115,116,32,73,109,112,111,114,116,97,110,116,32,70,111,114,101,105,103,110,32,80,111,108,105,99,121,32,83,112,101,101,99,104,32,83,105,110,99,101,32,82,111,110,97,108,100,32,82,101,97,103,97,110,39,32,45,32,66,114,101,105,116,98,97,114,116,32,104,116,116,112,115,58,47,47,116,46,99,111,47,50,105,-30,-128,-90,34,44,34,111,98,106,101,99,116,34,58,123,34,105,100,34,58,34,116,97,103,58,115,101,97,114,99,104,46,116,119,105,116,116,101,114,46,99,111,109,44,50,48,48,53,58,55,54,53,53,53,50,56,57,54,51,56,53,52,56,50,55,53,50,34,44,34,111,98,106,101,99,116,84,121,112,101,34,58,34,97,99,116,105,118,105,116,121,34,44,34,97,99,116,111,114,34,58,123,34,111,98,106,101,99,116,84,121,112,101,34,58,34,112,101,114,115,111,110,34,44,34,105,100,34,58,34,105,100,58,116,119,105,116,116,101,114,46,99,111,109,58,50,54,49,55,52,57,55,51,53,34,44,34,108,105,110,107,34,58,34,104,116,116,112,58,47,47,119,119,119,46,116,119,105,116,116,101,114,46,99,111,109,47,115,116,101,112,104,57,51,48,54,53,34,44,34,100,105,115,112,108,97,121,78,97,109,101,34,58,34,83,116,101,112,104,34,44,34,112,111,115,116,101,100,84,105,109,101,34,58,34,50,48,49,49,45,48,51,45,48,54,84,49,57,58,51,54,58,52,52,46,48,48,48,90,34,44,34,105,109,97,103,101,34,58,34,104,116,116,112,115,58,47,47,112,98,115,46,116,119,105,109,103,46,99,111,109,47,112,114,111,102,105,108,101,95,105,109,97,103,101,115,47,55,53,49,53,55,49,50,54,49,50,55,55,52,49,51,51,55,54,47,86,110,114,119,50,75,55,109,95,110,111,114,109,97,108,46,106,112,103,34,44,34,115,117,109,109,97,114,121,34,58,34,65,109,101,114,105,99,97,32,70,105,114,115,116,32,35,84,114,117,109,112,50,48,49,54,32,35,87,111,109,101,110,102,111,114,84,114,117,109,112,32,35,67,65,102,111,114,84,114,117,109,112,34,44,34,108,105,110,107,115,34,58,91,123,34,104,114,101,102,34,58,34,104,116,116,112,115,58,47,47,119,119,119,46,108,121,105,110,103,99,114,111,111,107,101,100,104,105,108,108,97,114,121,46,99,111,109,47,34,44,34,114,101,108,34,58,34,109,101,34,125,93,44,34,102,114,105,101,110,100,115,67,111,117,110,116,34,58,57,54,51,48,53,44,34,102,111,108,108,111,119,101,114,115,67,111,117,110,116,34,58,49,50,55,57,52,55,44,34,108,105,115,116,101,100,67,111,117,110,116,34,58,49,48,53,49,44,34,115,116,97,116,117,115,101,115,67,111,117,110,116,34,58,50,51,51,54,48,51,44,34,116,119,105,116,116,101,114,84,105,109,101,90,111,110,101,34,58,34,65,114,105,122,111,110,97,34,44,34,118,101,114,105,102,105,101,100,34,58,102,97,108,115,101,44,34,117,116,99,79,102,102,115,101,116,34,58,34,45,50,53,50,48,48,34,44,34,112,114,101,102,101,114,114,101,100,85,115,101,114,110,97,109,101,34,58,34,115,116,101,112,104,57,51,48,54,53,34,44,34,108,97,110,103,117,97,103,101,115,34,58,91,34,101,110,34,93,44,34,108,111,99,97,116,105,111,110,34,58,123,34,111,98,106,101,99,116,84,121,112,101,34,58,34,112,108,97,99,101,34,44,34,100,105,115,112,108,97,121,78,97,109,101,34,58,34,67,97,108,105,102,111,114,110,105,97,44,32,85,83,65,34,125,44,34,102,97,118,111,114,105,116,101,115,67,111,117,110,116,34,58,49,55,51,55,52,56,125,44,34,118,101,114,98,34,58,34,112,111,115,116,34,44,34,112,111,115,116,101,100,84,105,109,101,34,58,34,50,48,49,54,45,48,56,45,49,54,84,49,55,58,49,54,58,52,56,46,48,48,48,90,34,44,34,103,101,110,101,114,97,116,111,114,34,58,123,34,100,105,115,112,108,97,121,78,97,109,101,34,58,34,77,111,98,105,108,101,32,87,101,98,34,44,34,108,105,110,107,34,58,34,104,116,116,112,58,47,47,109,111,98,105,108,101,46,116,119,105,116,116,101,114,46,99,111,109,34,125,44,34,112,114,111,118,105,100,101,114,34,58,123,34,111,98,106,101,99,116,84,121,112,101,34,58,34,115,101,114,118,105,99,101,34,44,34,100,105,115,112,108,97,121,78,97,109,101,34,58,34,84,119,105,116,116,101,114,34,44,34,108,105,110,107,34,58,34,104,116,116,112,58,47,47,119,119,119,46,116,119,105,116,116,101,114,46,99,111,109,34,125,44,34,108,105,110,107,34,58,34,104,116,116,112,58,47,47,116,119,105,116,116,101,114,46,99,111,109,47,115,116,101,112,104,57,51,48,54,53,47,115,116,97,116,117,115,101,115,47,55,54,53,53,53,50,56,57,54,51,56,53,52,56,50,55,53,50,34,44,34,98,111,100,121,34,58,34,78,101,119,116,58,32,84,114,117,109,112,32,73,83,73,83,32,83,112,101,101,99,104,32,39,72,105,115,116,111,114,105,99,39,32,45,32,39,77,111,115,116,32,73,109,112,111,114,116,97,110,116,32,70,111,114,101,105,103,110,32,80,111,108,105,99,121,32,83,112,101,101,99,104,32,83,105,110,99,101,32,82,111,110,97,108,100,32,82,101,97,103,97,110,39,32,45,32,66,114,101,105,116,98,97,114,116,32,104,116,116,112,115,58,47,47,116,46,99,111,47,50,105,119,54,56,79,119,56,73,118,34,44,34,111,98,106,101,99,116,34,58,123,34,111,98,106,101,99,116,84,121,112,101,34,58,34,110,111,116,101,34,44,34,105,100,34,58,34,111,98,106,101,99,116,58,115,101,97,114,99,104,46,116,119,105,116,116,101,114,46,99,111,109,44,50,48,48,53,58,55,54,53,53,53,50,56,57,54,51,56,53,52,56,50,55,53,50,34,44,34,115,117,109,109,97,114,121,34,58,34,78,101,119,116,58,32,84,114,117,109,112,32,73,83,73,83,32,83,112,101,101,99,104,32,39,72,105,115,116,111,114,105,99,39,32,45,32,39,77,111,115,116,32,73,109,112,111,114,116,97,110,116,32,70,111,114,101,105,103,110,32,80,111,108,105,99,121,32,83,112,101,101,99,104,32,83,105,110,99,101,32,82,111,110,97,108,100,32,82,101,97,103,97,110,39,32,45,32,66,114,101,105,116,98,97,114,116,32,104,116,116,112,115,58,47,47,116,46,99,111,47,50,105,119,54,56,79,119,56,73,118,34,44,34,108,105,110,107,34,58,34,104,116,116,112,58,47,47,116,119,105,116,116,101,114,46,99,111,109,47,115,116,101,112,104,57,51,48,54,53,47,115,116,97,116,117,115,101,115,47,55,54,53,53,53,50,56,57,54,51,56,53,52,56,50,55,53,50,34,44,34,112,111,115,116,101,100,84,105,109,101,34,58,34,50,48,49,54,45,48,56,45,49,54,84,49,55,58,49,54,58,52,56,46,48,48,48,90,34,125,44,34,102,97,118,111,114,105,116,101,115,67,111,117,110,116,34,58,48,44,34,116,119,105,116,116,101,114,95,101,110,116,105,116,105,101,115,34,58,123,34,104,97,115,104,116,97,103,115,34,58,91,93,44,34,117,115,101,114,95,109,101,110,116,105,111,110,115,34,58,91,93,44,34,115,121,109,98,111,108,115,34,58,91,93,44,34,117,114,108,115,34,58,91,123,34,117,114,108,34,58,34,104,116,116,112,115,58,47,47,116,46,99,111,47,50,105,119,54,56,79,119,56,73,118,34,44,34,101,120,112,97,110,100,101,100,95,117,114,108,34,58,34,104,116,116,112,58,47,47,98,105,116,46,108,121,47,50,98,117,55,48,81,87,34,44,34,100,105,115,112,108,97,121,95,117,114,108,34,58,34,98,105,116,46,108,121,47,50,98,117,55,48,81,87,34,44,34,105,110,100,105,99,101,115,34,58,91,49,48,56,44,49,51,49,93,125,93,125,44,34,116,119,105,116,116,101,114,95,102,105,108,116,101,114,95,108,101,118,101,108,34,58,34,108,111,119,34,44,34,116,119,105,116,116,101,114,95,108,97,110,103,34,58,34,101,110,34,125,44,34,102,97,118,111,114,105,116,101,115,67,111,117,110,116,34,58,48,44,34,116,119,105,116,116,101,114,95,101,110,116,105,116,105,101,115,34,58,123,34,104,97,115,104,116,97,103,115,34,58,91,93,44,34,117,115,101,114,95,109,101,110,116,105,111,110,115,34,58,91,123,34,115,99,114,101,101,110,95,110,97,109,101,34,58,34,115,116,101,112,104,57,51,48,54,53,34,44,34,110,97,109,101,34,58,34,83,116,101,112,104,34,44,34,105,100,34,58,50,54,49,55,52,57,55,51,53,44,34,105,100,95,115,116,114,34,58,34,50,54,49,55,52,57,55,51,53,34,44,34,105,110,100,105,99,101,115,34,58,91,51,44,49,52,93,125,93,44,34,115,121,109,98,111,108,115,34,58,91,93,44,34,117,114,108,115,34,58,91,123,34,117,114,108,34,58,34,104,116,116,112,115,58,47,47,116,46,99,111,47,50,105,119,54,56,79,119,56,73,118,34,44,34,101,120,112,97,110,100,101,100,95,117,114,108,34,58,34,104,116,116,112,58,47,47,98,105,116,46,108,121,47,50,98,117,55,48,81,87,34,44,34,100,105,115,112,108,97,121,95,117,114,108,34,58,34,98,105,116,46,108,121,47,50,98,117,55,48,81,87,34,44,34,105,110,100,105,99,101,115,34,58,91,49,50,52,44,49,52,48,93,125,93,125,44,34,116,119,105,116,116,101,114,95,102,105,108,116,101,114,95,108,101,118,101,108,34,58,34,108,111,119,34,44,34,116,119,105,116,116,101,114,95,108,97,110,103,34,58,34,101,110,34,44,34,114,101,116,119,101,101,116,67,111,117,110,116,34,58,48,44,34,103,110,105,112,34,58,123,34,109,97,116,99,104,105,110,103,95,114,117,108,101,115,34,58,91,123,34,116,97,103,34,58,34,105,115,105,115,69,110,103,108,105,115,104,79,110,108,121,86,56,34,125,44,123,34,116,97,103,34,58,34,73,83,73,83,69,110,103,108,105,115,104,79,110,108,121,118,50,34,125,44,123,34,116,97,103,34,58,34,73,83,73,83,69,110,103,108,105,115,104,118,50,34,125,44,123,34,116,97,103,34,58,34,73,83,73,83,69,110,103,108,105,115,104,79,110,108,121,118,51,34,125,44,123,34,116,97,103,34,58,34,105,115,105,115,69,110,103,108,105,115,104,79,110,108,121,86,55,80,97,114,116,49,34,125,44,123,34,116,97,103,34,58,34,105,115,105,115,69,110,103,108,10
Is there any explanation to this result?
Your output looks like an array of characters. If you take the fist numbers and translate them, you can see this is an array of codes for a JSON String.
123 34 105 100 34 58 34 116 97 103 58 115 101 97 114 99 104 46...
Translates to:
{"id":"tag:search...
I believed this has been answered here:
Spring XD: "tcp" source outputs byte array instead of string? How to output regular text?
Related
YamlDotNet- packet to yaml
Hello I wanted to create program that will help me with converting packet data from game to yaml it should look like this monsters: - map_monster_id: 1678 vnum: 333 map_x: 30 map_y: 165 - map_monster_id: 1679 vnum: 333 map_x: 24 map_y: 157 i have code that is supposed to write those things in database and I want rework so it can write to yaml anyone who should tell me where to start thank you :)
Having trouble creating and saving a text file in shell scriptin
I attempt to save a few lines into an output file called Temporal_regions.txt in my folder, so I can continue with further process. I took a look at some of other post, and they suggested to do this way: echo "some file content" > /path/to/outputfile So my returned output looks like this: 1 3 13579 586 Right-Temporal 72 73 66 54 2 5 24680 587 Left-Temporal 89 44 65 56 3 7 34552 599 Right-Temporal 72 75 66 54 4 8 24451 480 Left-Temporal 68 57 47 66 *All of these lines were individually returned as output using the grep command from another file (call it TR.stats) If I want to store these outputs into the .txt (call it TemperolRegion.txt), I would have to first create the output file right? What would be the command to do this? Then, I can just simply use the above suggested way to store to the .txt file? grep "Left-Temporal" TR.stats > /path/to/TemperolRegion.txt I can't seem to get the commands right.
It sounds like you are missing the directories along the path. In Linux, you have to create the parent directory before you create the file. You can create every directory along the path to the parent by using mkdir with the -p option. mkdir -p /path/to grep "Left-Temporal" TR.stats > /path/to/TemperolRegion.txt Another problem you might have is that you are not in the right directory to TR.stats. In that case you should use the absolute path for that file as well. grep "Left-Temporal" /path/to/TR.stats > /path/to/TemperolRegion.txt
What does the parser expect when throwing a Psych::SyntaxError?
I'm setting up OpenProject on my computer (ubuntu). When I try to create a database with bundle exec rake db:create:all, a syntax error is returned: Psych::SyntaxError: (/home/openproject/config/configuration.yml): did not find expected key while parsing a block mapping at line 105 column 3 And these are lines from 102 to 107. Note: spaces after line number do not belong to the code. 102 # default configuration options for all environments 103 default: 104 # Outgoing emails configuration (see examples above) 105 email_delivery_method: :smtp 106 perform_deliveries: true 107 smtp_address: smtp.gmail.com What characters is the parser expecting? I must also note you that I'm not yet familiar with YAML, ruby or perl (if there are relations to these).
You should post the entire file. I had this same error occur and I was looking around the line numbers that the error thrown. Actually, it was much further down the file. Too many spaces before a key name (5 instead of 4).
Shell script Email bad formatting?
My script is perfectly fine and produce a file. The file is in plain text and is formatted like how (My expect results should look like this.) is formatted. However when I try to send my file to my email the formatting is completly wrong. The line of code I am using to send my email. cat ReportEmail | mail -s 'Report' bob#aol.com The result I am getting on my email. 30129 22.65 253 96187 72.32 294 109525 82.35 295 10235 7.7 105 5906 4.44 106 76096 57.22 251 My expect results should look like this. 30129 22.65 253 96187 72.32 294 109525 82.35 295 10235 7.7 105 5906 4.44 106 76096 57.22 251
Your source file achieves the column alignment by using a combination of tabs and spaces. The width assigned to a tab, however, can vary from program to program. Widths of 4, 5, or 8 spaces, for example, are common. If you want consistent formatting in plain text from one viewer to the next, use only spaces. As a workaround, you can expand the the tabs to spaces before passing the file to mail using the expand utility: expand -t 8 ReportEmail.txt | mail -s 'Report' bob#aol.com The option -t 8 tells expand to treat tabs as 8 spaces wide. Change the 8 to whatever number consistently makes the format in ReportEmail.txt work properly.
Reading a text file in Ruby gives wrong output
I am not an experienced ruby programmer, so bear with me. I have a problem with this specific text file containing two lines ( this issue shows up only on occasions) :
trim(0, 15447)
0, 15447
I am trying to read these two lines with the following code:
File.open(trim).each do |line|
puts line
end
I normally obtain the normal output, but here, I get only one line, with some characters missing:
0, 1544715447)
If I want to check the character codes, I get this:
irb(main):120:0> File.open(trim).each do |line|
irb(main):121:1* puts '========================'
irb(main):122:1> puts line
irb(main):123:1> puts '........................'
irb(main):124:1> puts line.each_byte {|c| print c, ' ' }
irb(main):125:1> end
========================
0, 1544715447)
........................
116 114 105 109 40 48 44 32 49 53 52 52 55 41 13 48 44 32 49 53 52 52 55 trim(0,0, 15447
=> #<File:E:\Public\Public_videos\Soccer\1995_0129_odp_es\950129-ODP_&m3_trim30.txt>
I frankly don't understand what is going on, as I don't see any hidden character, and this happen randomly, but consistently with some files.
Any suggestion to help me understand or avoid this issue would be greatly appreciated.
What happened is that your file had two "lines" separated by a carraige return character, and not a linefeed. You showed the bytes in your file as 116 114 105 109 40 48 44 32 49 53 52 52 55 41 13 48 44 32 49 53 52 52 55 That 13 is a carriage return, which is sometimes "displayed" by the writer going back to the start of the line it is writing. So first it wrote out trim(0, 15447) then it went back to the start of the same line and wrote 0, 15447 overlaying the initial line! What do you end up with? 0, 1544715447) Your "problem" is probably best fixed by reencoding that text file of yours to use a better way to separate lines. On Unix systems, including OSX these days, the line terminator is character 10 - known as LINE FEED. Windows uses the two-character combination 13 10 (CR LF). Only old Mac systems to my knowledge used the 13. Many text editors today will allow you to select a "line ending" option, so you might be able to just open that file, then save it using a different line ending option. FWIW my guess is that you are using Windows now, which is known for rendering CRs and LFs differently than *Nix systems.