I have a stream "myStream" of text records of different types, say "A", "B", "C", and so on...Each type of records needs to be processed differently. I want to create multiple taps on "myStream" for each type: tapA, tapB, tapC, etc. But it seems tap doesn't have any filtering capability. Is that true or am I missing something? Is there a better way to accomplish this?
You can have filter on a tap stream. You can see tap as a regular stream that has source channel (tap channel) set and this source channel can be combined with multiple processors and a single sink.
See this example:
xd:>stream create test --definition "http | log" --deploy
xd:>http post --data test
xd:>stream create tap-test --definition "tap:stream:test.http > filter --expression=payload.contains('a') | log" --deploy
xd:>http post --data test
xd:>http post --data apple
and in the container log you would see something like this:
2015-05-19 11:48:36,276 1.2.0.SNAP INFO pool-16-thread-11 sink.test - test
2015-05-19 11:48:41,445 1.2.0.SNAP INFO pool-16-thread-17 sink.test - apple
2015-05-19 11:48:41,445 1.2.0.SNAP INFO xd.localbus-2 sink.tap-test - apple
Note the payload 'apple' comes out of regular and tap stream.
Related
I want to automate a personal workflow that is based on camera usage on my MBP.
Basically I want to know if any of the cameras (built-in or USB) has been turned on or off, so I can run a program or script I'll create.
I think it's OK if I need to poll for the cameras statuses but an event or callback based solution would be ideal
This seems to work.
❯ log stream | grep "Post event kCameraStream"
2020-12-01 14:58:53.137796-0500 0xXXXXXX Default 0x0 XXX 0 VDCAssistant: [com.apple.VDCAssistant:device] [guid:0xXXXXXXXXXXXXXXXX] Post event kCameraStreamStart
2020-12-01 14:58:56.431147-0500 0xXXXXXX Default 0x0 XXX 0 VDCAssistant: [com.apple.VDCAssistant:device] [guid:0xXXXXXXXXXXXXXXXX] Post event kCameraStreamStop
2020-12-01 14:58:56.668970-0500 0xXXXXXX Default 0x0 XXX 0 VDCAssistant: [com.apple.VDCAssistant:device] [guid:0xXXXXXXXXXXXXXXXX] Post event kCameraStreamStart
Some of the numbers in the output are redacted with Xs because I don't know what they mean. :)
log stream --predicate 'eventMessage contains "Post event kCameraStream"' works up to macOS Big Sur, but not in macOS Monterey. You'll have to use a slightly different predicate:
$ log stream --predicate 'subsystem contains "com.apple.UVCExtension" and composedMessage contains "Post PowerLog"'
Filtering the log data using "subsystem CONTAINS "com.apple.UVCExtension" AND composedMessage CONTAINS "Post PowerLog""
Timestamp Thread Type Activity PID TTL
2021-10-27 12:21:13.366628+0200 0x147c5 Default 0x0 353 0 UVCAssistant: (UVCExtension) [com.apple.UVCExtension:device] UVCExtensionDevice:0x1234005d7 [0x7fe3ce008ca0] Post PowerLog {
"VDCAssistant_Device_GUID" = "00000000-1432-0000-1234-000022470000";
"VDCAssistant_Power_State" = On;
}
2021-10-27 12:21:16.946379+0200 0x13dac Default 0x0 353 0 UVCAssistant: (UVCExtension) [com.apple.UVCExtension:device] UVCExtensionDevice:0x1234005d7 [0x7fe3ce008ca0] Post PowerLog {
"VDCAssistant_Device_GUID" = "00000000-1432-0000-1234-000022470000";
"VDCAssistant_Power_State" = Off;
}
As far as I know, you can poll for the camera usage with:
$ lsof -n | grep "AppleCamera"
or change "AppleCamera" with the driver name of an external camera.
Other relevant names to try are: "USBVDC" or "VDCAssistant" or "FaceTime" (or "iSight" in older Macs).
You should get one line with the name and pid of the process using the webcam or nothing, which means that it is not in use.
You could check for all of the keywords and decide that the camera is in use if any of these keywords give you something back.
The -n option is to skip resolving DNS names of IP connections and this speeds the command a lot.
As a side note, I use this app to know when any app is using the microphone and/or webcam: OverSight
In macOS Ventura I find this incantation works:
log stream --predicate 'sender contains "appleh13camerad" and (composedMessage contains "PowerOnCamera" or composedMessage contains "PowerOffCamera")'
I created a stream named vfs-od-fe-hb，where the source is vfs, the processors are od and fe, and the sink is hb. I want to create a tap in the postion of fe and store the result into hdfs.
My shell command is:
stream create --name vfs-od-fe-hb-tap-ib-hdfs --definition "tap:stream:vfs-od-fe-hb.fe > ib > hdfs" --deploy
ib is my processor, it can handle the messages from fe successfully, but now the wrong messages:
Command failed org.springframework.xd.rest.client.impl.SpringXDException: XD133E:(pos 34): Expected channel prefix of 'queue' or 'topic' but found 'hdfs'
If ib is a processor, the syntax should be
tap:stream:vfs-od-fe-hb.fe > ib | hdfs
i.e. the tap (named channel) is ib's input channel and we pipe its output as normal.
The parser is reacting to the > hdfs because > can only be between a named channel and processor or sink.
(or after a processor when sending to a named channel - queue:..., topic:...).
I created a twitter stream to extract a tweet info that I created, the stream is below:
xd:>stream create --name twitter2 --definition "twittersearch --consumerSecret=xxx --consumerKey=xxx --query='#SpringXDisSunnyThisEvening' | file" --deploy
The stream worked fine, I ended up with a file.out "JSON" which contains my tweet info.
I tried to add a filer to my stream like the below:
xd:>stream create --name twitter2 --definition "twittersearch --consumerSecret=xxx --consumerKey=xxx --query='#SpringXDisSunnyThisEvening' | filter --expression=#jsonPath(payload,'$.name').contains('moha') | file" --deploy
It did not work, no output file.
Log:
015-07-04T16:32:48-0500 1.2.0.RELEASE ERROR inbound.twitter2.0-redis:queue-inbound-channel-adapter1 redis.RedisMessageBus$1 - Failed to deliver message; retries exhausted; message sent to queue 'ERRORS:twitter2.0' org.springframework.messaging.MessageHandlingException: Expression evaluation failed: #jsonPath(payload,'$.name').contains('mohammad'); nested exception is java.lang.reflect.InvocationTargetException
I think you need to use this expression
--expression=#jsonPath(payload,'$.user.screen_name').contains('something')
Note it is user.screen_name instead of just name from the twitter response.
The goal is to read data emitted over the network.
On the data generation side I have an app which spews to stdout. The content of this data is a JSON string.
Here's what I'm doing (on Linux Mint 17, using an BSD flavored netcat):
data generation:
my_app_which_outputs_json | netcat localhost 9999
In SpringXD: (with xd-singlenode)
xd:>stream create --name tcptest --definition "tcp --decoder=LF --port=9999 | file " --deploy
Created and deployed new stream 'tcptest'
Output:
/tmp/xd/output$ cat tcptest.out
82,117,110, ... (etc, lots more bytes)
I'm sure this is user error, but not sure what to change to make it right.
I should note that if I do this, it works as expected:
my_app_which_outputs_json > /tmp/somefile.txt
...
xd:>stream create --name filetest --definition "tail --name=/tmp/somefile.txt | file" --deploy
Following up to your own answer, the converters are currently not configured to understand that a byte[] can be converted to a String with content type application/json.
Of course, even if we configured it to do so, it wouldn't be able to determine if the content really is JSON.
I added the following to my stream defn and it is now doing what I expected. I found this in the "Type Conversion" section of the documentation.
stream create --name tcptest \
--definition "tcp --decoder=LF --port=9999 --outputType=text/plain \
| file " --deploy
(The newlines and backwhacks are not in my actual code, but are used for readability.)
I also tried ... --outputType=application/json... which did not work. Not entirely sure why.
text/plain with a byte[] payload activates a ByteArrayToStringMessageConverter. application/json does not currently. I have created https://jira.spring.io/browse/XD-2512 to address this
I have a question is how to create stream with multi processor. For ex:
stream create --name multiModuleTest --definition "tcp | processor1 | processor2 |file" --deploy
Or
stream create --name multiModuleTest --definition "(tcp, http) | processor1 | processor2 | (file,log)" --deploy
Please tell me how to do that? Thanks !
Fire up multiple containers and use the deployment manifest to set the module count.
If you want to multiprocess within a processor instance, change the input channel to an ExecectorChannel. Docs here and here.