So I have some ruby code that loops putting strings to stdout using puts then sleeps using sleep. I then have some node.js code that listens on stdin for data events and simply logs what it gets from stdin.
If I run echo 'something' | node my_code.js I'll see something, but if I run ruby my_code.rb | node my_code.js I don't see anything.
Am I not able to redirect the stdout from the ruby code to stdin of the node.js code using a UNIX pipe?
There should be very little difference between the two, and the Ruby code should be fine.
However, you are spotting some problems. What happens if you run the Ruby through tee?
ruby my_code.rb | tee file
Do you see the output? If not, start investigating your Ruby code. (Does it work when you run it without piping its output?). If you do see the output as you expect, does the Ruby program stop (exit)? Do you get your command line prompt back?
If there's nothing anomalous with the Ruby, what happens with the JavaScript when you pipe a multiline file to it:
cat my_code.rb | node my_code.js
I expect one of these scenarios to provide you with something to chase.
Try this:
ruby my_code.rb | awk '{print;fflush()}' | node my_code.js
Or this:
ruby my_code.rb | grep --line-buffer '.*' | node my_code.js
Related
I began with playing ctfs challenges, and I encountered a problem where I needed to send an exploit into a binary and then interact with the spawned shell.
I found a solution to this problem which looks something like this:
(echo -ne "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\xbe\xba\xfe\xca" && cat) | nc pwnable.kr 9000
Meaning:
without the "cat" sub-command, I couldn't interact with the shell, but with it, i now able to send commands into the spawned shell and get the returned output to my console stdout.
What exactly happens there? this command line confuses me
If you just type in cat at the command line, you'll be able to see that this command simply copies stdin to stdout one line at a time. It will carry on doing this until you either quit with Ctrl-C or send an EOF with Ctrl-D.
In this example you're running cat immediately after successfully printing the payload (the concatenator && tells the shell to run the second command only if the first command has an exit code of zero; i.e., no error). As a result, the remote terminal won't see an EOF until you terminate it as described above. When this is piped to nc, everything you type in is sent via cat to the remote server, and everything it sends back appears on your stdout.
So yes, in effect you end up with an interactive shell. You can get pretty much the same effect on your own machine by running cat | sh.
I like the idea of using systemd's journal to view and manage the logs of my own scripts. I have become aware you can log to journal from my user scripts on a per message basis..
echo 'hello' | systemd-cat -t myscript -p emerg
Is there a way to redirect all messages to journald, even those generated by other commands? Like..
exec &> systemd-cat
Update:
Some partial success.
Tried Inian's suggestion from terminal.
~/scripts/myscript.sh 2>&1 | systemd-cat -t myscript.sh
and it worked, stdout and stderr were directed to systemd's journal.
Curiously,
~/scripts/myscript.sh &> | systemd-cat -t myscript.sh
didn't work in my Bash terminal.
I still need to find a way to do this inside my script for when other programs call my script.
I tried..
exec 2>&1 | systemd-cat -t myscript.sh
but it doesn't work.
Update 2:
From terminal
systemd-cat ~/scripts/myscript.sh
works. But I'm still looking for a way to do this from within the script.
A pipe to systemd-cat is a process which needs to run concurrently with your script. Bash offers a facility for this, though it's not portable to POSIX sh.
exec > >(systemd-cat -t myscript -p emerg) 2>&1
The >(command) process substitution starts another process and returns a pseudo-filename (something like /dev/fd/63) which you can redirect into. This is basically a wrapper for the mkfifo hacks you could do if you wanted to port this to POSIX sh.
If your script happens to not be a shell script, but some other programming language that allows loading extension modules linked to -lsystemd, there is another way. There is a library function sd_journal_stream_fd that quite precisely matches the task at hand. Calling it from bash itself (as opposed to some child) seems difficult at best. In Python for instance, it is available as systemd.journal.stream. What this function does in essence is connecting a unix domain stream socket and communicating what kind of data is being transmitted (e.g. priority). The difficult part with a shell here is making it connect a unix domain socket (as opposed to connecting in a child).
The key idea to this answer was given by Freenode/libera.chat user grawity.
Apparently, and for reasons that are beyond me, you can't really redirect all stdout and stderr to journald from within a script because it has to be piped in. To work around that I found a trick people were using with syslog's logger which works similarly.
You can wrap all your code into a function and then pipe the function into systemd-cat.
#!/bin/bash
mycode(){
echo "hello world"
echor "echo typo producing error"
}
mycode | systemd-cat -t myscript.sh
exit 0
And then to search journal logs..
journalctl -t myscript.sh --since yesterday
I'm disappointed there isn't a more direct way of doing this.
I am using following lines of code in my ruby program on ubuntu:
data=ARGF.read
if data.length != 0
.....
end
The program runs fine when I run as "cat file.txt | ruby test.rb", however, I am unable to handle following issues:
When run as "cat | ruby test.rb", the program goes into endless loop.
When run as "ruby test.rb", the program goes into endless loop.
When run as "cat file1.txt | ruby test.rb", the program gives "cat: file1.txt: No such file or directory" error.
Any input will be highly appreciated.
I think you misunderstand what ARGF is used for. ARGF.read gives all the data of all the files passed as arguments.
When you don't give any input file, it is waiting for you to give the input through stdin. Since, you are in Ubuntu, you could just press (Control + D) which would end the stream and then you could process the data normally.
I tried to use herestrings to automate a script input and hence wanted to quit "more" of a file in it.
I've used:
$ echo q | more big_file.txt
$ more big_file.txt <<< q
$ yes q | more big_file.txt
$ echo -e "q\n" | more build.txt
but "more" command fails to quit.
Generally, the above mentioned input methods work for other commands in bash, but more seems to be exceptional,
Any idea what makes this a foul attempt?
NOTE: I don't want the data out of "more", but to quit "more" through an automated sequence is the target
When it detects it's running on a terminal, more only takes its input from it. This is so you can run things like:
$ cat multiple_files*.txt | more
(When it's not on a terminal, it doesn't even page, it degrades to behaving like cat.)
Seriously though, whatever your misguided reason for wanting to keep more in the loop is, you're not going to have it quit voluntarily with anything else than terminal input. So:
either you fake it. E.g. with expect
or you have it quit nonvoluntary. With a signal. Try SIGQUIT, move to SIGTERM, fall down to SIGKILL as a last resort.
Pretty new to Jenkins and I have simple yet annoying problem. When I run job (Build) on Jenkins I am triggering ruby command to execute my test script.
Problem is Jenkins is not displaying output in real time from console. Here is trigger log.
Building in workspace /var/lib/jenkins/workspace/foo_bar
No emails were triggered.
[foo_bar] $ /bin/sh -xe /tmp/hudson4042436272524123595.sh
+ ruby /var/lib/jenkins/test-script.rb
Basically it hangs on this output until build is complete than it just shows full output. Funny thing is this is not consistent behavior, sometimes it works as it should. But most of the time there is no real time console output.
Jenkins version: 1.461
To clarify some of the answers.
ruby or python or any sensible scripting language will buffer the output; this is in order to minimize the IO; writing to disk is slow, writing to a console is slow...
usually the data gets flush()'ed automatically after you have enough data in the buffer with special handling for newlines. e.g. writing a string without newline then sleep() would not write anything until after the sleep() is complete (I'm only using sleep as an example, feel free to substitute with any other expensive system call).
e.g. this would wait 8 seconds, print one line, wait 5 more seconds, print a second line.
from time import sleep
def test():
print "ok",
time.sleep(3)
print "now",
time.sleep(5)
print "done"
time.sleep(5)
print "again"
test()
for ruby, STDOUT.sync = true, turns the autoflush on; all writes to STDOUT are followed by flush(). This would solve your problem but result in more IO.
STDOUT.sync = true
for python, you can use python -u or the environment variable PYTHONUNBUFFERED to make stdin/stdout/stout not buffered, but there are other solutions that do not change stdin or stderr
export PYTHONUNBUFFERED=1
for perl, you have autoflush
autoflush STDOUT 1;
Make sure your script is flushing its stdout and stderr.
In my case I had a buffering issue similar to what you describe but I was using python.
The following python code fixed it for me:
import sys
sys.stdout.flush()
I'm not a Ruby coder, but Google reveals the following:
$stdout.flush
It seems to me that python -u works as well.
E.g. In batch command
python -u foo.py
Easiest solution here is to turn on syncing buffer to output. Something that #Craig wrote about in his answer but one line solution that will cover whole script, and not require you to flush buffer many times.
Just write
STDOUT.sync = true
Logic behind is simple, to avoid using IO operations many times output is buffered. To disable this use
STDOUT.sync = false
This is Ruby solution ofc.
Each of the other answers is specific to one program or another, but I found a more general solution here:
https://unix.stackexchange.com/a/25378
You can use stdbuf to alter the buffering behavior of any program.
In my case, I was piping output from a shell script through tee and grep to split lines into either the console or a file based on content. The console was hanging as described by OP. This solved it:
./slowly_parse.py login.csv |tee >(grep -v LOG: > out.csv) | stdbuf -oL -eL grep LOG:
Eventually I discovered I could just pass --line-buffered to grep for the same result:
./slowly_parse.py login.csv |tee >(grep -v LOG: > out.csv) | grep --line-buffered LOG:
The other answers are correct in saying that you need to ensure standard output is not buffered.
The other thing to be aware of is that Jenkins itself does line by line buffering. If you have a slow-running process that emits single characters (for example, an nunit test suite summary that prints a . for a successful test and an E for an error) you will not see anything until the end of line.
[True for my Jenkins 1.572 running on a Windows box.]
For some commands, including tee a the best choice for unbuffering is a program called unbuffer from expect package.
Usage example:
instead of
somecommand | tee /some/path
do
somecommand | unbuffer -p tee /some/path
Sources and more info:
https://stackoverflow.com/a/11337310/2693875
https://unix.stackexchange.com/a/25375/53245
The Operating-System is buffering output-data by nature, to save CPU, and so does Jenkins.
Looks like you are using a shell-command to run your Ruby script -
I suggest running your Ruby script directly via the dedicated plugin:
Jenkins Ruby Plugin
(may need to install it)
Python buffered its output traces and print it at the end of script to minimize writing on console as writing to console is slow.
You can use following command after your traces. It will flush all traces to console, which are queued before that command.
sys.stdout.flush()