I am on a system that does not have hexdump. I know there's a null character on STDIN, but I want to show/prove it. I've got Ruby on the system. I've found that I can directly print it like this:
$ printf 'before\000after' | (ruby -e "stdin_contents = STDIN.read_nonblock(10000) rescue nil; puts 'stdin contents: ' + stdin_contents.inspect")
stdin contents: "before\x00after"
However, I need to run this inside of a bash script i.e. STDIN is not being directly piped to my script. I have to get it via running read in bash.
When I try to use read to get the stdin characters, it seems to be truncating them and it doesn't work:
$ printf 'before\000after' | (read -r -t 1 -n 1000000; printf "$REPLY" | ruby -e "stdin_contents = STDIN.read_nonblock(10000) rescue nil; puts 'stdin contents: ' + stdin_contents.inspect")
stdin contents: "before"
My question is this: How can I get the full/raw output including the null character from read
Using Groovy and it's java.lang.Process support, how do I pipe multiple shell commands together?
Consider this bash command (and assume your username is foo):
ps aux | grep ' foo' | awk '{print $1}'
This will print out usernames - one line for some processes related to your user account.
Using Groovy, the ProcessGroovyMethods documentation and code says I should be able to do this to achieve the same result:
def p = "ps aux".execute() | "grep ' foo'".execute() | "awk '{print $1}'".execute()
p.waitFor()
println p.text
However, I can't get any text output for anything other than this:
def p = "ps aux".execute()
p.waitFor()
println p.text
As soon as I start piping, the println does not print out any anything.
Thoughts?
This works for me :
def p = 'ps aux'.execute() | 'grep foo'.execute() | ['awk', '{ print $1 }'].execute()
p.waitFor()
println p.text
for an unknown reason, the parameters of awk can't be send with only one string (i don't know why! maybe bash is quoting something differently). If you dump with your command the error stream, you'll see error relative to the compilation of the awk script.
Edit : In fact,
"-string-".execute() delegate to Runtime.getRuntime().exec(-string-)
It's bash job to handle arguments containing spaces with ' or ". Runtime.exec or the OS are not aware of the quotes
Executing "grep ' foo'".execute() execute the command grep, with ' as the first parameters, and foo' as the second one : it's not valid. the same for awk
You can do this to just let the shell sort it out:
// slash string at the end so we don't need to escape ' or $
def p = ['/bin/bash', '-c', /ps aux | grep ' foo' | awk '{print $1}'/].execute()
p.waitFor()
println p.text
This has worked for me
def command = '''
ps aux | grep bash | awk '{print $1}'
'''
def proc = ['bash', '-c', command].execute()
proc.waitFor()
println proc.text
If you want to run multiple commands, you can add it in the command.
def command = '''
ls -ltr
cat secret
'''
def proc = ['bash', '-c', command].execute()
proc.waitFor()
println proc.text
If you want it async I recommend
proc.consumeProcessOutputStream(new LineOrientedOutputStream() {
#Override
protected void processLine(String line) throws IOException {
println line
}
}
);
Using techniques mentioned here (Pipe output to two different commands) we can split a stdout into multiple processes.
expensive_command | tee >(proc_1) >(proc_2) | proc_3
my problem is this interlaces the output.
Is there a way to copy the stdout but force proc_2 to block until proc_1 finishes?
I'm thinking something like
expensive_command | tee >(proc_1) | wait for EOF | tee >(proc_2) ...
You can use a fifo as a cheap lock. Have proc1 write to it after it completes, and wait until a read from the fifo succeeds before running proc2.
mkfifo cheap_lock
expensive_command | tee >(proc1; echo foo > cheap_lock) \
>(read < cheap_lock; proc2 ) | proc3
(Of course, it's your responsibility to ensure that no other processes try to read from or write to cheap_lock.)
You can create a buffer holder that would release the output once data from input reaches eof like
expensive_command | awk '{ a[i++] = $0 }END{for (i = 0; i in a; ++i) { print a[i] | "tee temp.txt" } }'
Only that awk does not support process substitution.
In bash you can do:
readarray -t lines <(expressive_command | tee >(proc_1))
printf '%s\n' "${lines[#]}" | tee >(proc_2)
Depending on the peak data size of your output from expressive_command or version of your Bash, the command may require adjustments. You can also consider using another language.
Add: You can also use stdbuf. It runs command with modified buffering operations for its standard streams.
How do I get the full command line in ruby?
$ rails c
> $0
=> "script/rails"
> ARGV
[]
> `ps -eo "%p|$|%a" | grep '^\\s*#{Process.pid}'`.strip.split("|$|")[1]
=> "/home/sam/.rvm/rubies/ruby-1.9.3-p194-perf/bin/ruby script/rails console"
Is there anything cleaner than ninja ps I can do to get the same results?
To clarify, in case there is confusion, I want the exact same output as:
`ps -eo "%p|$|%a" | grep '^\\s*#{Process.pid}'`.strip.split("|$|")[1]
ARGV is coming back blank.
$0 is missing the full path.
I'd use:
#!/usr/bin/env ruby
puts "Process ID: #{ $$ }"
puts `ps axw`.split("\n").select{ |ps| ps[ /\A#{ $$ }/ ] }
Running that inside a script outputs:
18222 s000 S+ 0:00.25 /Users/foo/.rbenv/versions/1.9.3-p385/bin/ruby /Users/foo/.rbenv/versions/1.9.3-p385/bin/rdebug /Users/foo/Desktop/test.rb
How to I concatenate stdin to a string, like this?
echo "input" | COMMAND "string"
and get
inputstring
A bit hacky, but this might be the shortest way to do what you asked in the question (use a pipe to accept stdout from echo "input" as stdin to another process / command:
echo "input" | awk '{print $1"string"}'
Output:
inputstring
What task are you exactly trying to accomplish? More context can get you more direction on a better solution.
Update - responding to comment:
#NoamRoss
The more idiomatic way of doing what you want is then:
echo 'http://dx.doi.org/'"$(pbpaste)"
The $(...) syntax is called command substitution. In short, it executes the commands enclosed in a new subshell, and substitutes the its stdout output to where the $(...) was invoked in the parent shell. So you would get, in effect:
echo 'http://dx.doi.org/'"rsif.2012.0125"
use cat - to read from stdin, and put it in $() to throw away the trailing newline
echo input | COMMAND "$(cat -)string"
However why don't you drop the pipe and grab the output of the left side in a command substitution:
COMMAND "$(echo input)string"
I'm often using pipes, so this tends to be an easy way to prefix and suffix stdin:
echo -n "my standard in" | cat <(echo -n "prefix... ") - <(echo " ...suffix")
prefix... my standard in ...suffix
There are some ways of accomplish this, i personally think the best is:
echo input | while read line; do echo $line string; done
Another can be by substituting "$" (end of line character) with "string" in a sed command:
echo input | sed "s/$/ string/g"
Why i prefer the former? Because it concatenates a string to stdin instantly, for example with the following command:
(echo input_one ;sleep 5; echo input_two ) | while read line; do echo $line string; done
you get immediatly the first output:
input_one string
and then after 5 seconds you get the other echo:
input_two string
On the other hand using "sed" first it performs all the content of the parenthesis and then it gives it to "sed", so the command
(echo input_one ;sleep 5; echo input_two ) | sed "s/$/ string/g"
will output both the lines
input_one string
input_two string
after 5 seconds.
This can be very useful in cases you are performing calls to functions which takes a long time to complete and want to be continuously updated about the output of the function.
You can do it with sed:
seq 5 | sed '$a\6'
seq 5 | sed '$ s/.*/\0 6/'
In your example:
echo input | sed 's/.*/\0string/'
I know this is a few years late, but you can accomplish this with the xargs -J option:
echo "input" | xargs -J "%" echo "%" "string"
And since it is xargs, you can do this on multiple lines of a file at once. If the file 'names' has three lines, like:
Adam
Bob
Charlie
You could do:
cat names | xargs -n 1 -J "%" echo "I like" "%" "because he is nice"
Also works:
seq -w 0 100 | xargs -I {} echo "string "{}
Will generate strings like:
string 000
string 001
string 002
string 003
string 004
...
The command you posted would take the string "input" use it as COMMAND's stdin stream, which would not produce the results you are looking for unless COMMAND first printed out the contents of its stdin and then printed out its command line arguments.
It seems like what you want to do is more close to command substitution.
http://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html#Command-Substitution
With command substitution you can have a commandline like this:
echo input `COMMAND "string"`
This will first evaluate COMMAND with "string" as input, and then expand the results of that commands execution onto a line, replacing what's between the ‘`’ characters.
cat will be my choice: ls | cat - <(echo new line)
With perl
echo "input" | perl -ne 'print "prefix $_"'
Output:
prefix input
A solution using sd (basically a modern sed; much easier to use IMO):
# replace '$' (end of string marker) with 'Ipsum'
# the `e` flag disables multi-line matching (treats all lines as one)
$ echo "Lorem" | sd --flags e '$' 'Ipsum'
Lorem
Ipsum#no new line here
You might observe that Ipsum appears on a new line, and the output is missing a \n. The reason is echo's output ends in a \n, and you didn't tell sd to add a new \n. sd is technically correct because it's doing exactly what you are asking it to do and nothing else.
However this may not be what you want, so instead you can do this:
# replace '\n$' (new line, immediately followed by end of string) by 'Ipsum\n'
# don't forget to re-add the `\n` that you removed (if you want it)
$ echo "Lorem" | sd --flags e '\n$' 'Ipsum\n'
LoremIpsum
If you have a multi-line string, but you want to append to the end of each individual line:
$ ls
foo bar baz
$ ls | sd '\n' '/file\n'
bar/file
baz/file
foo/file
I want to prepend my sql script with "set" statement before running it.
So I echo the "set" instruction, then pipe it to cat. Command cat takes two parameters : STDIN marked as "-" and my sql file, cat joins both of them to one output. Next I pass the result to mysql command to run it as a script.
echo "set #ZERO_PRODUCTS_DISPLAY='$ZERO_PRODUCTS_DISPLAY';" | cat - sql/test_parameter.sql | mysql
p.s. mysql login and password stored in .my.cnf file