Piping the output of a program that doesn't write to stdout - bash

I'm looking for the best way to use pipes when the input program doesn't write to stdout. Specifically I want to pipe objcopy into hexdump like this
objcopy -I ifmt -O binary ifile - | hexdump -C
but objcopy doesn't accept '-' as a file meaning 'write to stdout' as some programs do.
At the moment I am doing
objcopy -I ifmt -O binary ifile tmpfile; hexdump -C tmpfile; rm tmpfile
but was wondering if there was a better way.
I am using bash 4.1.10 on cygwin.

I wrote a comment prescribing a process substitution, but it can't work for objcopy since objcopy will try to open a seekable file (as it may need to move back and forth in the file).
In short: objcopy can't write to a stream as stdout, that's why its output must be a file that can be seeked. Your solution is very likely the only reasonable possibility.
To answer your question
I'm looking for the best way to use pipes when the input program doesn't write to stdout
in a more general fashion (but that's not applicable with objcopy or any command that requires seeking the file), in Bash you can use process substitution: if mycommand takes a parameter that is an output file, and doesn't accept - for standard output and doesn't write to standard output by default, you can use it as:
mycommand >(cat)
or if you want to pipe it through, e.g., hexdump -C:
mycommand >(hexdump -C)
With this, mycommand will see an argument of the form /dev/fd/42 (where 42 may differ), and will be able to open it for writing, as if it where a regular file (but not seekable), and hexdump will get on its standard input the written data.
You can experiment process substitution like so: call the following script mycommand:
#!/bin/bash
if [[ $1 ]]; then
echo "Hi, this is mycommand, and I was called with first argument: \`$1'"
echo "I'm outputting this to the file given as argument" > "$1"
else
echo >&2 "Please provide an argument (file to write to)"
exit 1
fi
This script makes sure that you give a non-empty argument (otherwise shows an error message), outputs this argument to standard output, and a little line in the file the name of which is given as argument.
Then chmod +x mycommand and play with it:
$ ./mycommand
Please provide an argument (file to write to)
$ ./mycommand -
Hi, this is mycommand, and I was called with first argument: `-'
$ ls
- mycommand
$ rm ./-
$ ./mycommand >(cat)
Hi, this is mycommand, and I was called with first argument: `/dev/fd/63'
I'm outputting this to the file given as argument
$ ./mycommand >(tr -d e)
Hi, this is mycommand, and I was called with first argument: `/dev/fd/63'
I'm outputting this to th fil givn as argumnt
$ ./mycommand >(hexdump -C)
Hi, this is mycommand, and I was called with first argument: `/dev/fd/63'
00000000 49 27 6d 20 6f 75 74 70 75 74 74 69 6e 67 20 74 |I'm outputting t|
00000010 68 69 73 20 74 6f 20 74 68 65 20 66 69 6c 65 20 |his to the file |
00000020 67 69 76 65 6e 20 61 73 20 61 72 67 75 6d 65 6e |given as argumen|
00000030 74 0a |t.|
00000032
$ ./mycommand >(cat) > /dev/null
I'm outputting this to the file given as argument

Actually there is a way if you use zsh
(){objcopy -O binary -j .text <input file> $1; cat $1} =(touch -c .) | hexdump -C
Explanation
=() is zsh process substitution, which supports the seeking property needed.
$1 inside the function body referes to the file created by =(touch -c .).
Note that the cmd "touch -c ."
-c is ---no-create so the command itself does nothing, we are simply using it for the side effect =() has, which is a temporary file zsh creates containing the output of touch -c . (which is nothing) and deletes once the command is finished.
(){} is anonymous function syntax which keeps the file created by =() inside the function body around long enough to be cat'ed to stdout.

Related

Why does only one output of the 'cat' command come with a linebreak?

Since my reputation is too low to post an image I will reproduce the terminal
output where my question originated from:
username#computer:/run$ cat rsyslogd.pid
599username#computer:/run$ cat acpid.pid
636
username#computer:/run$
cat acpid.pid
comes with a linebreak whereas
cat rsyslog.pid
doesn't.
But if I open both files there is no visible difference (e.g. the file
acpid.pid
doesn't have an additional blank line)
The Question is: Why does one .pid file come with a linebreak and the other one doesn't?
Addditional Information: My operating system is Ubuntu 18.04.3
The rsyslogd.pid file probably doesn't end with a newline character (ASCII 0x0A).
You didn't mention how you opened the files, but, I suspect you used a text editor which will not display non-printable characters (like newline and backspace). Rather than using a text editor try looking at the raw file with the hexdump tool. Then compare the hex values against an ASCII table. I think you will find that the non-printable characters after the 599 and 636 are different.
hexdump -C rsyslogd.pid
hexdump -C acpid.pid
The following sequence of commands reproduces your output. The key is to use the -n flag for the echo command to create a file without a newline character at the end.
$ echo -n test > file_no_new_line.txt
$ echo test > file_with_new_line.txt
$ cat file_no_new_line.txt
test$ cat file_with_new_line.txt
test
$
Here is the output of hexdump for the two files shown in my example.
$ hexdump -C file_no_new_line.txt
00000000 74 65 73 74 |test|
00000004
$ hexdump -C file_with_new_line.txt
00000000 74 65 73 74 0a |test.|
00000005
$
The command output, in this case from cat, and the shell prompt ($) running into each other is also shell dependent. If the behavior can't be reproduce with the steps above try another shell (e.g. /bin/sh)

How do I scan for part of a character?

I’m using bash shell with Mac El Capitan. How can I scan for part of an 8-byte character in a series of files? I got an error when building a project that read
Incorrect string value: '\xF3\x95\x90\x8D\xEA\x93...' for column 'CODE' at row 1
and I’d like to figure out where this string value is coming into play. Unfortunately the error does not give me more information but I know the directory of all the potential files where this could live.
I have corrupted one of my shell files on purpose in the current directory, inserting a 0xf3 char using an hex editor.
I've written this (clumsy) script which uses od (octal dump) in hex mode char-by-char, with hex offset, and greps for the infamous f3 char in the current directory and in all directories below, filtering on files and name (so you can remove the name filtering it still works)
find . -type f -name "*.sh" | while read f
do
line=$(od -Ax -t x1 $f | grep -w f3)
if [ $? = 0 ] ; then
echo file $f is corrupt: $line
fi
done
result on my directory:
file ./quote.sh is corrupt: 000010 69 6d 61 6c f3 3d 24 28 6d 79 73 71 6c 20 2d 75

How to use `ps` to find a command line?

I want to find top CPU usage pid, my script is here:
#!/bin/sh
ppid=`top -n 1 |sed -n 8p |awk '{print $1}'`
echo $ppid
ps aux|grep $ppid
but I get an error:
grep: Unmatched [ or [^
Why? How can I fix it?
OK! I found it! The problem is that top is including terminal control sequences in its output. So you don't actually see it in the echo, but I noticed because it had put my terminal session into bold output and later I was trying to figure out how it happened and I traced it back to my testing for this question. So, the [ that grep was complaining about is in the escape sequence that ppid gets set to. I got a work around by adding |tr -dc 0-9 after the awk, i.e.
ppid=`top -n 1 |sed -n 8p |awk '{print $1}'|tr -dc 0-9`
That will delete anything that's not a digit at the end. But the sed needs to be adjusted, too, I think. And, I suspect some of the digits may be from the escape sequence, so you need to come up with a cleaner way to excise the escape sequence.
But, in the final analysis this will be pretty useless. The highest user of the CPU every time I ran that pipeline was the top process that's part of it. In retrospect that's probably not surprising.
When all else fails, examine the input:
$ top -n 1 | awk 'NR==8 {print $1 ": " $2}' | hexdump -C
00000000 1b 28 42 1b 5b 6d 1b 5b 31 6d 31 38 37 31 35 3a |.(B.[m.[1m18715:|
00000010 20 6a 6b 6c 6f 77 64 65 6e 0a | jklowden.|
0000001a
(I shortened your command. Nearly every command that combines sed and awk can be better expressed with just awk. Then I added the second field, so we could see what's going on.)
The result is nondeterministic. top will highlight changed lines; to do so, it emits an ANSI escape sequence. If you capture one, you'll capture that sequence -- esc(Besc[mesc[1m -- which will look very weird indeed to ps. The brackets in that sequence doubtless provoked your error message.
To fix that, your top probably has a batch mode. In mine, top -n 1 -b does the trick.
How to use ps to find a command line?
I'm afraid the best answer is RTFM. ps is one of those commands with a lot of variation across systems. My GNU version likes this:
$ ps -c -f -p $(top -n 1 -b | awk 'NR==8 {print $1}')
UID PID PPID CLS PRI STIME TTY TIME CMD
root 1300 1 TS 19 Jun30 ? 00:14:46 /usr/bin/python /usr/bin/la
Other observations:
Prefer $() to backticks
While this kind of thing is fine for learning, look for solutions that don't involve parsing output from interactive utilities. Under account in the manual, you'll find ways to capture much more information than just command line of what happens to be on top at the moment.
HTH.
Because your ppid retrieve null value
change your command to retrieve ppid value as below
top -n 1 | sed -n 8p | awk -F " " '{print$2}'
And other thing for happening this is top process will end immediately
This is in my ubuntu 14.04 os.

ffmpeg fails with a wall of red hex output but only when in a bash loop

Similar question here, but no solution: https://www.reddit.com/r/linuxquestions/comments/m00f0/hex_output_from_ffmpeg_when_run_in_a_loop/
I'm trying to batch encode some video. I have a list of filenames in a text file and read each one in a bash loop. This is it:
OUT=./out
mkdir -p $OUT
while read inputfile; do
b=$(basename "$inputfile")
outputfile="$OUT/$b"
echo
echo
echo "$inputfile"
echo "$outputfile"
echo
ffmpeg -i "$inputfile" "$outputfile" || exit
done < files.txt
When I run it, the ffmpeg command seems to start OK, then spits out a massive wall of red hex dump like text (as below). It does finish without error, but the file is only a few hundred frames long.
[libx264 # 0x1b68d60] frame= 537 QP=31.44 NAL=0 Slice:B Poc:70 I:2 P:239 SKIP:987 size=516 bytes
stream #1:
keyframe=1
duration=0.024
dts=25.992 pts=25.992
size=336
00000000 ff fb 84 64 1a 0e d3 21 3f d4 93 18 6a e6 33 a4 ...d...!?...j.3.
00000010 6a 63 25 22 4a 0c d1 1d 52 6c bd ab c0 bf 23 69 jc%"J...Rl....#i
... LOTS MORE
The other really weird thing is that the next inputfile in the bash loop is broken and has only the last few characters, which does not happen if I simply comment out the ffmpeg line.
What could ffmpeg be doing to interfere with the bash loop like this?
(ffmpeg 2.8.6, bash 4.3.42(1), fedora 23)
Found the answer here: https://unix.stackexchange.com/questions/241535/problem-with-ffmpeg-in-bash-loop
Somehow the read command's stdin ends up shared by ffmpeg. Still not sure why, but the answer is to re-route the ffmpeg stdin from /dev/null. E.g.:
ffmpeg -i "$inputfile" "$outputfile" < /dev/null || exit
The ffmpeg verbosity is quite tricky. Some ideas can be found in their online manual page. A good option is to use the warning level options ‘warning, 24’ like this:
ffmpeg -v 24 -nostdin ...

Different zgrep output from command line and bash script

I have a bash script with that loops through lines of a file (a.txt) containing paths to a list of gzipped files and searches for phrases in each of those files using zgrep.
My bash script is:
for i in $(cat $1); do
echo $i;
echo zgrep -E '"phrase1|phrase2|phrase3|phrase4|phrase5|phrase6"' $i;
zgrep -E '"phrase1|phrase2|phrase3|phrase4|phrase5|phrase6"' $i;
done
which I call by myscript.sh a.txt:
The output is:
zgrep -E "phrase1|phrase2|phrase3|phrase4|phrase5|phrase6" myzippedfile.1.gz
phrase2 48 48.00 48
phrase3 35 35.00 35
phrase4 67 67.00 67
phrase5 99 99.00 99
(repeated for each file listed in a.txt).
However, when I execute the zgrep command that is being executed inside the scripts for loop, I get a different output.
Executing:
zgrep -E "phrase1|phrase2|phrase3|phrase4|phrase5|phrase6" myzippedfile.1.gz
at the command line yields:
phrase1 29 29.00 29
phrase2 48 48.00 48
phrase3 35 35.00 35
phrase4 67 67.00 67
phrase5 99 99.00 99
phrase6 54 54.00 54
This output is correct, whereas the output generated from the bash script for loop is missing the first and last lines. How can this be?
Does anyone see any issues in my bash script? Why would the first and last lines be missing from the output?
The first and last elements in your alternatives contain literal " characters. So you'll only match phrase1 if there's a " before it, and only match phrase6 if there's a " after it. You shouldn't have those double quotes in the pattern, it should be:
zgrep -E 'phrase1|phrase2|phrase3|phrase4|phrase5|phrase6' $i;
You don't have the extra quotes when you execute the zgrep command by hand.

Resources