a=`cat /etc/redhat-release | awk '{print $2}' > /tmp/a.txt`
The above command is not redirecting the output to file.
A command substitution captures stdout of the command contained. When you redirect that output to a file, it's no longer on stdout, so it's no longer captured.
Use tee to create two copies -- one in a file, one on stdout.
a=$(awk '{print $2}' </etc/redhat-release | tee /tmp/a)
Note also:
cat shouldn't be used when it isn't needed: Giving awk a direct handle on the input file saves an extra process, allowing a direct write from a file rather than a FIFO -- and is following a practice that will generate much larger efficiency games with programs like sort, shuf, tail, or wc -c that can use more efficient algorithms when reading from a file.
The modern (and standard-compliant, since 1991) syntax for command substitution is $(...). It nests better than the ancient backtick syntax it replaces, and use of backslashes within is less confusing.
Related
Basically I want to copy several lines of code from a template file to a script file.
Is it even possible to use sed to copy a string full of symbols that interact with the script?
I used these lines:
$SWAP='sudo cat /home/kaarel/template'
sed -i -e "s/#pointer/${SWAP}/" "script.sh"
The output is:
./line-adder.sh: line 11: =sudo cat /home/kaarel/template: No such file or directory
No, it is not possible to do this robustly with sed. Just use awk:
awk -v swap="$SWAP" '{sub(/#pointer/,swap)}1' script.sh > tmp && mv tmp script.sh
With recent versions of GNU awk there's a -i inplace flag for inplace editing if that's something you care about.
Good point about "&&". Here's the REALLY robust version that will work for absolutely any character in the search or replacement strings:
awk -v old="#pointer" -v new="$SWAP" 's=index($0,old){$0 = substr($0,1,s-1) new substr($0,s+length(old))} 1'
e.g.:
$ echo "abc" | awk -v old="b" -v new="m&&n" 's=index($0,old){$0 = substr($0,1,s-1) new substr($0,s+length(old))} 1'
am&&nc
There are two issues with the line:
$SWAP='sudo cat /home/kaarel/template'
The first is that, before executing the line, bash performs variable expansion and replaces $SWAP with the current value of SWAP. That is not what you wanted. You wanted bash to assign a value to SWAP.
The second issue is that the right-hand side is enclosed in single-quotes which protect the string from expansion. You didn't want to protect the string from expansion: you wanted to execute it. To execute it, you can use back-quotes which may look similar but act very differently.
Back-quotes, however, are an ancient form of asking for command execution. The more modern form is $(...) which eliminates some problems that back-quotes had.
Putting it all together, use:
SWAP=$(sudo cat /home/kaarel/template)
sed -i -e "s/#pointer/${SWAP}/" "script.sh"
Be aware, though, that the sed command may have problems if there are any sed-active characters in the template file.
I am using the below script. When I have it echo $f as shown below, it gives the correct result:
#/bin/bash
var="\/home\/"
while read p; do
f=$(echo $p | sed "s/${var}/\\n/g")
f=${f%.sliced.bam}.fastq
echo $f
~/bin/samtools view $p | awk '{print "#"$1"\n"$10"\n+\n"$11}' > $f
./run.sh $f ${f%.fastq}
rm ${f%.sliced.bam}.fastq
done < $1
I get the output as expected
test.fastq
But the file being created by awk > $f has the name
?test.fastq
Note that the overall goal here is to run this loop on every file listed in a file with absolute paths but then write locally (which is what the sed call is for)
edit: Run directly on the command line (without variables) the samtools | awk line runs correctly.
Awk cannot possibly have anything to do with your problem. The shell is completely responsible for file redirection, so f MUST have a weird character in it.
Most likely whatever you are sending to this script has a special character in it (e.g. perhaps a UTF character, and your terminal is showing ASCII only). When you do the echo, the shell doesn't know how to display the char, and probably just shows it as whitespace, and when you send it through ls (which might be doing things like colorization) it combines in a strange way and ends up showing the ?.
Oh wait...why are you putting a newline into the filename with sed??? That is possibly your problem...try just:
sed "s/${var}//g"
I have not learned Bash in a formal way so please do give me suggestions for a more descriptive question title.
Instead of creating a temporary file whose lifespan is limited to that of the command it is used by (in this case, command), as in:
zcat input.txt.gz > input.txt
command input.txt
rm input.txt
we can avoid it as follows:
zcat input.txt.gz | command -
Now my question is whether this is possible with two inputs. I wish to avoid creating two temporary files, as in:
zcat input1.txt.gz > input1.txt
zcat input2.txt.gz > input2.txt
command input1.txt input2.txt
rm input1.txt input2.txt
I am guessing that the following solution can remove the need to create one of the two temporary files, as:
zcat input1.txt.gz > input1.txt
zcat input2.txt.gz | command input1.txt -
rm input1.txt
but I wonder if there is a way to completely avoid creating the temporary file.
I hope my question was clear enough. Though I used zcat as an example, the solution I am looking for should be more general. Thanks in advance.
If you're trying to combine the output of multiple commands into a single pipe, use a subshell:
(cat file1.txt; cat file2.txt) | nl
If you want to use the output of a command as a filename for a command, use process substitution:
diff <(zcat file1.gz) <(zcat file2.gz)
Subshells might get you what you want:
command $(zcat input1.txt.gz) $(zcat input2.txt)
So long the stdout of the 2 subshells (above) make up arguments for 'command'
For example, if I want to do a diff of two files after preprocessing both of them with sed, is there any way to do this without temporary files?
I have tried things like this and (as I expected) it did not work:
(sed "$expr" file1; sed "$expr" file2) | diff - -
I was thinking there might be a way to create pipes explicitly or something.
Try doing this :
diff <(sed "$expr" file1) <(sed "$expr" file2)
This uses Process Substitution. <( ) is replaced by a temporary filename. Writing or reading that file causes bytes to get piped to the command inside. Often used in combination with file redirection:
cmd1 2> >(cmd2)
See
http://mywiki.wooledge.org/ProcessSubstitution
http://mywiki.wooledge.org/BashFAQ/024
I am having trouble with this simple task:
cat file | grep -E ^[0-9]+$ > file_grep
diff file file_grep
Problem is, I want to do this without file_grep
I have tried:
diff file `cat file | grep -E ^[0-9]+$`
and
diff file "`cat file | grep -E ^[0-9]+$`"
and a few other combinations :-) but I can't get it to work.
I always get an error, when the diff gets extra argument which is content of file filtered by grep.
Something similar always worked for me, when I wanted to echo command outputs from within a script like this (using backtick escapes):
echo `ls`
Thanks
If you're using bash:
diff file <(grep -E '^[0-9]+$' file)
The <(COMMAND) sequence expands to the name of a pseudo-file (such as /dev/fd/63) from which you can read the output of the command.
But for this particular case, ruakh's solution is simpler. It takes advantage of the fact that - as an argument to diff causes it to read its standard input. The <(COMMAND) syntax becomes more useful when both arguments to diff are command output, such as:
diff <(this_command) <(that_command)
The simplest approach is:
grep -E '^[0-9]+$' file | diff file -
The hyphen - as the filename is a specific notation that tells diff "use standard input"; it's documented in the diff man-page. (Most of the common utilities support the same notation.)
The reason that backticks don't work is that they capture the output of a command and pass it as an argument. For example, this:
cat `echo file`
is equivalent to this:
cat file
and this:
diff file "`cat file | grep -E ^[0-9]+$`"
is equivalent to something like this:
diff file "123
234
456"
That is, it actually tries to pass 123234345 (plus newlines) as a filename, rather than as the contents of a file. Technically, you could achieve the latter by using Bash's "process substitution" feature that actually creates a sort of temporary file:
diff file <(cat file | grep -E '^[0-9]+$')
but in your case it's not needed, because of diff's support for -.
grep -E '^[0-9]+$' file | diff - file
where - means "read from standard input".
Try process substitution:
$ diff file <(grep -E "^[0-9]+$" file)
From the bash manpage:
Process Substitution
Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of
naming open files. It takes the form of <(list) or >(list). The process list is run with its input or
output connected to a FIFO or some file in /dev/fd. The name of this file is passed as an argument to
the current command as the result of the expansion. If the >(list) form is used, writing to the file
will provide input for list. If the <(list) form is used, the file passed as an argument should be read
to obtain the output of list.
In bash, the syntax is
diff file <(cat file | grep -E ^[0-9]+$)