script.sh takes two files as arguments, and it calls a python script that opens them and does further processing on these files:
script.sh file1.txt file2.txt
I would like to remove all lines from file1.txt that start with #.
grep -v '^#' file1.txt
I tried the following but it fails :
script.sh $(grep -v '^#' file1.txt) file2.txt
How can I pipe the output of grep as the first file argument of script.sh?
script.sh <(grep -v '^#' file1.txt) file2.txt
Command substitution as you had it, $(), will insert the output from the grep command verbatim as arguments to script.sh. In contrast, to quote the bash man page:
Process substitution allows a process's input or output to be referred to using a filename. It takes the form of <(list) or >(list). The process list is run asynchronously, and its input or output appears as a filename. This filename is passed as an argument to the current command as the result of the expansion.
and
If the <(list) form is used, the file passed as an argument should be read to obtain the output of list.
So, my understanding is that when you use $(), the output of your grep command is substituted into the final statement with a result something like:
script.sh contents of file1 without comments \n oops newline etc file2.txt
But when you use <(), the grep command is run and the output goes into a named pipe or temporary file or some equivalent, and the name of that "file" is substituted into your statement, so you end up with something along the lines of
script.sh /dev/fd/temp_thing_containing_grep_results file2.txt
Related
This question already has answers here:
Is it OK to use the same input file as output of a piped command?
(3 answers)
Closed last year.
File1:
foo
bar
baz
File2:
bar.patch
grep -f file1 file2 > file1
Expected result, file1 contains bar.patch, instead it is empty.
How is grep processing the input file such that I cannot redirect to it?
Redirection > happens before the command is executed, hence the grep sees empty file and returns nothing. If you really want to write to the same file you can use the sponge command (it is part of the moreutils). It buffers the input and writes to the output file only after the end of data. Example:
grep -f file1 file2 | sponge file1
Check the bash manual for details about redirection:
Before a command is executed, its input and output may be redirected
using a special notation interpreted by the shell. Redirection allows
commands’ file handles to be duplicated, opened, closed, made to refer
to different files, and can change the files the command reads from
and writes to. Redirection may also be used to modify file handles in
the current shell execution environment. The following redirection
operators may precede or appear anywhere within a simple command or
may follow a command. Redirections are processed in the order they
appear, from left to right.
Or just redirect to a temporary file and then mv it over the file1.
grep -f file1 file2 > file3
mv -f file3 file1
File info has some certain info on a line starting with myline. Im trying to pass it to a script like this:
bash myscript `cat info | grep myline`
This works well. Script gets "myline" as first argument. But now i want to add a "w" at the end of that. I tried
bash myscript `cat info | grep myline`w
This is already problematic, the script gets "wyline" as first argument.
And now the next step is that i actually want to have an if statement whether i want to add w or not. Tried this:
bash myscript `cat info | grep myline``[ "condition" == "condition"] && echo "w"`
This works the same way. Script gets "wyline" as first argument.
So I have two questions:
1) How to fix the "wyline" result to get desired "mylinew"
2) Is there a better way to write this if statement after cat?
Do not use backticks `, use $(...) instead. bash hackers obsolete deprecated syntax
cat file | grep is a useless use of cat useless use of cat award. Just grep file.
Just quote the result and add w:
myscript "$(grep myline info)w"
You can add a trailing w to the last line of input with sed:
myscript "$(grep myline info | sed '$s/$/w/')"
I would advise to always quote your variable expansions.
Script gets "wyline" as first argument.
Your input file has dos line endings. Inspect output with cut -v or hexdump -C or xxd. Use dos2unix and remove carriage return characters.
I am a beginer in Shell script. Below is my requirement in UNIX Korn Shell.
Example:
When we list files using ls command redirect to a file the file names will be stored as below.
$ ls FILE*>FLIST.TXT
$ cat FLIST.TXT
FILE1
FILE2
FILE3
But I need output as below with a prefixed constant string STR,:
$ cat FLIST.TXT
STR,FILE1
STR,FILE2
STR,FILE3
Please let me what should be the ls command to acheive this output.
You can't use ls alone to append data before each file. ls exists to list files.
You will need to use other tools along side ls.
You can append to the front of each line using the sed command:
cat FLIST.TXT | sed 's/^/STR,/'
This will send the changes to stdout.
If you'd like to change the actual file, run sed in place:
sed -i -e 's/^/STR,/' FLIST.TXT
To do the append before writing to the file, pipe ls into sed:
ls FILE* | sed 's/^/STR,/' > FLIST.TXT
The following should work:
ls FILE* | xargs -i echo "STR,{}" > FLIST.TXT
It takes every one of the file names filtered by ls and adds the "STR," prefix to it prior to the appending
Is there a bash command like cat except that it takes the filename from stdin rather than the first argument? For example:
echo "/home/root/file.txt" | somecommand
would have the same output as cat /home/root/file.txt
Found the answer:
xargs cat
For example:
echo "/home/root/file.txt" | xargs cat
If the filename that you're expecting doesn't have a space:
#!/bin/bash
read filename
cat $filename
The read builtin shell command reads tokens from stdin into environment variables.
I am having trouble with this simple task:
cat file | grep -E ^[0-9]+$ > file_grep
diff file file_grep
Problem is, I want to do this without file_grep
I have tried:
diff file `cat file | grep -E ^[0-9]+$`
and
diff file "`cat file | grep -E ^[0-9]+$`"
and a few other combinations :-) but I can't get it to work.
I always get an error, when the diff gets extra argument which is content of file filtered by grep.
Something similar always worked for me, when I wanted to echo command outputs from within a script like this (using backtick escapes):
echo `ls`
Thanks
If you're using bash:
diff file <(grep -E '^[0-9]+$' file)
The <(COMMAND) sequence expands to the name of a pseudo-file (such as /dev/fd/63) from which you can read the output of the command.
But for this particular case, ruakh's solution is simpler. It takes advantage of the fact that - as an argument to diff causes it to read its standard input. The <(COMMAND) syntax becomes more useful when both arguments to diff are command output, such as:
diff <(this_command) <(that_command)
The simplest approach is:
grep -E '^[0-9]+$' file | diff file -
The hyphen - as the filename is a specific notation that tells diff "use standard input"; it's documented in the diff man-page. (Most of the common utilities support the same notation.)
The reason that backticks don't work is that they capture the output of a command and pass it as an argument. For example, this:
cat `echo file`
is equivalent to this:
cat file
and this:
diff file "`cat file | grep -E ^[0-9]+$`"
is equivalent to something like this:
diff file "123
234
456"
That is, it actually tries to pass 123234345 (plus newlines) as a filename, rather than as the contents of a file. Technically, you could achieve the latter by using Bash's "process substitution" feature that actually creates a sort of temporary file:
diff file <(cat file | grep -E '^[0-9]+$')
but in your case it's not needed, because of diff's support for -.
grep -E '^[0-9]+$' file | diff - file
where - means "read from standard input".
Try process substitution:
$ diff file <(grep -E "^[0-9]+$" file)
From the bash manpage:
Process Substitution
Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of
naming open files. It takes the form of <(list) or >(list). The process list is run with its input or
output connected to a FIFO or some file in /dev/fd. The name of this file is passed as an argument to
the current command as the result of the expansion. If the >(list) form is used, writing to the file
will provide input for list. If the <(list) form is used, the file passed as an argument should be read
to obtain the output of list.
In bash, the syntax is
diff file <(cat file | grep -E ^[0-9]+$)