Use sed to print matching lines - shell

Sample data in a stackoverflow.csv file I have:
foo
foo
foo
foo
bar
bar
bar
bar
baz
baz
baz
baz
I know with sed -n /foo/p stackoverflow.csv > foo.csv
I'll get all records matching foo directed to that file, but I don't want to specify the matching pattern on the cli, I'd rather put it in a script and have all records (foo, bar and baz) sent to their own file.
Basically this in a script:
sed -n /foo/p stackoverflow.csv > foo.csv
sed -n /bar/p stackoverflow.csv > bar.csv
sed -n /baz/p stackoverflow.csv > baz.csv
Like this:
#!/bin/sh
sleep 2
sed -n /foo/p > foo.csv
sed -n /bar/p > bar.csv
sed -n /baz/p > baz.csv
This creates the files but they're all empty.
Also, if the script were to have just one print statement, it works.
Any input?

You missed the input filename
#!/bin/sh
sleep 2
sed -n /foo/p stackoverflow.csv > foo.csv
sed -n /bar/p stackoverflow.csv > bar.csv
sed -n /baz/p stackoverflow.csv > baz.csv
You can provide input file as an argument to your script, in that case change the script to read an argument and pass it to the sed command.
Run the script like this: ./script.sh input_filename, where you can specify different input files as argument.
#!/bin/sh
file=${1}
sed -n /foo/p ${file} > foo.csv
sed -n /bar/p ${file} > bar.csv
sed -n /baz/p ${file} > baz.csv

Here's a solution with grep in a loop where you don't need to provideeach term in advance. Assuming a bash shell:
for i in $(sort stackoverflow.csv | uniq); do grep $i testfile > $i; done

The w command in sed allows you to write to a named file.
sed -n -e '/foo/wfoo.csv' -e '/bar/wbar.csv' \
-e '/baz/wbaz.csv' stackoverflow.csv
Not all sed dialects accept multiple -e arguments; a multi-line string with the commands separated by newlines may be the most portable solution.
sed -n '/foo/wfoo.csv
/bar/wbar.csv
/baz/wbaz.csv' stackoverflow.csv
This examines the input file just once, and writes out the matching lines as it proceeds through the file.
To create a script which accepts one or more file names as arguments, replace the hard-coded file name with "$#".
#!/bin/sh
sed -n ... "$#"

Related

Sed find and replace expression works with literal but not with variable interpolation

For the following MVCE:
echo "test_num: 0" > test.txt
test_num=$(grep 'test_num:' test.txt | cut -d ':' -f 2)
new_test_num=$((test_num + 1))
echo $test_num
echo $new_test_num
sed -i "s/test_num: $test_num/test_num: $new_test_num/g" test.txt
cat test.txt
echo "sed -i "s/test_num: $test_num/test_num: $new_test_num/g" test.txt"
sed -i "s/test_num: 0/test_num: 1/g" test.txt
cat test.txt
Which outputs
0 # parsed original number correctly
1 # increment the number
test_num: 0 # sed with interpolated variable, does not work
sed -i s/test_num: 0/test_num: 1/g test.txt # interpolated parameter looks right
test_num: 1 # ???
Why does sed -i "s/test_num: $test_num/test_num: $new_test_num/g" test.txt not produce the expected result when sed -i "s/test_num: 0/test_num: 1/g" test.txt works just fine in the above example?
As mentioned in the comment, there is a white space in ${test_num}. Therefore in your sed there should not be an empty space between the colon and your variable.
Also I guess you should surround your variable with curly bracket {} to increase readability.
sed "s/test_num:${test_num}/test_num: ${new_test_num}/g" test.txt
test_num: 1
If you just want the number in ${test_num}, you can try something like:
grep 'test_num:' test.txt | awk -F ': ' '{print $2}'
awk allows to specify delimiter with more than 1 character.
Instead of grep|cut you can also use sed.
#! /bin/bash
exec <<EOF
test_num: 0
EOF
grep 'test_num:' | cut -d ':' -f 2
exec <<EOF
test_num: 0
EOF
sed -n 's/^test_num: //p'
When using regexp replace in sed there is special meaning to $ .
Suggesting to rebuild your sed command segments as follow:
sed -i 's/test_num: '$test_num'/test_num: '$new_test_num'/g' test.txt
Other option, use echo command to expand variables in sed command.
sed_command=$(echo "s/test_num:${test_num}/test_num: ${new_test_num}/g")
sed -i "$sed_command" test.txt

How to pass an bash command line argument to sed -n p?

I want to output a specific line from bash argument using sed, I have tried many ways, but none work:
#!/bin/bash
sed -n '$2p' $1
sed -n '${2}p' $1
sed -n "$2p" $1
sed -n "${2}p" $1
sed -n ''"$2"'p' $1
How on earth do I get the correct result?
Try
sed -n "$2p" $1
Demo:
$seq 10 > file.txt
$cat temp.ksh
#!/bin/bash
set -x
sed -n "$2p" $1
$./temp.ksh file.txt 3
+ sed -n 3p file.txt
3
$

How to write a command line script that will loop through every line in a text file and append a string at the end of each? [duplicate]

How do I add a string after each line in a file using bash? Can it be done using the sed command, if so how?
If your sed allows in place editing via the -i parameter:
sed -e 's/$/string after each line/' -i filename
If not, you have to make a temporary file:
typeset TMP_FILE=$( mktemp )
touch "${TMP_FILE}"
cp -p filename "${TMP_FILE}"
sed -e 's/$/string after each line/' "${TMP_FILE}" > filename
I prefer echo. using pure bash:
cat file | while read line; do echo ${line}$string; done
I prefer using awk.
If there is only one column, use $0, else replace it with the last column.
One way,
awk '{print $0, "string to append after each line"}' file > new_file
or this,
awk '$0=$0"string to append after each line"' file > new_file
If you have it, the lam (laminate) utility can do it, for example:
$ lam filename -s "string after each line"
Pure POSIX shell and sponge:
suffix=foobar
while read l ; do printf '%s\n' "$l" "${suffix}" ; done < file |
sponge file
xargs and printf:
suffix=foobar
xargs -L 1 printf "%s${suffix}\n" < file | sponge file
Using join:
suffix=foobar
join file file -e "${suffix}" -o 1.1,2.99999 | sponge file
Shell tools using paste, yes, head
& wc:
suffix=foobar
paste file <(yes "${suffix}" | head -$(wc -l < file) ) | sponge file
Note that paste inserts a Tab char before $suffix.
Of course sponge can be replaced with a temp file, afterwards mv'd over the original filename, as with some other answers...
This is just to add on using the echo command to add a string at the end of each line in a file:
cat input-file | while read line; do echo ${line}"string to add" >> output-file; done
Adding >> directs the changes we've made to the output file.
Sed is a little ugly, you could do it elegantly like so:
hendry#i7 tmp$ cat foo
bar
candy
car
hendry#i7 tmp$ for i in `cat foo`; do echo ${i}bar; done
barbar
candybar
carbar

Passing variables between sed script and bash script

I have a shell script with up to 5 parameters. There is files with placeholders. I would like to use sed script file depending of the one variable. The problem is that when I have variables defined in the sed script - values of those variables are not put in placeholders.
#!/bin/bash
A=$1
B=$2
echo "Some string with _PH1_ place holders _PH2_"|sed -i -f script1.sed >> file.txt
one of the sed scripts file
#Sed script 1
s/_PH1_/${A}/g
s/_PH2_/${B}/g
If your sed script is that short, you might as well inline it in bash:
#!/bin/bash
A=$1
B=$2
#your sed commands but still part of the bash script:
sed -i -e "s/_PH1_/${A}/g" file.txt
sed -i -e "/_PH2_/${B}/g" file.txt
You need to create the sed script as part of your bash script so the variable substitution takes place:
#!/bin/bash
A=$1
B=$2
cat >> script1.sed << EOF
s/_PH1_/${A}/g
s/_PH2_/${B}/g
EOF
echo "Some string with _PH1_ place holders _PH2_"|sed -i -f script1.sed >> file.txt
$ cat tst.sh
#!/bin/bash
set -a
A="$1"
B="$2"
sedScript=$(mktemp)
printf 'sed "\n' >> "$sedScript"
cat script1.sed >> "$sedScript"
printf '"\n' >> "$sedScript"
echo "Some string with _PH1_ place holders _PH2_" | "$sedScript"
rm -f "$sedScript"
$ ./tst.sh foo bar
Some string with foo place holders bar

Pipe stdout to command which itself needs to read from own stdin

I would like to get the stdout from a process into another process not using stdin, as that one is used for another purpose.
In short I want to accomplish something like that:
echo "a" >&4
cat | grep -f /dev/fd/4
I got it running using an file as source for file descriptor 4, but that is not what I want:
# Variant 1
cat file | grep -f /dev/fd/4 4<pattern
# Variant 2
exec 4<pattern
cat | grep -f /dev/fd/4
exec 4<&-
My best try is that, but I got the following error message:
# Variant 3
cat | (
echo "a" >&4
grep -f /dev/fd/4
) <&4
Error message:
test.sh: line 5: 4: Bad file descriptor
What is the best way to accomplish that?
You don't need to use multiple streams to do this:
$ printf foo > pattern
$ printf '%s\n' foo bar | grep -f pattern
foo
If instead of a static file you want to use the output of a command as the input to -f you can use a process substitution:
$ printf '%s\n' foo bar | grep -f <(echo foo)
foo
For POSIX shells that lack process substitution, (e.g. dash, ash, yash, etc.).
If the command allows string input, (grep allows it), and the input string containing search targets isn't especially large, (i.e. the string doesn't exceed the length limit for the command line), there's always command substitution:
$ printf '%s\n' foo bar baz | grep $(echo foo)
foo
Or if the input file is multi-line, separating quoted search items with '\n' works the same as grep OR \|:
$ printf '%s\n' foo bar baz | grep "$(printf "%s\n" foo bar)"
foo
bar

Resources