bash sed fail in while loop - bash

#!/bin/bash
fname=$2
rname=$1
echo "$(<$fname)" | while read line ; do
result=`echo "$(<$rname)" | grep "$line"; echo $?`
if [ $result != 0 ]
then
sed '/$line/d' $fname > newkas
fi 2> /dev/null
done
Hi all, i am new to bash.
i have two lists one older than another. I wish to compare the names on 'fname' against 'rname'. 'Result' is the standard out put which i will get if the name is still available in 'rname'. if is not then i will get the non-zero output.
Using sed to delete that line and re route it to a new file.
I have tried part by part of the code and it works until i add in the while loop function. sed don't seems to work as the final output of 'newkas' is the same as the initial input 'fname'.
Is my method wrong or did i miss out any parts?

Part 1: What's wrong
The reason your sed expression "doesn't work" is because you used single quotes. You said
sed '/$line/d' $fname > newkas
Supposing fname=input.txt' and line='example text' this will expand to:
sed '/$line/d' input.txt > newkas
Note that $line is still literally present. This is because bash will not interpolate variables inside single quotes, thus sed sees the $ literally.
You could fix this by saying
sed "/$line/d/" $fname > newkas
Because inside double quotes the variable will expand. However, if your sed expression becomes more complicated you could run into difficulty in cases where bash interprets things which you intended to be interpreted by sed. I tend to use the form
sed '/'"$line"'/d/' $fname > newkas
Which is a bit harder to read but, if you look carefully, single-quotes everything I intend to be part of the sed expression and double quotes the variable I want to expand.
Part 2: How to improve it
Your script contains a number things which could be improved.
echo "$(<$fname)" | while read line ; do
:
done
In the first place you're reading the file with "$(<$fname)" when you could just redirect the stdin of the while loop. This is a bit redundant, but more importantly you're piping to while, which creates an extra subshell and means you can't modify any variables from the enclosing scope. Better to say
while IFS= read -r line ; do
:
done < "$fname"
Next, consider your grep
echo "$(<$rname)" | grep "$line"
Again you're reading the file and echoing it to grep. But, grep can read files directly.
grep "$line" "$rname"
Afterwards you echo the return code and check its value in an if statement, which is a classic useless construct.
result=$( grep "$line" "$rname" ; echo $?)
Instead you can just pass grep directly to if, which will test its return code.
if grep -q "$line" "$rname" ; then
sed "/$line/d" "$fname" > newkas
fi
Note here that I have quoted $fname, which is important if it might ever contain a space. I have also added -q to grep, which suppresses its output.
There's now no need to suppress error messages from the if statement, here, because we don't have to worry about $result containing an unusual value or grep not returning properly.
The final result is this script
while IFS= read -r line ; do
if grep -q "$line" "$rname" ; then
sed "/$line/d" "$fname" > newkas
fi
done < "$fname"
Which will not work, because newkas is overwritten on every loop. This means that in the end only the last line in $fname was used. Instead you could say:
cp "$fname" newkas
while IFS= read -r line ; do
if grep -q "$line" "$rname" ; then
sed -i '' "/$line/d" newkas
fi
done < "$fname"
Which, I believe, will do what you expect.
Part 3: But don't do that
But this is all tangential to solving your actual problem. It appears to me that you want to simply create a file newkas which contains the all the lines of $fname except those that appear in $rname. This is easily done with the comm utility:
comm -2 -3 <(sort "$fname") <(sort "$rname") > newkas
This also changes the sort order of the lines, which may not be good for you. If you want to do it without changing the ordering then using the method #fge suggests is best.
grep -F -v -x -f "$rname" "$fname"

If I understand your need correctly, you want a file newaks which contains the lines in $fname which are also in $rname.
If this is what you want, using sed is overkill. Use fgrep:
fgrep -x -f $fname $rname > newkas
Also, there are problems with your script:
you capture the output of grep in result, which means it will never be exactly 0; what you want is executing the command and simply check for $?
your echoes are convoluted, just do grep whatever thefilename, or while...done <thefile;
finally, you take the line as is from the source file: the line can potentially be a regex, which means you will try and match a regex in $rname, which may yield to unexpected results.
And others.

Related

Why does my variable set in a do loop disappear? (unix shell)

This part of my script is comparing each line of a file to find a preset string. If the string does NOT exist as a line in the file, it should append it to the end of the file.
STRING=foobar
cat "$FILE" | while read LINE
do
if [ "$STRING" == "$LINE" ]; then
export ISLINEINFILE="yes"
fi
done
if [ ! "$ISLINEINFILE" == yes ]; then
echo "$LINE" >> "$FILE"
fi
However, it appears as if both $LINE and $ISLINEINFILE are both cleared upon finishing the do loop. How can I avoid this?
Using shell
If we want to make just the minimal change to your code to get it working, all we need to do is switch the input redirection:
string=foobar
while read line
do
if [ "$string" == "$line" ]; then
islineinfile="yes"
fi
done <"$file"
if [ ! "$islineinfile" == yes ]; then
echo "$string" >> "$file"
fi
In the above, we changed cat "$file" | while do ...done to while do...done<"$file". With this one change, the while loop is no longer in a subshell and, consequently, shell variables created in the loop live on after the loop completes.
Using sed
I believe that the whole of your script can be replaced with:
sed -i.bak '/^foobar$/H; ${x;s/././;x;t; s/$/\nfoobar/}' file*
The above adds line foobar to the end of each file that doesn't already have a line that matches ^foobar$.
The above shows file* as the final argument to sed. This will apply the change to all files matching the glob. You could list specific files individually if you prefer.
The above was tested on GNU sed (linux). Minor modifications may be needed for BSD/OSX sed.
Using GNU awk (gawk)
awk -i inplace -v s="foobar" '$0==s{f=1} {print} ENDFILE{if (f==0) print s; f=0}' file*
Like the sed command, this can tackle multiple files all in one command.
Why does my variable set in a do loop disappear?
It disappears because it is set in a shell pipeline component. Most shells run each part of a pipeline in a subshell. By Unix design, variables set in a subshell cannot affect their parent or any already running other shell.
How can I avoid this?
There are several ways:
The simplest is to use a shell that doesn't run the last component of a pipeline in a subshell. This is ksh default behavior, e.g. use that shebang:
#!/bin/ksh
This behavior can also be bash one when the lastpipe option is set:
shopt -s lastpipe
You might use the variable in the same subshell that set it. Note that your original script indentation is wrong and might lead to the incorrect assumption that the if block is inside the pipeline, which isn't the case. Enclosing the whole block with parentheses will rectify that and would be the minimal change (two extra characters) to make it working:
STRING=foobar
cat "$FILE" | ( while read LINE
do
if [ "$STRING" == "$LINE" ]; then
export ISLINEINFILE="yes"
fi
done
if [ ! "$ISLINEINFILE" == yes ]; then
echo "$LINE" >> "$FILE"
fi
)
The variable would still be lost after that block though.
You might simply avoid the pipeline, which is straigthforward in your case, the cat being unnecessary:
STRING=foobar
while read LINE
do
if [ "$STRING" == "$LINE" ]; then
export ISLINEINFILE="yes"
fi
done < "$FILE"
if [ ! "$ISLINEINFILE" == yes ]; then
echo "$LINE" >> "$FILE"
fi
You might use another argorithmic approach, like using sed or gawk as suggested by John1024.
See also https://unix.stackexchange.com/a/144137/2594 for standard compliance details.

Making bash script with command already containing '$1'

Somewhere I found this command that sorts lines in an input file by number of characters(1st order) and alphabetically (2nd order):
while read -r l; do echo "${#l} $l"; done < input.txt | sort -n | cut -d " " -f 2- > output.txt
It works fine but I would like to use the command in a bash script where the name of the file to be sorted is an argument:
& cat numbersort.sh
#!/bin/sh
while read -r l; do echo "${#l} $l"; done < $1 | sort -n | cut -d " " -f 2- > sorted-$1
Entering numbersort.sh input-txt doesn't give the desired result, probably because $1 is already in using as an argument for something else.
How do I make the command work in a shell script?
There's nothing wrong with your original script when used with simple arguments that don't involve quoting issues. That said, there are a few bugs addressed in the below version:
#!/bin/bash
while IFS= read -r line; do
printf '%d %s\n' "${#line}" "$line"
done <"$1" | sort -n | cut -d " " -f 2- >"sorted-$1"
Use #!/bin/bash if your goal is to write a bash script; #!/bin/sh is the shebang for POSIX sh scripts, not bash.
Clear IFS to avoid pruning leading and trailing whitespace from input and output lines
Use printf rather than echo to avoid ambiguities in the POSIX standard (see http://pubs.opengroup.org/onlinepubs/009604599/utilities/echo.html, particularly APPLICATION USAGE and RATIONALE sections).
Quote expansions ("$1" rather than $1) to prevent them from being word-split or glob-expanded
Note also that this creates a new file rather than operating in-place. If you want something that operates in-place, tack a && mv -- "sorted-$1" "$1" on the end.

Script to call either from file or user input

I'm trying to write a small script that either takes input from a file or from user, then it gets rid of any blank lines from it.
I'm trying to make it so that if there is no file name specified it will prompt the user for input. Also is the best way to output the manual input to a file then run the code or to store it in a variable?
So far I have this but when I run it with a file it give 1 line of error before returning the output I want. The error says ./deblank: line 1: [blank_lines.txt: command not found
if [$# -eq "$NO_ARGS"]; then
cat > temporary.txt; sed '/^$/d' <temporary.txt
else
sed '/^$/d' <$#
fi
Where am I going wrong?
You need spaces around [ and ]. In bash, [ is a command and you need spaces around it for bash to interpret it so.
You can also check for the presence of arguments by using (( ... )). So your script could be rewritten as:
if ((!$#)); then
cat > temporary.txt; sed '/^$/d' <temporary.txt
else
sed '/^$/d' "$#"
fi
If you want to use only the first argument, then you need to say $1 (and not $#).
Try using this
if [ $# -eq 0 ]; then
cat > temporary.txt; sed '/^$/d' <temporary.txt
else
cat $# | sed '/^$/d'
fi
A space is needed between [ and $# and your usage of $# is not good. $# represents all arguments and -eq is used to compare numeric values.
There are multiple problems here:
You need to leave a space between the square brackets [ ] and the variables.
When using a string type, you cannot use -eq, use == instead.
When using a string comparison you need to use double square brackets.
So the code should look like:
if [[ "$#" == "$NO_ARGS" ]]; then
cat > temporary.txt; sed '/^$/d' <temporary.txt
else
sed '/^$/d' <$#
fi
Or else use $# instead.
Instead of forcing user input to a file, I'd force the given file to stdin:
#!/bin/bash
if [[ $1 && -r $1 ]]; then
# it's a file
exec 0<"$1"
elif ! tty -s; then
: # input is piped from stdin
else
# get input from user
echo "No file specified, please enter your input, ctrl-D to end"
fi
# now, let sed read from stdin
sed '/^$/d'

shell script grep to grep a string

The output is blank fr the below script. What is it missing? I am trying to grep a string
#!/bin/ksh
file=$abc_def_APP_13.4.5.2
if grep -q abc_def_APP $file; then
echo "File Found"
else
echo "File not Found"
fi
In bash, use the <<< redirection from a string (a 'Here string'):
if grep -q abc_def_APP <<< $file
In other shells, you may need to use:
if echo $file | grep -q abc_def_APP
I put my then on the next line; if you want your then on the same line, then add ; then after what I wrote.
Note that this assignment:
file=$abc_def_APP_13.4.5.2
is pretty odd; it takes the value of an environment variable ${abc_def_APP_13} and adds .4.5.2 to the end (it must be an env var since we can see the start of the script). You probably intended to write:
file=abc_def_APP_13.4.5.2
In general, you should enclose references to variables holding file names in double quotes to avoid problems with spaces etc in the file names. It is not critical here, but good practices are good practices:
if grep -q abc_def_APP <<< "$file"
if echo "$file" | grep -q abc_def_APP
Yuck! Use the shell's string matching
if [[ "$file" == *abc_def_APP* ]]; then ...

How to grep a string in until loop in bash?

I work on a script compressing files. I want to do an 'until loop' til' the content of variable matches the pattern. The script is using zenity. This is the major part:
part="0"
pattern="^([0-9]{1}[0-9]*([km])$"
until `grep -E "$pattern" "$part"` ; do
part=$(zenity --entry \
--title="Zip the file" \
--text "Choose the size of divided parts:
(0 = no division, *m = *mb, *k = *kb)" \
--entry-text "0");
if grep -E "$pattern" "$part" ; then
zenity --warning --text "Wrong text entry, try again." --no-cancel;
fi
done
I want it to accept string containing digits ended with 'k' or 'm' (but not both of them) and don't accept string started with '0'.
Is the pattern ok?
$ grep -w '^[1-9][0-9]*[km]$' <<< 45k
45k
$ grep -w '^[1-9][0-9]*[km]$' <<< 001023m
$ grep -w '^[1-9][0-9]*[km]$' <<< 1023m
1023m
Don't forget the <<< in your expression, you're not grep'ing a file, but a string. To be more POSIX-compliant, you can also use:
echo 1023m | grep -w '^[1-9][0-9]*[km]$'
But it is kinda ugly.
Edit:
Longer example:
initmessage="Choose the size of divided parts:\n(0 = no division, *m = *mb, *k = *kb)"
errmessage="Wrong input. Please re-read carefully the following:\n\n$initmessage"
message="$initmessage"
while true ; do
part=$(zenity --entry \
--title="Zip the file" \
--text "$message")
if grep -qw '^[1-9][0-9]*[km]$' <<< "$part" ; then
zenity --info --text 'Thank you !'
break
else
message="$errmessage"
fi
done
Also, this is not directly related to the question, but you may want to have a look at Yad, which does basically the same things Zenity does, but has more options. I used it a lot when I had to write Bash scripts, and found it much more useful than Zenity.
You don't want the back-quotes in the until line. You might write:
until grep -E "$pattern" "$part"
do
...body of loop...
done
Or you might add arguments to grep to suppress the output (or send the output to /dev/null). As written, the script tries to execute the output of the grep command and use the success/failure status of that (not the grep per se) as an indication of whether to continue the loop or not.
Additionally, your pattern needs some work. It is:
pattern="^([0-9]{1}[0-9]*([km])$"
There is an unmatched open parenthesis in there. It also looks to me as though it is trying to allow a leading zero. You probably want:
pattern='^[1-9][0-9]*[km]$'
Single quotes are generally safer than double quotes for things like regular expressions.
I just want to check if my variable called part is well-formed after writing it in Zenity entry dialog. I just realised that grep needs a file, but my part is a variable initialised in this script. How to get along now?
In bash, you can use the <<< operator to redirect from a string:
until grep -E "$pattern" <<< "$part"
In most other shells, you'd write:
until echo "$part" | grep -E "$pattern"
This also works in bash, of course.

Resources