I'm trying to write a small script that either takes input from a file or from user, then it gets rid of any blank lines from it.
I'm trying to make it so that if there is no file name specified it will prompt the user for input. Also is the best way to output the manual input to a file then run the code or to store it in a variable?
So far I have this but when I run it with a file it give 1 line of error before returning the output I want. The error says ./deblank: line 1: [blank_lines.txt: command not found
if [$# -eq "$NO_ARGS"]; then
cat > temporary.txt; sed '/^$/d' <temporary.txt
else
sed '/^$/d' <$#
fi
Where am I going wrong?
You need spaces around [ and ]. In bash, [ is a command and you need spaces around it for bash to interpret it so.
You can also check for the presence of arguments by using (( ... )). So your script could be rewritten as:
if ((!$#)); then
cat > temporary.txt; sed '/^$/d' <temporary.txt
else
sed '/^$/d' "$#"
fi
If you want to use only the first argument, then you need to say $1 (and not $#).
Try using this
if [ $# -eq 0 ]; then
cat > temporary.txt; sed '/^$/d' <temporary.txt
else
cat $# | sed '/^$/d'
fi
A space is needed between [ and $# and your usage of $# is not good. $# represents all arguments and -eq is used to compare numeric values.
There are multiple problems here:
You need to leave a space between the square brackets [ ] and the variables.
When using a string type, you cannot use -eq, use == instead.
When using a string comparison you need to use double square brackets.
So the code should look like:
if [[ "$#" == "$NO_ARGS" ]]; then
cat > temporary.txt; sed '/^$/d' <temporary.txt
else
sed '/^$/d' <$#
fi
Or else use $# instead.
Instead of forcing user input to a file, I'd force the given file to stdin:
#!/bin/bash
if [[ $1 && -r $1 ]]; then
# it's a file
exec 0<"$1"
elif ! tty -s; then
: # input is piped from stdin
else
# get input from user
echo "No file specified, please enter your input, ctrl-D to end"
fi
# now, let sed read from stdin
sed '/^$/d'
Related
I'm having trouble in a script and i want to know if it's possible to store the path of the matching result of a grep ?
I'm on RHEL 7, the script is a check of the rsyslog.conf file which complete or add the correct value to a parameter (CIS rhel7 benchmark, part 4.2.1.3).
Full script so far :
#!/bin/bash
if grep "^\$FileCreateMode" /etc/rsyslog.conf /etc/rsyslog.d/*.conf
then
read -p "Is $FileCreateMode superior or equal to 0640 ? [y/n]" rep
if [ $rep == "y" ]
then
echo "No action needed"
else
read -p "Enter the new $FileCreateMode value (0640 recommanded)" rep2
sed -i "/^\$FileCreateMode/ $rep2"
echo "$FileCreateMode new value is now $rep2"
fi
else
echo "$FileCreateMode doesn't exist in rsyslog conf files"
read -p "What's the path of the file to modify ?(Press [ENTER] for default /etc/rsyslog.conf)" path
if [ $path -z ]
then
echo "$FileCreateMode 0640" >> /etc/rsyslog.conf
else
echo "$FileCreateMode 0640" >> $path
fi
fi
So my problem is on the sed at the 11th line.
Am i able to get the right path if my grep on 3rd line matched into a variable to reuse it on the 11th.
And i'm struggling with the same sed because i want him to replace the value after $FileCreateMode but it keep changing the $FileCreateMode string.
i've tried this syntax too but i still don't get the result i want
sed -i -e "s,^\($FileCreateMode[ ]*\).*,\1 0640 ,g" /etc/rsyslog.conf
Thanks in advance for any help you can bring, and have a good day :)
Edit :
As requested i'm simplifying here.
I want to grep $FileCreateMode in /etc/rsyslog.conf and /etc/rsyslog.d/*.conf and i'm trying to get the destination file (could be rsyslog.conf but it can be testpotato.conf in rsyslog.d) into a variable (like $var) to be able to use the path in my sed on the 11th line like
sed -i "/^\$FileCreateMode/ 0640" $var
And for the sed problem when i execute this command i would like to have something like
old : $FileCreateMode 0777
sed -i "/^\$FileCreateMode/ 0640" $var
new : $FileCreateMode 0640
But instead i get
old : $FileCreateMode 0777
sed -i "/^\$FileCreateMode/ 0640" $var
new : 0640 ($FileCreateMode is deleted)
hope i'm more understable, thanks again and feel free to ask for more details
Use $() to assign the results of grep into a variable, and then use a for loop to process files one by one:
# Assign grep results to FILES
FILES=$(grep -l '^$FileCreateMode' /etc/rsyslog.conf /etc/rsyslog.d/*.conf)
# Check if FILES variable is not empty
if [[ -n ${FILES} ]]; then
# Loop through all the files
for file in ${FILES}; do
# ...
sed -iE "s/^(\\\$FileCreateMode\s+)[[:digit:]]+/\1${rep2}/" ${file}
# ...
done
else
# OP's logic for when $FileCreateMode doesn't exist in any of the files
sed fix:
Notice that I've also updated your sed expression (above). You were very close, but you had to double escape the dollar sign: once for using it inside "", and once so that it isn't interpreted as END_OF_LINE in the regex.
If your grep supports -H, you could just do:
while grep -H "^\$FileCreateMode" /etc/rsyslog.conf /etc/rsyslog.d/*.conf \
| IFS=: read path line; do
# Here, $path is the path to the file that matches
# and $line is the line that matched.
done
If you prefer, you can use if ...; then instead of while ...; do Be aware that the variables will lose their value after the subshell terminates.
I have a file files.txt with content:
F1.txt
F2.txt
F3.txt
F4.txt
I need to read the file files.txt line by line, and check if it exists in the current directory, if it does, I need to append the date at the end of each of line of files.txt, so that the output should be
F1.txt16032017
F2.txt16032017
F3.txt16032017
F4.txt16032017
I have used following simple shell script.
#!/bin/bash
DT=`date +%d%m%Y`
while IFS=read -r line
do
if [ -f $line]
then
echo "$line$DT" > files.ok
else
echo "$line" > files.notok
fi
done < files.txt
It executes without any error, but does not provide expected output with date append. Can someone tell me if the file existence test is correct
The problem is with your file write operator > which creates a creates a new file on every successful case, you need to have used the >> operator which appends to the existing file.
A much neater approach to your code would be to do
#!/bin/bash
dateToday="$(date +%d%m%Y)"
while IFS= read -r file; do
[ -f "$file" ] && printf "%s\n" "$file$dateToday" >> files.OK || printf "%s\n" "$file" >> files.NOK
done < files.txt
What updates that I have made to improve the script,
Removed the outdated command substitution syntax using backticks `` and used the $(..) for running them.
Double-quoted the variables, lower-cased the local variables.
Fixed the file write operator from > to >>
Used a single line condition making use of the return code of the test ([]) operator. The command after && runs if the condition [ -f "$file" ] is successful and the command after || runs if the condition fails.
This part of my script is comparing each line of a file to find a preset string. If the string does NOT exist as a line in the file, it should append it to the end of the file.
STRING=foobar
cat "$FILE" | while read LINE
do
if [ "$STRING" == "$LINE" ]; then
export ISLINEINFILE="yes"
fi
done
if [ ! "$ISLINEINFILE" == yes ]; then
echo "$LINE" >> "$FILE"
fi
However, it appears as if both $LINE and $ISLINEINFILE are both cleared upon finishing the do loop. How can I avoid this?
Using shell
If we want to make just the minimal change to your code to get it working, all we need to do is switch the input redirection:
string=foobar
while read line
do
if [ "$string" == "$line" ]; then
islineinfile="yes"
fi
done <"$file"
if [ ! "$islineinfile" == yes ]; then
echo "$string" >> "$file"
fi
In the above, we changed cat "$file" | while do ...done to while do...done<"$file". With this one change, the while loop is no longer in a subshell and, consequently, shell variables created in the loop live on after the loop completes.
Using sed
I believe that the whole of your script can be replaced with:
sed -i.bak '/^foobar$/H; ${x;s/././;x;t; s/$/\nfoobar/}' file*
The above adds line foobar to the end of each file that doesn't already have a line that matches ^foobar$.
The above shows file* as the final argument to sed. This will apply the change to all files matching the glob. You could list specific files individually if you prefer.
The above was tested on GNU sed (linux). Minor modifications may be needed for BSD/OSX sed.
Using GNU awk (gawk)
awk -i inplace -v s="foobar" '$0==s{f=1} {print} ENDFILE{if (f==0) print s; f=0}' file*
Like the sed command, this can tackle multiple files all in one command.
Why does my variable set in a do loop disappear?
It disappears because it is set in a shell pipeline component. Most shells run each part of a pipeline in a subshell. By Unix design, variables set in a subshell cannot affect their parent or any already running other shell.
How can I avoid this?
There are several ways:
The simplest is to use a shell that doesn't run the last component of a pipeline in a subshell. This is ksh default behavior, e.g. use that shebang:
#!/bin/ksh
This behavior can also be bash one when the lastpipe option is set:
shopt -s lastpipe
You might use the variable in the same subshell that set it. Note that your original script indentation is wrong and might lead to the incorrect assumption that the if block is inside the pipeline, which isn't the case. Enclosing the whole block with parentheses will rectify that and would be the minimal change (two extra characters) to make it working:
STRING=foobar
cat "$FILE" | ( while read LINE
do
if [ "$STRING" == "$LINE" ]; then
export ISLINEINFILE="yes"
fi
done
if [ ! "$ISLINEINFILE" == yes ]; then
echo "$LINE" >> "$FILE"
fi
)
The variable would still be lost after that block though.
You might simply avoid the pipeline, which is straigthforward in your case, the cat being unnecessary:
STRING=foobar
while read LINE
do
if [ "$STRING" == "$LINE" ]; then
export ISLINEINFILE="yes"
fi
done < "$FILE"
if [ ! "$ISLINEINFILE" == yes ]; then
echo "$LINE" >> "$FILE"
fi
You might use another argorithmic approach, like using sed or gawk as suggested by John1024.
See also https://unix.stackexchange.com/a/144137/2594 for standard compliance details.
The output is blank fr the below script. What is it missing? I am trying to grep a string
#!/bin/ksh
file=$abc_def_APP_13.4.5.2
if grep -q abc_def_APP $file; then
echo "File Found"
else
echo "File not Found"
fi
In bash, use the <<< redirection from a string (a 'Here string'):
if grep -q abc_def_APP <<< $file
In other shells, you may need to use:
if echo $file | grep -q abc_def_APP
I put my then on the next line; if you want your then on the same line, then add ; then after what I wrote.
Note that this assignment:
file=$abc_def_APP_13.4.5.2
is pretty odd; it takes the value of an environment variable ${abc_def_APP_13} and adds .4.5.2 to the end (it must be an env var since we can see the start of the script). You probably intended to write:
file=abc_def_APP_13.4.5.2
In general, you should enclose references to variables holding file names in double quotes to avoid problems with spaces etc in the file names. It is not critical here, but good practices are good practices:
if grep -q abc_def_APP <<< "$file"
if echo "$file" | grep -q abc_def_APP
Yuck! Use the shell's string matching
if [[ "$file" == *abc_def_APP* ]]; then ...
#!/bin/bash
fname=$2
rname=$1
echo "$(<$fname)" | while read line ; do
result=`echo "$(<$rname)" | grep "$line"; echo $?`
if [ $result != 0 ]
then
sed '/$line/d' $fname > newkas
fi 2> /dev/null
done
Hi all, i am new to bash.
i have two lists one older than another. I wish to compare the names on 'fname' against 'rname'. 'Result' is the standard out put which i will get if the name is still available in 'rname'. if is not then i will get the non-zero output.
Using sed to delete that line and re route it to a new file.
I have tried part by part of the code and it works until i add in the while loop function. sed don't seems to work as the final output of 'newkas' is the same as the initial input 'fname'.
Is my method wrong or did i miss out any parts?
Part 1: What's wrong
The reason your sed expression "doesn't work" is because you used single quotes. You said
sed '/$line/d' $fname > newkas
Supposing fname=input.txt' and line='example text' this will expand to:
sed '/$line/d' input.txt > newkas
Note that $line is still literally present. This is because bash will not interpolate variables inside single quotes, thus sed sees the $ literally.
You could fix this by saying
sed "/$line/d/" $fname > newkas
Because inside double quotes the variable will expand. However, if your sed expression becomes more complicated you could run into difficulty in cases where bash interprets things which you intended to be interpreted by sed. I tend to use the form
sed '/'"$line"'/d/' $fname > newkas
Which is a bit harder to read but, if you look carefully, single-quotes everything I intend to be part of the sed expression and double quotes the variable I want to expand.
Part 2: How to improve it
Your script contains a number things which could be improved.
echo "$(<$fname)" | while read line ; do
:
done
In the first place you're reading the file with "$(<$fname)" when you could just redirect the stdin of the while loop. This is a bit redundant, but more importantly you're piping to while, which creates an extra subshell and means you can't modify any variables from the enclosing scope. Better to say
while IFS= read -r line ; do
:
done < "$fname"
Next, consider your grep
echo "$(<$rname)" | grep "$line"
Again you're reading the file and echoing it to grep. But, grep can read files directly.
grep "$line" "$rname"
Afterwards you echo the return code and check its value in an if statement, which is a classic useless construct.
result=$( grep "$line" "$rname" ; echo $?)
Instead you can just pass grep directly to if, which will test its return code.
if grep -q "$line" "$rname" ; then
sed "/$line/d" "$fname" > newkas
fi
Note here that I have quoted $fname, which is important if it might ever contain a space. I have also added -q to grep, which suppresses its output.
There's now no need to suppress error messages from the if statement, here, because we don't have to worry about $result containing an unusual value or grep not returning properly.
The final result is this script
while IFS= read -r line ; do
if grep -q "$line" "$rname" ; then
sed "/$line/d" "$fname" > newkas
fi
done < "$fname"
Which will not work, because newkas is overwritten on every loop. This means that in the end only the last line in $fname was used. Instead you could say:
cp "$fname" newkas
while IFS= read -r line ; do
if grep -q "$line" "$rname" ; then
sed -i '' "/$line/d" newkas
fi
done < "$fname"
Which, I believe, will do what you expect.
Part 3: But don't do that
But this is all tangential to solving your actual problem. It appears to me that you want to simply create a file newkas which contains the all the lines of $fname except those that appear in $rname. This is easily done with the comm utility:
comm -2 -3 <(sort "$fname") <(sort "$rname") > newkas
This also changes the sort order of the lines, which may not be good for you. If you want to do it without changing the ordering then using the method #fge suggests is best.
grep -F -v -x -f "$rname" "$fname"
If I understand your need correctly, you want a file newaks which contains the lines in $fname which are also in $rname.
If this is what you want, using sed is overkill. Use fgrep:
fgrep -x -f $fname $rname > newkas
Also, there are problems with your script:
you capture the output of grep in result, which means it will never be exactly 0; what you want is executing the command and simply check for $?
your echoes are convoluted, just do grep whatever thefilename, or while...done <thefile;
finally, you take the line as is from the source file: the line can potentially be a regex, which means you will try and match a regex in $rname, which may yield to unexpected results.
And others.