grep behavior in cli and script differs - bash

I've got a web server with a bunch of domains that have different name servers and I'm trying to clean up the mess. I'm trying to get a list of the domains and their name servers. I've got this simple script written and it almost works:
#!/bin/bash
for f in `cat mydomains.txt`
do
echo $f " " >> mydns.txt
dig ns $f | grep '^$f' | cut -d $'\t' -f 5 >> mydns.txt
echo "" >> mydns.txt
done
As of right now, all this does is echo $f " " >> mydns.txt.
If I take the dig line and substitute what $f should be in the command line, I get the expected results. However, I get nothing in my script. I know that the $f variable is populated because it echoes $f in the previous line. Why doesn't it work in the script ?

Did you mean to grep for '^$f'? It should be '^'"$f", i.e., without the ticks around the variable. It won't be expanded that way.

Single quotes prevent the shell from expanding the variable. You should use double quotes:
dig ns "$f" | grep "^$f" | ...
Note that I also used double quotes around $f in the dig call. Quoting your variables is good practice; see this other SO question for details.

It's possible that it's a Windows-style/Unix-style newline issue. It might help to convert your newlines to \n instead of \r\n.
Have you tried the dos2unix suggestion given here: grep fails inside bash script but works on command line?

Related

Some tips to improve a bash script for count fastq files

Hi guys I got this bash one line that i wish to make a script
for i in 'ls *.fastq.gz'; do echo $(zcat ${i} | wc -l)/4|bc; done
I would like to make it as a script to read from a data dir and print out the result with the name of the file.
I tried to put the dir in front of the 'data/*.fastq.gz' but got am error No such dir exist...
I would like some like this:
name1.fastq.gz 1898516
name2.fastq.gz 2467421
namen.fastq.gz 1234532
I am not experienced in bash.
Could you guys give a help?
Thanks
Take the dir as an argument, but default to the current dir if it's not set.
dir="${1-.}"
Then put it in the glob: "$dir"/*.fastq.gz
As well:
Quote variables and command expansions.
Don't parse ls.
Don't trust echo with arbitrary data (filenames). Use printf instead.
Use an end-of-options flag -- when giving filenames to commands.
I prefer to not have any inline command expansions, but that's just personal preference
Putting it together:
#!/bin/bash
dir="${1-.}"
for file in "$dir"/*.fastq.gz; do
printf '%s ' "$file"
lines="$(zcat -- "$file" | wc -l)"
bc <<< "$lines/4" # Using a here-string (Bash feature)
done
There is no need to escape to bc for integer math (divide by 4), or to use 'ls' to enumerate the files. The original version will do with minor changes:
#!/bin/bash
dir="${1-.}"
for i in "$dir"/*.fastq.gz; do
lines=$(zcat "${i}" | wc -l)
printf '%s %d\n' "$i" "$((lines/4))"
done

error because eval command seems to remove backslashes in stored command

I want to be able to add newline characters before every occurences of some tokens appearing in some .tex files that I possess, some of those tokens are '\itemQ', '\pagebreakQ'. I created a procedure that ends up creating a command for sed stored in $sedInpt:
~$ echo "$sedInpt"
-e s/\(\\itemQ\)/\n\1/ -e s/\(\\pagebreakQ\)/\n\1/
I want to use "$sedInpt" as a command for sed:
echo "$inputText" | eval "sed ${sedInpt}"
but if I do the following as a test:
echo 'hello\itemQ' | eval "sed ${sedInpt}"
hello\itemQ
you can see there ain't any newline that has been added before \itemQ.
So I've tried debugging this way of doing thing by calling bash -x to see what's happened in detail:
~$ bash -x
~$ echo "hello\itemQ" | eval "sed ${sedInpt}"
+ echo 'hello\itemQ'
+ eval 'sed -e s/\(\\itemQ\)/\n\1/ -e s/\(\\pagebreakQ\)/\n\1/'
++ sed -e 's/(\itemQ)/n1/' -e 's/(\pagebreakQ)/n1/'
hello\itemQ
you can see that the backslashes of \n and \1 and even the ones before ( and ) that I had placed in "$sedInpt" seem to have disappeared when parsed by eval.
So I am bit lost on what to do next to do what I want.. any ideas?
You could also just combine them into a single command, which in my opinion is more straightforward:
$ cat /tmp/sed.sh
sedInpt='s/\(\\itemQ\)/\n\1/; s/\(\\pagebreakQ\)/\n\1/'
echo "hello\itemQ" | sed "$sedInpt"
$ /tmp/sed.sh
hello
\itemQ
Edit: As #123 rightly points out, storing commands in variables is dangerous and should be avoided if possible. If you have complete control over what is stored, it should be safe, but if it comes from any sort of user input, it is a "Command Injection" vulnerability.
Following #Inian advice I managed to achieve what I wanted to do in this way:
~$ sedInpt=( -e 's/\(\\itemQ\)/\n\1/' -e 's/\(\\pagebreakQ\)/\n\1/' )
~$ echo "hello\itemQ" | sed "${sedInpt[#]}"
hello
\itemQ

Escaping Shebang in grep

in a shell script, i'm trying to find out if another file is a shell script. i'm doing that by grepping the shebang line. but my grep statement doesn't work:
if [[ $($(cat $file) | grep '^#! /bin' | wc -l) -gt 0 ]]
then
echo 'file is a script!'
else
echo "no script"
fi
i always get the error "bash: #!: command not found". i tried several things to escape the shebang but that didn't work.
maybe you can help me with that? :)
cheers,
narf
I would suggest that you change your condition to this:
if grep -q '^#! */bin' "$file"
The -q option to grep is useful in this case as it tells grep to produce no output, exiting successfully if the pattern is matched. This can be used with if directly; there's no need to wrap everything in a test [[ (and especially no need for a useless use of cat).
I also modified your pattern slightly so that the space between #! and /bin is optional.
It's worth noting that this will produce false positives in cases where the match is on a different line of the file, or when another shebang is used. You could work around the first issue by piping head -n 1 to grep, so that only the first line would be checked:
if head -n 1 "$file" | grep -q '^#! */bin'
If you are searching for a known list of shebangs, e.g. /bin/sh and /bin/bash, you could change the pattern to something like ^#! */bin/\(sh\|bash\).

Very weird behavior using sed

I have a big problem doing a script: basically, I read a line from files.
All lines are made of 3 to 8 characters contiguous (no space).
Then I used sed to replace those lines inside a pattern (aka "var" in my minimal script below)
var="iao"
for m in `more meshing/junction_names.txt`
do
echo $m
echo -n $m | xxd -ps | sed 's/[[:xdigit:]]\{2\}/\\x&/g'
echo $var |sed "s/a/b/"
echo $var |sed "s/a/$m/"
done
Now these are the first 3 record of my output (they are all the same anyway).
I am using linux. According kate, all files are encoded UTF-8. Very weird huh? Any idea why that is is welcome.
J_LEAK
\x4a\x5f\x4c\x45\x41\x4b\x0d
ibo
oJ_LEAK
JO_1
\x4a\x4f\x5f\x31\x0d
ibo
oJO_1
JPL2_F
\x4a\x50\x4c\x32\x5f\x46\x0d
ibo
oJPL2_F
JF_PL2
Your input file contains DOS carriage returns (or possibly, the absurd attempt to read it with more introduces them). The hex dump shows this clearly; every value ends with \x0d which translates to a control code which causes the terminal to jump the cursor back to the beginning of the line.
This is a massive FAQ and you can find many examples of how to troubleshoot this basic problem, including in the bash tag wiki.
Tangentially, you should always quote strings unless you specifically require the shell to perform wildcard expansion and whitespace tokenization on the value; and Bash has built-ins to avoid the inelegant and somewhat error-prone echo | sed. Finally, don't read lines with for.
var="iao"
tr -d '\015' <meshing/junction_names.txt |
while read -r m; do # don't use a for loop
echo "$m" # quote!
echo -n "$m" | xxd -ps | sed 's/[[:xdigit:]]\{2\}/\\x&/g'
echo "${var/a/b}" # quote; use Bash built-in substitution mechanism
echo "${var/a/$m}"
done
Perhaps you want to remove the carriage returns once and for all, and then just use while read .... done <fixed-file instead of the tr pipeline.

overwrite a file then append

I have a loop in my script that will append a list of email address's to a file "$CRN". If this script is executed again, it will append to this old list. I want it to overwrite with the new list rather then appending to the old list. I can submit my whole script if needed. I know I could test if "$CRN" exists then remove file, but I'm interested in some other suggestions? Thanks.
for arg in "$#"; do
if ls /students | grep -q "$arg"; then
echo "${arg}#mail.ccsf.edu">>$CRN
((students++))
elif ls /users | grep -q "$arg$"; then
echo "${arg}#ccsf.edu">>$CRN
((faculty++))
fi
Better do this :
CRN="/path/to/file"
:> "$CRN"
for arg; do
if printf '%s\n' /students/* | grep -q "$arg"; then
echo "${arg}#mail.ccsf.edu" >> "$CRN"
((students++))
elif printf '%s\n'/users/* | grep -q "${arg}$"; then
echo "${arg}#ccsf.edu" >> "$CRN"
((faculty++))
fi
done
don't parse ls output ! use bash glob instead. ls is a tool for interactively looking at file information. Its output is formatted for humans and will cause bugs in scripts. Use globs or find instead. Understand why: http://mywiki.wooledge.org/ParsingLs
"Double quote" every expansion, and anything that could contain a special character, eg. "$var", "$#", "${array[#]}", "$(command)". See http://mywiki.wooledge.org/Quotes http://mywiki.wooledge.org/Arguments and http://wiki.bash-hackers.org/syntax/words
take care to false positives like arg=foo and glob : foobar, that will match. You need grep -qw then if you want word boundaries. UP2U

Resources