How would one grep backtick from files in a for-loop.
I would like to run grep for a pattern '`define'. The pattern works in standalone grep command but fails in for-loop.
foreach xxx ( `grep -r '`define' $idirectory --no-filename | sed -e 's ; //.* ; ; ' -e 's ; #.* ; ; ' -e 's ; ^\s* ; ; ' | grep -v ^$ | sort -n | awk '{print $2}' | uniq -d`)
echo $xxx
end
The backticks are conflicting in the for-loop.
regards
Srisurya
Simply, don't use ' and escape the backtick with backshlash.
So, the next didn't works:
grep -r '`def' *
and prints
No matching command
But this:
grep -r \`def *
works and prints
ewdwedwe `define`
So, simiarly for your script, the next works (file btick.tcsh):
#!/bin/tcsh
set greparg = \`def
foreach xxx ( `grep -l $greparg *` )
echo ===$xxx===
end
and pruduces the next result
===btick.tcsh===
===btick1.txt===
===btick2.txt===
the content of btick.txt files:
btick1 `def`
This is an alternate solution.
Use of ASCII code for grep argument
grep -rP '\x60define' $idirectory
where \x60 is the ascii code for "`"
You should not use old and outdated back ticks, use parentheses like this $(code)
Try this:
for xxx in $(some code $(som more code)
echo "$xxx"
done
Nesting and back tics makes it complicated, it need to be escaped. Compare this to:
listing=`ls -l \`cat filenames.txt\``
vs
listing=$(ls -l $(cat filenames.txt))
Related
I'm new to shell scripting and I Can't understand those lines:
wc -l $x|sed 's/\s\+/|/g'
rc=`echo "$BTEQ_OUT"|grep "RC (return code)"| sed 's/ //g' | cut -d '=' -f2|tr -d "\r\n "`;
When you see a long pipeline, one useful technique for understanding it is to execute it piece by piece:
first, what's in $x?
echo $x
is that the name of a file?
ls -l $x
what does wc do?
wc -l $x
ok, what does the sed part do? (note, \s requires GNU sed)
wc -l $x | sed 's/\s\+/|/g'
Similarly:
echo "$BTEQ_OUT"
echo "$BTEQ_OUT"|grep "RC (return code)"
echo "$BTEQ_OUT"|grep "RC (return code)"| sed 's/ //g'
echo "$BTEQ_OUT"|grep "RC (return code)"| sed 's/ //g' | cut -d '=' -f2
echo "$BTEQ_OUT"|grep "RC (return code)"| sed 's/ //g' | cut -d '=' -f2|tr -d "\r\n ";
wc -l $x|sed 's/\s\+/|/g'
wc is a tools used for counting, with the -l flag, this will count the lines in a file or a string.
$x is the variable holding probably a file name to be passed into wc
| called 'pipe' passes the output of the command before as the input into the command after
sed is another scripting tool used to edit text in files.
's/\s\+/|/g' is regex which globally (g) substitutes any number of white space chars with pipe symbols '|'
This program does the following
Count how many lines are in $x and whatever you output replace empty characters with pipe symbols.
The fact that they expect multiple outputs from wc -l hints that $x might store more than one file ...
I'd suggest looking into what some of the other commands are and what they do, and how they interact. List below
echo
tr
cut
pipe
I'm trying to make a small function that removes all the chars that are not digits.
123a45a ---> will become ---> 12345
I've came up with :
temp=$word | grep -o [[:digit:]]
echo $temp
But instead of 12345 I get 1 2 3 4 5. How to I get rid of the spaces?
Pure bash:
word=123a45a
number=${word//[^0-9]}
Here's a pure bash solution
var='123a45a'
echo ${var//[^0-9]/}
12345
is this what you are looking for?
kent$ echo "123a45a"|sed 's/[^0-9]//g'
12345
grep & tr
echo "123a45a"|grep -o '[0-9]'|tr -d '\n'
12345
I would recommend using sed or perl instead:
temp="$(sed -e 's/[^0-9]//g' <<< "$word")"
temp="$(perl -pe 's/\D//g' <<< "$word")"
Edited to add: If you really need to use grep, then this is the only way I can think of:
temp="$( grep -o '[0-9]' <<< "$word" \
| while IFS= read -r ; do echo -n "$REPLY" ; done
)"
. . . but there's probably a better way. (It uses grep -o, like your solution, then runs over the lines that it outputs and re-outputs them without line-breaks.)
Edited again to add: Now that you've mentioned that you use can use tr instead, this is much easier:
temp="$(tr -cd 0-9 <<< "$word")"
What about using sed?
$ echo "123a45a" | sed -r 's/[^0-9]//g'
12345
As I read you are just allowed to use grep and tr, this can make the trick:
$ echo "123a45a" | grep -o [[:digit:]] | tr -d '\n'
12345
In your case,
temp=$(echo $word | grep -o [[:digit:]] | tr -d '\n')
tr will also work:
echo "123a45a" | tr -cd '[:digit:]'
# output: 12345
Grep returns the result on different lines:
$ echo -e "$temp"
1
2
3
4
5
So you cannot remove those spaces during the filtering, but you can afterwards, since $temp can transform itself like this:
temp=`echo $temp | tr -d ' '`
$ echo "$temp"
12345
I'm processing some data from a text file using a bash script (Ubuntu 12.10).
The basic idea is that I select a certain line from a file using grep. Next, I process the line to get the number with sed. Both the grep and sed command are working. I can echo the number.
But the concatenation of the result with a string goes wrong.
I get different results when combining string when I do a grep command from a variable or a file. The concatenation goes wrong when I grep a file. It works as expected when I grep a variable with the same text as in the file.
What am I doing wrong with the grep from a file?
Contents of test.pdb
REMARK overall = 324.88
REMARK bon = 24.1918
REMARK coup = 0
My script
#!/bin/bash
#Correct function
echo "Working code"
TEXT="REMARK overall = 324.88\nREMARK bon = 24.1918\nREMARK coup = 0\n"
DATA=$(echo -e $TEXT | grep 'overall' | sed -n -e "s/^.*= //p" )
echo "Data: $DATA"
DATA="$DATA;0"
echo $DATA
#Not working
echo ""
echo "Not working code"
DATA=$(grep 'overall' test.pdb | sed -n -e "s/^.*= //p")
echo "Data: $DATA"
DATA="$DATA;0"
echo $DATA
Output
Working code
Data: 324.88
324.88;0
Not working code
Data: 324.88
;04.88
I went crazy with the same issue.
The real problem is that your "test.pdb" has probably a wrong EOL (end of line) character.
Linux EOL: LF (aka \n)
Windows EOL: CR LF (aka \r \n)
This mean that echo and grep will have problem with this extra character (\r), luckily tr, sed and awk manage it correctly.
So you can try also with:
DATA=$(grep 'overall' test.pdb | sed -n -e "s/^.*= //p" | sed -e 2s/\r$//")
or
DATA=$(grep 'overall' test.pdb | sed -n -e "s/^.*= //p" | tr -d '\r')
With awk, it will be more reliable and cleaner I guess :
$ awk '$2=="overall"{print "Working code\nData: " $4 "\n" $4 ";0"}' file.txt
Working code
Data: 324.88
324.88;0
Try this:
SUFFIX=";0"
DATA="${DATA}${SUFFIX}"
I would like to use the lines coming from 'wc' as variables. For example:
echo 'foo bar' > file.txt
echo 'blah blah blah' >> file.txt
wc file.txt
2 5 23 file.txt
I would like to have something like $lines, $words and $characters associated to the values 2, 5, and 23. How can I do that in bash?
In pure bash: (no awk)
a=($(wc file.txt))
lines=${a[0]}
words=${a[1]}
chars=${a[2]}
This works by using bash's arrays. a=(1 2 3) creates an array with elements 1, 2 and 3. We can then access separate elements with the ${a[indice]} syntax.
Alternative: (based on gonvaled solution)
read lines words chars <<< $(wc x)
Or in sh:
a=$(wc file.txt)
lines=$(echo $a|cut -d' ' -f1)
words=$(echo $a|cut -d' ' -f2)
chars=$(echo $a|cut -d' ' -f3)
There are other solutions but a simple one which I usually use is to put the output of wc in a temporary file, and then read from there:
wc file.txt > xxx
read lines words characters filename < xxx
echo "lines=$lines words=$words characters=$characters filename=$filename"
lines=2 words=5 characters=23 filename=file.txt
The advantage of this method is that you do not need to create several awk processes, one for each variable. The disadvantage is that you need a temporary file, which you should delete afterwards.
Be careful: this does not work:
wc file.txt | read lines words characters filename
The problem is that piping to read creates another process, and the variables are updated there, so they are not accessible in the calling shell.
Edit: adding solution by arnaud576875:
read lines words chars filename <<< $(wc x)
Works without writing to a file (and do not have pipe problem). It is bash specific.
From the bash manual:
Here Strings
A variant of here documents, the format is:
<<<word
The word is expanded and supplied to the command on its standard input.
The key is the "word is expanded" bit.
lines=`wc file.txt | awk '{print $1}'`
words=`wc file.txt | awk '{print $2}'`
...
you can also store the wc result somewhere first.. and then parse it.. if you're picky about performance :)
Just to add another variant --
set -- `wc file.txt`
chars=$1
words=$2
lines=$3
This obviously clobbers $* and related variables. Unlike some of the other solutions here, it is portable to other Bourne shells.
I wanted to store the number of csv file in a variable. The following worked for me:
CSV_COUNT=$(ls ./pathToSubdirectory | grep ".csv" | wc -l | xargs)
xargs removes the whitespace from the wc command
I ran this bash script not in the same folder as the csv files. Thus, the pathToSubdirectory
You can assign output to a variable by opening a sub shell:
$ x=$(wc some-file)
$ echo $x
1 6 60 some-file
Now, in order to get the separate variables, the simplest option is to use awk:
$ x=$(wc some-file | awk '{print $1}')
$ echo $x
1
declare -a result
result=( $(wc < file.txt) )
lines=${result[0]}
words=${result[1]}
characters=${result[2]}
echo "Lines: $lines, Words: $words, Characters: $characters"
Suppose I have the string 1:2:3:4:5 and I want to get its last field (5 in this case). How do I do that using Bash? I tried cut, but I don't know how to specify the last field with -f.
You can use string operators:
$ foo=1:2:3:4:5
$ echo ${foo##*:}
5
This trims everything from the front until a ':', greedily.
${foo <-- from variable foo
## <-- greedy front trim
* <-- matches anything
: <-- until the last ':'
}
Another way is to reverse before and after cut:
$ echo ab:cd:ef | rev | cut -d: -f1 | rev
ef
This makes it very easy to get the last but one field, or any range of fields numbered from the end.
It's difficult to get the last field using cut, but here are some solutions in awk and perl
echo 1:2:3:4:5 | awk -F: '{print $NF}'
echo 1:2:3:4:5 | perl -F: -wane 'print $F[-1]'
Assuming fairly simple usage (no escaping of the delimiter, for example), you can use grep:
$ echo "1:2:3:4:5" | grep -oE "[^:]+$"
5
Breakdown - find all the characters not the delimiter ([^:]) at the end of the line ($). -o only prints the matching part.
You could try something like this if you want to use cut:
echo "1:2:3:4:5" | cut -d ":" -f5
You can also use grep try like this :
echo " 1:2:3:4:5" | grep -o '[^:]*$'
One way:
var1="1:2:3:4:5"
var2=${var1##*:}
Another, using an array:
var1="1:2:3:4:5"
saveIFS=$IFS
IFS=":"
var2=($var1)
IFS=$saveIFS
var2=${var2[#]: -1}
Yet another with an array:
var1="1:2:3:4:5"
saveIFS=$IFS
IFS=":"
var2=($var1)
IFS=$saveIFS
count=${#var2[#]}
var2=${var2[$count-1]}
Using Bash (version >= 3.2) regular expressions:
var1="1:2:3:4:5"
[[ $var1 =~ :([^:]*)$ ]]
var2=${BASH_REMATCH[1]}
$ echo "a b c d e" | tr ' ' '\n' | tail -1
e
Simply translate the delimiter into a newline and choose the last entry with tail -1.
Using sed:
$ echo '1:2:3:4:5' | sed 's/.*://' # => 5
$ echo '' | sed 's/.*://' # => (empty)
$ echo ':' | sed 's/.*://' # => (empty)
$ echo ':b' | sed 's/.*://' # => b
$ echo '::c' | sed 's/.*://' # => c
$ echo 'a' | sed 's/.*://' # => a
$ echo 'a:' | sed 's/.*://' # => (empty)
$ echo 'a:b' | sed 's/.*://' # => b
$ echo 'a::c' | sed 's/.*://' # => c
There are many good answers here, but still I want to share this one using basename :
basename $(echo "a:b:c:d:e" | tr ':' '/')
However it will fail if there are already some '/' in your string.
If slash / is your delimiter then you just have to (and should) use basename.
It's not the best answer but it just shows how you can be creative using bash commands.
If your last field is a single character, you could do this:
a="1:2:3:4:5"
echo ${a: -1}
echo ${a:(-1)}
Check string manipulation in bash.
Using Bash.
$ var1="1:2:3:4:0"
$ IFS=":"
$ set -- $var1
$ eval echo \$${#}
0
echo "a:b:c:d:e"|xargs -d : -n1|tail -1
First use xargs split it using ":",-n1 means every line only have one part.Then,pring the last part.
Regex matching in sed is greedy (always goes to the last occurrence), which you can use to your advantage here:
$ foo=1:2:3:4:5
$ echo ${foo} | sed "s/.*://"
5
A solution using the read builtin:
IFS=':' read -a fields <<< "1:2:3:4:5"
echo "${fields[4]}"
Or, to make it more generic:
echo "${fields[-1]}" # prints the last item
for x in `echo $str | tr ";" "\n"`; do echo $x; done
improving from #mateusz-piotrowski and #user3133260 answer,
echo "a:b:c:d::e:: ::" | tr ':' ' ' | xargs | tr ' ' '\n' | tail -1
first, tr ':' ' ' -> replace ':' with whitespace
then, trim with xargs
after that, tr ' ' '\n' -> replace remained whitespace to newline
lastly, tail -1 -> get the last string
For those that comfortable with Python, https://github.com/Russell91/pythonpy is a nice choice to solve this problem.
$ echo "a:b:c:d:e" | py -x 'x.split(":")[-1]'
From the pythonpy help: -x treat each row of stdin as x.
With that tool, it is easy to write python code that gets applied to the input.
Edit (Dec 2020):
Pythonpy is no longer online.
Here is an alternative:
$ echo "a:b:c:d:e" | python -c 'import sys; sys.stdout.write(sys.stdin.read().split(":")[-1])'
it contains more boilerplate code (i.e. sys.stdout.read/write) but requires only std libraries from python.