I'm processing some data from a text file using a bash script (Ubuntu 12.10).
The basic idea is that I select a certain line from a file using grep. Next, I process the line to get the number with sed. Both the grep and sed command are working. I can echo the number.
But the concatenation of the result with a string goes wrong.
I get different results when combining string when I do a grep command from a variable or a file. The concatenation goes wrong when I grep a file. It works as expected when I grep a variable with the same text as in the file.
What am I doing wrong with the grep from a file?
Contents of test.pdb
REMARK overall = 324.88
REMARK bon = 24.1918
REMARK coup = 0
My script
#!/bin/bash
#Correct function
echo "Working code"
TEXT="REMARK overall = 324.88\nREMARK bon = 24.1918\nREMARK coup = 0\n"
DATA=$(echo -e $TEXT | grep 'overall' | sed -n -e "s/^.*= //p" )
echo "Data: $DATA"
DATA="$DATA;0"
echo $DATA
#Not working
echo ""
echo "Not working code"
DATA=$(grep 'overall' test.pdb | sed -n -e "s/^.*= //p")
echo "Data: $DATA"
DATA="$DATA;0"
echo $DATA
Output
Working code
Data: 324.88
324.88;0
Not working code
Data: 324.88
;04.88
I went crazy with the same issue.
The real problem is that your "test.pdb" has probably a wrong EOL (end of line) character.
Linux EOL: LF (aka \n)
Windows EOL: CR LF (aka \r \n)
This mean that echo and grep will have problem with this extra character (\r), luckily tr, sed and awk manage it correctly.
So you can try also with:
DATA=$(grep 'overall' test.pdb | sed -n -e "s/^.*= //p" | sed -e 2s/\r$//")
or
DATA=$(grep 'overall' test.pdb | sed -n -e "s/^.*= //p" | tr -d '\r')
With awk, it will be more reliable and cleaner I guess :
$ awk '$2=="overall"{print "Working code\nData: " $4 "\n" $4 ";0"}' file.txt
Working code
Data: 324.88
324.88;0
Try this:
SUFFIX=";0"
DATA="${DATA}${SUFFIX}"
Related
Below is the piece of code of my bash script, I want to get duplicate output of that script.
This is how my script runs
#bash check_script -a used_memory
Output is: used_memory: 812632
Desired Output: used_memory: 812632 | used_memory: 812632
get_vals() {
metrics=`command -h $hostname -p $port -a $pass info | grep -w $opt_var | cut -d ':' -f2 > ${filename}`
}
output() {
get_vals
if [ -s ${filename} ];
then
val1=`cat ${filename}`
echo "$opt_var: $val1"
# rm $filename;
exit $ST_OK;
else
echo "Parameter not found"
exit $ST_UK
fi
}
But when i used echo "$opt_var: $val1 | $opt_var: $val1" the output become: | used_memory: 812632
$opt_var is an argument.
I had a similar problem when capturing results from cat with Windows-formatted text files. One way to circumvent this issue is to pipe your result to dos2unix, e.g.:
val1=`cat ${filename} | dos2unix`
Also, if you want to duplicate lines, you can use sed:
sed 's/^\(.*\)$/\1 | \1/'
Then pipe it to your echo command:
echo "$opt_var: $val1" | sed 's/^\(.*\)$/\1 | \1/'
The sed expression works like that:
's/<before>/<after>/' means that you want to substitute <before> with <after>
on the <before> side: ^.*$ is a regular expression meaning you get the entire line, ^\(.*\)$ is basically the same regex but you get the entire line and you capture everything (capturing is performed inside the \(\) expression)
on the <after> side: \1 | \1 means you write the 1st captured expression (\1), then the space character, then the pipe character, then the space character and then the 1st captured expression again
So it captures your entire line and duplicates it with a "|" separator in the middle.
I would like to grep for "String = " and when nothing is mentioned...when it's empty...echo with an if then else statement:
I tried something like that:
if grep "String" input.txt | sed -e 's/ String = //g' | egrep -q "^$"
then
<command> > output.txt
else
echo "--" > output.txt
fi
When I use the sed command the line is empty.
Thx in advance!
UPDATE
these are some possible lines in input.txt
String = TextA
String =
String = TextB
You can just use this grep to check for String = in the input file:
if grep -Eq 'String *= *$' input.txt; then
<command> > output.txt
else
echo "--" > output.txt
fi
You might consider using awk:
CMD=$(awk '$1=="String" {print $3}' input.txt)
if [ -n "$CMD" ]
then
CMD="--"
fi
echo "$CMD" > output.txt
some explanations:
awk prints the third column (since there is a space before and after the = sign; that is the 2nd column)
the line if [ -n "$CMD" ] just tests if the variable CMD is empty
From your question, I assume you are having trouble finding the specified lines in the file. The following grep command will find those lines:
egrep 'String[[:space:]]*=[[:space:]]*' input.txt
Adapt it to your if/else statements and it should work.
Or, if you want to specify that there is one space before the "=", do:
egrep 'String[[:space:]]{1}=[[:space:]]*;
How can I extract all the words from a file, every word on a single line?
Example:
test.txt
This is my sample text
Output:
This
is
my
sample
text
The tr command can do this...
tr [:blank:] '\n' < test.txt
This asks the tr program to replace white space with a new line.
The output is stdout, but it could be redirected to another file, result.txt:
tr [:blank:] '\n' < test.txt > result.txt
And here the obvious bash line:
for i in $(< test.txt)
do
printf '%s\n' "$i"
done
EDIT Still shorter:
printf '%s\n' $(< test.txt)
That's all there is to it, no special (pathetic) cases included (And handling multiple subsequent word separators / leading / trailing separators is by Doing The Right Thing (TM)). You can adjust the notion of a word separator using the $IFS variable, see bash manual.
The above answer doesn't handle multiple spaces and such very well. An alternative would be
perl -p -e '$_ = join("\n",split);' test.txt
which would. E.g.
esben#mosegris:~/ange/linova/build master $ echo "test test" | tr [:blank:] '\n'
test
test
But
esben#mosegris:~/ange/linova/build master $ echo "test test" | perl -p -e '$_ = join("\n",split);'
test
test
This might work for you:
# echo -e "this is\tmy\nsample text" | sed 's/\s\+/\n/g'
this
is
my
sample
text
perl answer will be :
pearl.214> cat file1
a b c d e f pearl.215> perl -p -e 's/ /\n/g' file1
a
b
c
d
e
f
pearl.216>
I'm running the following but it's returning with empty lines at the top and bottom of the new file.
How do I output to a new file without these empty lines?
input | sed -E '/^$/d' > file.txt
The following has no effect either.
sed '1d'
sed '$d'
I'm unsure of where the expression has problems.
If you are comfortable using awk then this would work -
awk 'NF' INPUT_FILE > OUTPUT_FILE
grep . file_name > outfile would do the job for you.
This might work for you:
echo -e " \t\r\nsomething\n \t \r\n" | sed '/^\s*$/d' | cat -n
1 something
N.B. This removes all blank lines, to preserve blank lines in the body of a file use:
echo -e " \t\r\n something\n \nsomething else \n \t \r\n" |
sed ':a;$!{N;ba};s/^\(\s*\n\)*\|\(\s*\n\)*$//g'
something
something else
my question seems to be general, but i can't find any answers.
In sed command, how can you replace the substitution pattern by a value returned by a simple bash function.
For instance, I created the following function :
function parseDates(){
#Some process here with $1 (the pattern found)
return "dateParsed;
}
and the folowing sed command :
myCatFile=`sed -e "s/[0-3][0-9]\/[0-1][0-9]\/[0-9][0-9]/& parseDates &\}/p" myfile`
I found that the caracter '&' represents the current pattern found, i'd like it to be passed to my bash function and the whole pattern to be substituted by the pattern found +dateParsed.
Does anybody have an idea ?
Thanks
you can use the "e" option in sed command like this:
cat t.sh
myecho() {
echo ">>hello,$1<<"
}
export -f myecho
sed -e "s/.*/myecho &/e" <<END
ni
END
you can see the result without "e":
cat t.sh
myecho() {
echo ">>hello,$1<<"
}
export -f myecho
sed -e "s/.*/myecho &/" <<END
ni
END
Agree with Glenn Jackman.
If you want to use bash function in sed, something like this :
sed -rn 's/^([[:digit:].]+)/`date -d #&`/p' file |
while read -r line; do
eval echo "$line"
done
My file here begins with a unix timestamp (e.g. 1362407133.936).
Bash function inside sed (maybe for other purposes):
multi_stdin(){ #Makes function accepet variable or stdin (via pipe)
[[ -n "$1" ]] && echo "$*" || cat -
}
sans_accent(){
multi_stdin "$#" | sed '
y/àáâãäåèéêëìíîïòóôõöùúûü/aaaaaaeeeeiiiiooooouuuu/
y/ÀÁÂÃÄÅÈÉÊËÌÍÎÏÒÓÔÕÖÙÚÛÜ/AAAAAAEEEEIIIIOOOOOUUUU/
y/çÇñÑߢÐð£Øø§µÝý¥¹²³ªº/cCnNBcDdLOoSuYyY123ao/
'
}
eval $(echo "Rogério Madureira" | sed -n 's#.*#echo & | sans_accent#p')
or
eval $(echo "Rogério Madureira" | sed -n 's#.*#sans_accent &#p')
Rogerio
And if you need to keep the output into a variable:
VAR=$( eval $(echo "Rogério Madureira" | sed -n 's#.*#echo & | desacentua#p') )
echo "$VAR"
do it step by step. (also you could use an alternate delimiter , such as "|" instead of "/"
function parseDates(){
#Some process here with $1 (the pattern found)
return "dateParsed;
}
value=$(parseDates)
sed -n "s|[0-3][0-9]/[0-1][0-9]/[0-9][0-9]|& $value &|p" myfile
Note the use of double quotes instead of single quotes, so that $value can be interpolated
I'd like to know if there's a way to do this too. However, for this particular problem you don't need it. If you surround the different components of the date with ()s, you can back reference them with \1 \2 etc and reformat however you want.
For instance, let's reverse 03/04/1973:
echo 03/04/1973 | sed -e 's/\([0-9][0-9]\)\/\([0-9][0-9]\)\/\([0-9][0-9][0-9][0-9]\)/\3\/\2\/\1/g'
sed -e 's#[0-3][0-9]/[0-1][0-9]/[0-9][0-9]#& $(parseDates &)#' myfile |
while read -r line; do
eval echo "$line"
done
You can glue together a sed-command by ending a single-quoted section, and reopening it again.
sed -n 's|[0-3][0-9]/[0-1][0-9]/[0-9][0-9]|& '$(parseDates)' &|p' datefile
However, in contrast to other examples, a function in bash can't return strings, only put them out:
function parseDates(){
# Some process here with $1 (the pattern found)
echo dateParsed
}