This question already has answers here:
Are shell scripts sensitive to encoding and line endings?
(14 answers)
Closed 9 months ago.
I have a simple bash script that downloads stock prices and appends them to a variable, which is then outputted:
#!/bin/bash
output=" "
for stock in GOOG AAPL
do
price=$(curl -s "http://download.finance.yahoo.com/d/quotes.csv?s=$stock&f=l1")
output+="$stock: $price "
done
echo "$output"
This script only displays AAPL: 524.896, the last piece of data fetched. According to whatswrongwithmyscript, there isn't anything wrong with the script, and I thought I was following this answer properly. This answer discussed a similar problem (appending to a string variable inside a loop) and suggested a different method which I used like this:
#!/bin/bash
output=" "
for stock in GOOG AAPL
do
price=$(curl -s "http://download.finance.yahoo.com/d/quotes.csv?s=$stock&f=l1")
output="$output$stock: $price "
done
echo "$output"
The output is still the same. I'm using bash 4.2.45 on debian jessie x64.
More info
I echoed the result in a loop to debug, and from the first script, this is what I get:
GOOG: 1030.42
AAPL: 524.896
AAPL: 524.896
And the second script gives the same thing:
GOOG: 1030.42
AAPL: 524.896
AAPL: 524.896
When I run your script and pipe the output to od -c, the result is illuminating:
0000000 G O O G : 1 0 3 0 . 4 2 \r
0000020 A A P L : 5 2 4 . 8 9 6 \r \n
0000040
So you can see that it IS in fact getting all the entries and concatenating them, but its ALSO getting CR characters (the \r in the od output), which causes them to print over the top of each other when you print the string.
You can pipe the output of curl to tr -d '\r' to strip off the problematic CRs:
price=$(curl -s "...." | tr -d '\r')
I'm pretty sure that the problem is that curl is returning a carriage return and this is messing with the printing of both values. If you redirect the output of the curl command to a file and view it in vi, you'll see it's created a DOS file.
This seems to work:
#!/bin/bash
output=""
for stock in GOOG AAPL
do
price=$(curl -s "http://download.finance.yahoo.com/d/quotes.csv?s=$stock&f=l1" | tr -d '\r')
output+="$stock $price\n"
done
echo -e "$output"
Related
This question already has answers here:
How can I store a command in a variable in a shell script?
(12 answers)
Dynamic variable names in Bash
(19 answers)
Closed 1 year ago.
I am running different program with different config. I tried to convert string (kmeans and bayes) in the inner loop to variables I defined at the beginning, so I can run the programs and capture the console output. kmeans_time and bayes_time are used to record execution time of each program.
#!/bin/bash
kmeans="./kmeans -m40 -n40 -t0.00001 -p 4 -i inputs/random-n1024-d128-c4.txt"
bayes="./bayes -t 4 -v32 -r1024 -n2 -p20 -s0 -i2 -e2"
kmeans_time=0
bayes_time=0
for n in {1..10}
do
for prog in kmeans bayes
do
output=($(${prog} | tail -1))
${$prog + "_time"}=$( echo $kmeans_time + ${output[1]} | bc)
echo ${output[1]}
done
done
However, I got the following errors. It seems that the prog is executed as a string instead of command I defined. Also, concatenation of the time variable filed. I've tried various ways. How is this accomplished in Bash?
./test.sh: line 11: kmeans: command not found
./test.sh: line 12: ${$app + "_time"}=$( echo $kmeans_time + ${output[1]} | bc): bad substitution
What I am trying to do is to execute as follow, which can work properly.
kmeans="./kmeans -m40 -n40 -t0.00001 -p 4 -i inputs/random-n1024-d128-c4.txt"
output=($($kmeans | tail -1))
# output[1] is the execution time
echo "${output[1]}"
kmeas_times=$kmeans_times+${output[1]}
I want to iterate over different programs and calculate each of their average execution time
I am vaguely guessing you are looking for printf -v.
The string in bayes is not a valid command, nor a valid sequence of arguments to another program, so I really can't guess what you are hoping for it to do.
Furthermore, output is not an array, so ${output[1]} is not well-defined. Are you trying to get the first token from the line? You seem to have misplaced the parentheses to make output into an array; but you can replace the tail call with a simple Awk script to just extract the token you want.
Your code would always add the value of kmeans_time to output; if you want to use the variable named by $prog you can use indirect expansion to specify the name of the variable, but you will need a temporary variable for that.
Mmmmaybe something like this? Hopefully this should at least show you what valid Bash syntax looks like.
kmeans_time=0
bayes_time=0
for n in {1..10}
do
for prog in kmeans bayes
do
case $prog in
kmeans) cmd=(./kmeans -m40 -n40 -t0.00001 -p 4 -i inputs/random-n1024-d128-c4.txt);;
bayes) cmd=(./bayes -t 4 -v32 -r1024 -n2 -p20 -s0 -i2 -e2);;
esac
output=$("${cmd[#]}" | awk 'END { print $2 }')
var=${prog}_time
printf -v "$var" %i $((!var + output))
echo "$output"
done
done
As an alternative to the indirect expansion, maybe use an associative array for the accumulated time. (Bash v5+ only, though.)
If running the two programs alternatingly is not important, your code can probably be simplified.
kmeans () {
./kmeans -m40 -n40 -t0.00001 -p 4 -i inputs/random-n1024-d128-c4.txt
}
bayes () {
./bayes -t 4 -v32 -r1024 -n2 -p20 -s0 -i2 -e2
}
get_output () {
awk 'END { print $2 }'
}
loop () {
time=0
for n in {1..10}; do
do
output=("$#" | get_output)
time=$((time+output))
print "$output"
done
printf -v "${0}_time" %i "$time"
}
loop kmeans
loop bayes
Maybe see also http://mywiki.wooledge.org/BashFAQ/050 ("I'm trying to put a command in a variable, but the complex cases always fail").
I have a simple command (my_cc) that computes the number of characters in each line.
This command yields 5, 6, 7, and 8 for text file respectively.
$ cat text
12345
123456
1234567
12345678
$ cat text | ./my_cc
5
6
7
8
My question is how to zip stdin along with stdout line by line like (without multiple processes):
$ cat text | some_magic_command with my_cc
12345 5
123456 6
1234567 7
12345678 8
A possible answer is:
$ cat text | xargs -I {} bash -c "echo {} | ./my_cc | sed 's/^/{} /g'"
12345 5
123456 6
1234567 7
12345678 8
But this invokes processes of my_cc as the number of lines in text.
I can not use this command because my_cc is too heavy to run for each line. Also I can not modify the my_cc.
You can use paste:
paste -d ' ' text <(./my_cc < text)
This puts a space between each line of text and the output of your command.
If you have a shell that doesn't support process substitution, you can read from standard input instead:
./my_cc < text | paste -d ' ' text -
If
my_cc doesn't buffer its output, but writes a line of output immediately after receiving each line of input (most commands don't do that), and
your text doesn't come from a file but is e.g. generated from another command on the fly,
you can do the following:
my_cc() {
perl -nle 'BEGIN { $| = 1 } print length'
}
coproc my_cc
while read -r; do
printf '%s ' "$REPLY"
printf '%s\n' "$REPLY" >&${COPROC[1]}
read -r <&${COPROC[0]}
printf '%s\n' "$REPLY"
done < <( echo '12345
123456
.
1234567
12345678' )
exec {COPROC[0]}<&- {COPROC[1]}>&-
wait $COPROC_PID
Output:
12345 5
123456 6
. 5
1234567 7
12345678 8
Note:
Condition #1 is essential. If my_cc buffers its output, this code will deadlock.
Condition #2 is not strictly required. You could easily run this code on a file (while read -r; do ... done < sometextfile), but a file can be read multiple times, so simpler solutions (that don't require condition #1) are possible.
Explanation:
my_cc is defined as a shell function to stand in for your actual command. It does what you described (prints the length of each line), but $| = 1 deserves comment: This statement enables autoflush mode on the currently selected output handle (which defaults to stdout), i.e. output is written immediately after each print command.
coproc is a bash built-in command that runs the specified command in the background (as a co-process).
The while read -r loop reads input line by line from another command (here played by echo '...').
Each line read ($REPLY) is first printed followed by a space, then sent to the coprocess.
Then we read a single line of output from the coprocess and print it followed by a newline.
At the end we close the file descriptors of our coprocess and wait for it to terminate.
Im trying to make a script that takes a .txt file containing lines
like:
davda103:David:Davidsson:800104-1234:TNCCC_1:TDDB46 TDDB80:
and then sort them etc. Thats just the background my problem lies here:
#!/bin/sh -x
cat $1 |
while read a
do
testsak = `echo $a | cut -f 1 -d :`; <---**
echo $testsak;
done
Where the arrow is, when I try to run this code I get some kind of weird error.
+ read a
+ cut -f+ echo 1 -d :davda103:David:Davidsson:800104-1234:TNCCC_1:TDDB46
TDDB80:
+ testsak = davda103
scriptTest.sh: testsak: Det går inte att hitta
+ echo
(I have my linux in swedish because school -.-) Anyways that error just says that it cant find... something. Any ideas what could be causing my problem?
You have extra spaces around the assignment operator, remove them:
testsak=`echo $a | cut -f 1 -d :`; <---**
The spaces around the equal sign
testsak = `echo $a | cut -f 1 -d :`; <---**
causes bash to interpret this as a command testak with arguments = and the result of the command substitution. Removing the spaces will fix the immediate error.
A much more efficient way to extract the value from a is to let read do it (and use input redirection instead of cat):
while IFS=: read testak the_rest; do
echo $testak
done < "$1"
How to I concatenate stdin to a string, like this?
echo "input" | COMMAND "string"
and get
inputstring
A bit hacky, but this might be the shortest way to do what you asked in the question (use a pipe to accept stdout from echo "input" as stdin to another process / command:
echo "input" | awk '{print $1"string"}'
Output:
inputstring
What task are you exactly trying to accomplish? More context can get you more direction on a better solution.
Update - responding to comment:
#NoamRoss
The more idiomatic way of doing what you want is then:
echo 'http://dx.doi.org/'"$(pbpaste)"
The $(...) syntax is called command substitution. In short, it executes the commands enclosed in a new subshell, and substitutes the its stdout output to where the $(...) was invoked in the parent shell. So you would get, in effect:
echo 'http://dx.doi.org/'"rsif.2012.0125"
use cat - to read from stdin, and put it in $() to throw away the trailing newline
echo input | COMMAND "$(cat -)string"
However why don't you drop the pipe and grab the output of the left side in a command substitution:
COMMAND "$(echo input)string"
I'm often using pipes, so this tends to be an easy way to prefix and suffix stdin:
echo -n "my standard in" | cat <(echo -n "prefix... ") - <(echo " ...suffix")
prefix... my standard in ...suffix
There are some ways of accomplish this, i personally think the best is:
echo input | while read line; do echo $line string; done
Another can be by substituting "$" (end of line character) with "string" in a sed command:
echo input | sed "s/$/ string/g"
Why i prefer the former? Because it concatenates a string to stdin instantly, for example with the following command:
(echo input_one ;sleep 5; echo input_two ) | while read line; do echo $line string; done
you get immediatly the first output:
input_one string
and then after 5 seconds you get the other echo:
input_two string
On the other hand using "sed" first it performs all the content of the parenthesis and then it gives it to "sed", so the command
(echo input_one ;sleep 5; echo input_two ) | sed "s/$/ string/g"
will output both the lines
input_one string
input_two string
after 5 seconds.
This can be very useful in cases you are performing calls to functions which takes a long time to complete and want to be continuously updated about the output of the function.
You can do it with sed:
seq 5 | sed '$a\6'
seq 5 | sed '$ s/.*/\0 6/'
In your example:
echo input | sed 's/.*/\0string/'
I know this is a few years late, but you can accomplish this with the xargs -J option:
echo "input" | xargs -J "%" echo "%" "string"
And since it is xargs, you can do this on multiple lines of a file at once. If the file 'names' has three lines, like:
Adam
Bob
Charlie
You could do:
cat names | xargs -n 1 -J "%" echo "I like" "%" "because he is nice"
Also works:
seq -w 0 100 | xargs -I {} echo "string "{}
Will generate strings like:
string 000
string 001
string 002
string 003
string 004
...
The command you posted would take the string "input" use it as COMMAND's stdin stream, which would not produce the results you are looking for unless COMMAND first printed out the contents of its stdin and then printed out its command line arguments.
It seems like what you want to do is more close to command substitution.
http://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html#Command-Substitution
With command substitution you can have a commandline like this:
echo input `COMMAND "string"`
This will first evaluate COMMAND with "string" as input, and then expand the results of that commands execution onto a line, replacing what's between the ‘`’ characters.
cat will be my choice: ls | cat - <(echo new line)
With perl
echo "input" | perl -ne 'print "prefix $_"'
Output:
prefix input
A solution using sd (basically a modern sed; much easier to use IMO):
# replace '$' (end of string marker) with 'Ipsum'
# the `e` flag disables multi-line matching (treats all lines as one)
$ echo "Lorem" | sd --flags e '$' 'Ipsum'
Lorem
Ipsum#no new line here
You might observe that Ipsum appears on a new line, and the output is missing a \n. The reason is echo's output ends in a \n, and you didn't tell sd to add a new \n. sd is technically correct because it's doing exactly what you are asking it to do and nothing else.
However this may not be what you want, so instead you can do this:
# replace '\n$' (new line, immediately followed by end of string) by 'Ipsum\n'
# don't forget to re-add the `\n` that you removed (if you want it)
$ echo "Lorem" | sd --flags e '\n$' 'Ipsum\n'
LoremIpsum
If you have a multi-line string, but you want to append to the end of each individual line:
$ ls
foo bar baz
$ ls | sd '\n' '/file\n'
bar/file
baz/file
foo/file
I want to prepend my sql script with "set" statement before running it.
So I echo the "set" instruction, then pipe it to cat. Command cat takes two parameters : STDIN marked as "-" and my sql file, cat joins both of them to one output. Next I pass the result to mysql command to run it as a script.
echo "set #ZERO_PRODUCTS_DISPLAY='$ZERO_PRODUCTS_DISPLAY';" | cat - sql/test_parameter.sql | mysql
p.s. mysql login and password stored in .my.cnf file
I have a bash scripts which an argument enclosed with double quotes, which creates a shape-file of map within the given boundries, e.g.
$ export_map "0 0 100 100"
Within the script, there are two select statements:
select ENCODING in UTF8 WIN1252 WIN1255 ISO-8859-8;
...
select NAV_SELECT in Included Excluded;
Naturally, these two statements require the input to enter a number as an input. This can by bypassed by piping the numbers, followed by a newline, to the script.
In order to save time, I would like to have a script that would create 8 maps - for each combination of ENCODING (4 options) and NAV_SELECT (2 options).
I have written another bash script, create_map, to server as a wrapper:
#!/bin/bash
for nav in 1 2 3 4;
do
for enc in 1 2;
do
printf "$nav\n$enc\n" | /bin/bash -c "./export_map.sh \"0 0 100 100\""
done
done
**This works (thanks, Brian!), but I can't find a way to have the numeric argument "0 0 100 100" being passed from outside the outer script. **
Basically, I'm looking for way to accept an argument within double quotes to a wrapper bash script, and pass it - with the double quotes - to an inner script.
CLARIFICATIONS:
export_map is the main script, being called from create_map 8 times.
Any ideas?
Thanks,
Adam
If I understand your problem correctly (which I'm not sure about; see my comment), you should probably add another \n to your printf; printf does not add a trailing newline by default the way that echo does. This will ensure that the second value will be read properly by the select command which I'm assuming appears in export_map.sh.
printf "$nav\n$enc\n" | /bin/bash -c "./export_map.sh \"100 200 300 400\""
Also, I don't think that you need to add the /bin/bash -c and quote marks. The following should be sufficient, unless I'm missing something:
printf "$nav\n$enc\n" | ./export_map.sh "100 200 300 400"
edit Thanks for the clarification. In order to pass an argument from your wrapper script, into the inner script, keeping it as a single argument, you can pass in "$1", where the quotes indicate that you want to keep this grouped as one argument, and $1 is the first parameter to your wrapper script. If you want to pass all parameters from your outer script in to your inner script, each being kept as a single parameter, you can use "$#" instead.
#!/bin/bash
for nav in 1 2 3 4;
do
for enc in 1 2;
do
printf "$nav\n$enc\n" | ./export_map.sh "$1"
done
done
Here's a quick example of how "$#" works. First, inner.bash:
#!/bin/bash
for str in "$#"
do
echo $str
done
outer.bash:
#!/bin/bash
./inner.bash "$#"
And invoking it:
$ ./outer.bash "foo bar" baz "quux zot"
foo bar
baz
quux zot