I want to use a string to control a for loop in bash. My first test code produces what I would expect and what I want:
$ aa='1 2 3 4'
$ for ii in $aa; do echo $ii; done
1
2
3
4
I'd like to use something like the following instead. This doesn't give the output I'd like (I can see why it does what it does).
$ aa='1..4'
$ for ii in $aa; do echo $ii; done
1..4
Any suggestions on how I should modify the second example to give the same output as the first?
Thanks in advance for any thoughts. I'm slowly learning bash but still have a lot to learn.
Mike
The notation could be written out as:
for ii in {1..4}; do echo "$ii"; done
but the {1..4} needs to be written out like that, no variables involved, and not as the result of variable substitution. That is brace expansion in the Bash manual, and it happens before string expansions, etc. You'll probably be best off using:
for ii in $(seq 1 4); do echo "$ii"; done
where either the 1 or the 4 or both can be shell variables.
You could use seq command (see man seq).
$ aa='1 4'
$ for ii in $(seq $aa); do echo $ii; done
Bash won't do brace expansion with variables, but you can use eval:
$ aa='1..4'
$ for ii in $(eval echo {$aa}); do echo $ii; done
1
2
3
4
You could also split aa into an array:
IFS=. arr=($aa)
for ((ii=arr[0]; ii<arr[2]; ii++)); do echo $ii; done
Note that IFS can only be a single character, so the .. range places the numbers into indexes 0 and 2.
Note There are certainly more elegant ways of doing this, as Ben Grimm's answer, and this is not pure bash, as uses seq and awk.
One way of achieving this is by calling seq. It would be trivial if you knew the numbers in the string beforehand, so there would be no need to do any conversion, as you could simple do seq 1 4 or seq $a $b for that matter.
I assume, however, that your input is indeed a string in the format you mentioned, that is, 1..4 or 20..100. For this purpose you could convert the string into 2 numbers ans use them as parameters for seq.
One of possibly many ways of achieving this is:
$ `echo "1..4" | sed -e 's/\.\./ /g' | awk '{print "seq", $1, $2}'`
1
2
3
4
Note that this will work the same way for any input in the given format. If desired, sed can be changed by tr with similar results.
$ x="10..15"
$ `echo $x | tr "." " " | awk '{print "seq", $1, $2}'`
10
11
12
13
14
15
Related
echo 3+3
How can I evaluate such expressions in Bash, in this case to 6?
echo $(( 3+3 ))
expr is the standard way, but it only handles integers.
bash has a couple of extensions, which only handle integers as well:
$((3+3)) returns 6
((3+3)) used in conditionals, returns 0 for true (non-zero) and 1 for false
let 3+3 same as (( ))
let and (( )) can be used to assign values, e.g.
let a=3+3
((a=3+3))
for floating point you can use bc
echo 3+3 | bc
in shells such as zsh/ksh, you can use floats for maths. If you need more maths power, use tools like bc/awk/dc
eg
var=$(echo "scale=2;3.4+43.1" | bc)
var=$(awk 'BEGIN{print 3.4*43.1}')
looking at what you are trying to do
awk '{printf "%.2f\n",$0/59.5}' ball_dropping_times >bull_velocities
You can make use of the expr command as:
expr 3 + 3
To store the result into a variable you can do:
sum=$(expr 3 + 3)
or
sum=`expr 3 + 3`
Lots of ways - most portable is to use the expr command:
expr 3 + 3
I believe the ((3+3)) method is the most rapid as it's interpreted by the shell rather than an external binary.
time a large loop using all suggested methods for the most efficient.
Solved thanks to Dennis, an example of BC-use:
$ cat calc_velo.sh
#!/bin/bash
for i in `cat ball_dropping_times`
do
echo "scale=20; $i / 59.5" | bc
done > ball_velocities
My understanding of math processing involves floating point processing.
Using bashj (https://sourceforge.net/projects/bashj/) you can call a java method (with floating point processing, cos(), sin(), log(), exp()...) using simply
bashj +eval "3+3"
bashj +eval "3.5*5.5"
or in a bashj script, java calls of this kind:
#!/usr/bin/bashj
EXPR="3.0*6.0"
echo $EXPR "=" u.doubleEval($EXPR)
FUNCTIONX="3*x*x+cos(x)+1"
X=3.0
FX=u.doubleEval($FUNCTIONX,$X)
echo "x="$X " => f(x)=" $FUNCTIONX "=" $FX
Note the interesting speed : ~ 10 msec per call (the answer is provided by a JVM server).
Note also that u.doubleEval(1/2) will provide 0.5 (floating point) instead of 0 (integer)
One use case that might be useful in this regard is, if one of your operand itself is a bash command then try this.
echo $(( `date +%s\`+10 )) or even echo $(( `date +%s\`+(60*60) ))
In my case I was trying to get Unixtime 10 seconds and hour later than current time respectively.
I have the following string,
szTemp="23.4 78.9"
I want to split it into two smaller strings,
szIndoorTemp="23.4"
szOutdoorTemp="78.9"
However doing this,
szIndoorTemp=${szTemp::-5}
szOutdoorTemp=${szTemp::5}
Does not produce this and causes the script to crash.
In addition, when I do
echo $szTemp
I get
23.4
78.9
I suspect that this may mean that there is a carriage return in the string.
Update
This may be close...
#!/bin/bash
szTemps="23.4 67.8"
szIndoorTemp=${szTemps:0:4}
szOutdoorTemp=${szTemps:5}
echo $szIndoorTemp
echo $szOutdoorTemp
Rather than depend on exact number of characters, I would go with relying on the space as a separator:
read szIndoorTemp szOutdoorTemp <<< "$szTemp"
As an alternative, you could do:
read szIndoorTemp szOutdoorTemp < <(yourPythonScript)
I think if I am getting it right here -5 could be the culprit, when I run command mentioned by you I see following error:
szIndoorTemp=${szTemp::-5}
bash: -5: substring expression < 0
But when I change from -5 to 5 it flies as follows:
szIndoorTemp=${szTemp::5}
echo $szIndoorTemp
23.4
Try this...
$ cat example.sh
szTemp="23.4 78.9"
echo "szIndoorTemp=${szTemp:0:4}"
echo "szOutdoorTemp=${szTemp:(-4)}"
$ sh example.sh
szIndoorTemp=23.4
szOutdoorTemp=78.9
You can also try this, if it works for you..
$ cat example.sh
szTemp="23.4 78.9"
szarray=($szTemp)
echo "szIndoorTemp=${szarray[0]}"
echo "szOutdoorTemp=${szarray[1]}"
$ sh example.sh
szIndoorTemp=23.4
szOutdoorTemp=78.9
I use filter, a function in my ~/.bashrc, because I need it so often, that the simple awk-solution is too complicated for me:) :
echo "23.4 78.9" | filter 1 2
23.4 78.9
And filter picks arguments in specified order by their index, counting from 1:
filter ()
{
declare -a arr;
while read val; do
arr=($val);
for param in "$#";
do
echo -n -e ${arr[$param-1]} "\t";
done;
echo;
done
}
Usage:
echo "23.4 78.9" | filter 1
23.4
echo "23.4 78.9" | filter 2
78.9
The equivalent operation in awk is:
echo $szTemp | awk '{print "$1"}'
23.4
echo $szTemp | awk '{print "$2"}'
78.9
but the curly braces are hard to type on a German keyboard; I guess we have the äüöß characters, where English keyboards have { and }.
Variable assignment becomes:
first=$(echo "23.4 78.9" | filter 1)
23.4
second=$(echo "23.4 78.9" | filter 2)
78.9
How can print a value, either 1, 2 or 3 (at random). My best guess failed:
#!/bin/bash
1 = "2 million"
2 = "1 million"
3 = "3 million"
print randomint(1,2,3)
To generate random numbers with bash use the $RANDOM internal Bash function:
arr[0]="2 million"
arr[1]="1 million"
arr[2]="3 million"
rand=$[ $RANDOM % 3 ]
echo ${arr[$rand]}
From bash manual for RANDOM:
Each time this parameter is
referenced, a random integer between 0
and 32767 is generated. The sequence
of random numbers may be initialized
by assigning a value to RANDOM. If
RANDOM is unset,it loses its
special properties, even if it is
subsequently reset.
Coreutils shuf
Present in Coreutils, this function works well if the strings don't contain newlines.
E.g. to pick a letter at random from a, b and c:
printf 'a\nb\nc\n' | shuf -n1
POSIX eval array emulation + RANDOM
Modifying Marty's eval technique to emulate arrays (which are non-POSIX):
a1=a
a2=b
a3=c
eval echo \$$(expr $RANDOM % 3 + 1)
This still leaves the RANDOM non-POSIX.
awk's rand() is a POSIX way to get around that.
64 chars alpha numeric string
randomString32() {
index=0
str=""
for i in {a..z}; do arr[index]=$i; index=`expr ${index} + 1`; done
for i in {A..Z}; do arr[index]=$i; index=`expr ${index} + 1`; done
for i in {0..9}; do arr[index]=$i; index=`expr ${index} + 1`; done
for i in {1..64}; do str="$str${arr[$RANDOM%$index]}"; done
echo $str
}
~.$ set -- "First Expression" Second "and Last"
~.$ eval echo \$$(expr $RANDOM % 3 + 1)
and Last
~.$
Want to corroborate using shuf from coreutils using the nice -n1 -e approach.
Example usage, for a random pick among the values a, b, c:
CHOICE=$(shuf -n1 -e a b c)
echo "choice: $CHOICE"
I looked at the balance for two samples sizes (1000, and 10000):
$ for lol in $(seq 1000); do shuf -n1 -e a b c; done > shufdata
$ less shufdata | sort | uniq -c
350 a
316 b
334 c
$ for lol in $(seq 10000); do shuf -n1 -e a b c; done > shufdata
$ less shufdata | sort | uniq -c
3315 a
3377 b
3308 c
Ref: https://www.gnu.org/software/coreutils/manual/html_node/shuf-invocation.html
I need to find a faster way to number lines in a file in a specific way using tools like awk and sed. I need the first character on each line to be numbered in this fashion: 1,2,3,1,2,3,1,2,3 etc.
For example, if the input was this:
line 1
line 2
line 3
line 4
line 5
line 6
line 7
The output needs to look like this:
1line 1
2line 2
3line 3
1line 4
2line 5
3line 6
1line 7
Here is a chunk of what I have. $lines is the number of lines in the data file divided by 3. So for a file of 21000 lines I process this loop 7000 times.
export i=0
while [ $i -le $lines ]
do
export start=`expr $i \* 3 + 1`
export end=`expr $start + 2`
awk NR==$start,NR==$end $1 | awk '{printf("%d%s\n", NR,$0)}' >> data.out
export i=`expr $i + 1`
done
Basically this grabs 3 lines at a time, numbers them, and adds to an output file. It's slow...and then some! I don't know of another, faster, way to do this...any thoughts?
Try the nl command.
See https://linux.die.net/man/1/nl (or another link to the documentation that comes up when you Google for "man nl" or the text version that comes up when you run man nl at a shell prompt).
The nl utility reads lines from the
named file or the standard input if
the file argument is ommitted, applies
a configurable line numbering filter
operation and writes the result to the
standard output.
edit: No, that's wrong, my apologies. The nl command doesn't have an option for restarting the numbering every n lines, it only has an option for restarting the numbering after it finds a pattern. I'll make this answer a community wiki answer because it might help someone to know about nl.
It's slow because you are reading the same lines over and over. Also, you are starting up an awk process only to shut it down and start another one. Better to do the whole thing in one shot:
awk '{print ((NR-1)%3)+1 $0}' $1 > data.out
If you prefer to have a space after the number:
awk '{print ((NR-1)%3)+1, $0}' $1 > data.out
Perl comes to mind:
perl -pe '$_ = (($.-1)%3)+1 . $_'
should work. No doubt there is an awk equivalent. Basically, ((line# - 1) MOD 3) + 1.
This might work for you:
sed 's/^/1/;n;s/^/2/;n;s/^/3/' input
Another way is just to use grep and match everything. For example this will enumerate files:
grep -n '.*' <<< `ls -1`
Output will be:
1:file.a
2:file.b
3:file.c
awk '{printf "%d%s\n", ((NR-1) % 3) + 1, $0;}' "$#"
Python
import sys
for count, line in enumerate(sys.stdin):
stdout.write( "%d%s" % ( 1+(count % 3), line )
You don't need to leave bash for this:
i=0; while read; do echo "$((i++ % 3 + 1)) $REPLY"; done < input
This should solve the problem. $_ will print the whole line.
awk '{print ((NR-1)%3+1) $_}' < input
1line 1
2line 2
3line 3
1line 4
2line 5
3line 6
1line 7
# cat input
line 1
line 2
line 3
line 4
line 5
line 6
line 7
What's an easy way to read random line from a file in a shell script?
You can use shuf:
shuf -n 1 $FILE
There is also a utility called rl. In Debian it's in the randomize-lines package that does exactly what you want, though not available in all distros. On its home page it actually recommends the use of shuf instead (which didn't exist when it was created, I believe). shuf is part of the GNU coreutils, rl is not.
rl -c 1 $FILE
Another alternative:
head -$((${RANDOM} % `wc -l < file` + 1)) file | tail -1
sort --random-sort $FILE | head -n 1
(I like the shuf approach above even better though - I didn't even know that existed and I would have never found that tool on my own)
This is simple.
cat file.txt | shuf -n 1
Granted this is just a tad slower than the "shuf -n 1 file.txt" on its own.
perlfaq5: How do I select a random line from a file? Here's a reservoir-sampling algorithm from the Camel Book:
perl -e 'srand; rand($.) < 1 && ($line = $_) while <>; print $line;' file
This has a significant advantage in space over reading the whole file in. You can find a proof of this method in The Art of Computer Programming, Volume 2, Section 3.4.2, by Donald E. Knuth.
using a bash script:
#!/bin/bash
# replace with file to read
FILE=tmp.txt
# count number of lines
NUM=$(wc - l < ${FILE})
# generate random number in range 0-NUM
let X=${RANDOM} % ${NUM} + 1
# extract X-th line
sed -n ${X}p ${FILE}
Single bash line:
sed -n $((1+$RANDOM%`wc -l test.txt | cut -f 1 -d ' '`))p test.txt
Slight problem: duplicate filename.
Here's a simple Python script that will do the job:
import random, sys
lines = open(sys.argv[1]).readlines()
print(lines[random.randrange(len(lines))])
Usage:
python randline.py file_to_get_random_line_from
Another way using 'awk'
awk NR==$((${RANDOM} % `wc -l < file.name` + 1)) file.name
A solution that also works on MacOSX, and should also works on Linux(?):
N=5
awk 'NR==FNR {lineN[$1]; next}(FNR in lineN)' <(jot -r $N 1 $(wc -l < $file)) $file
Where:
N is the number of random lines you want
NR==FNR {lineN[$1]; next}(FNR in lineN) file1 file2
--> save line numbers written in file1 and then print corresponding line in file2
jot -r $N 1 $(wc -l < $file) --> draw N numbers randomly (-r) in range (1, number_of_line_in_file) with jot. The process substitution <() will make it look like a file for the interpreter, so file1 in previous example.
#!/bin/bash
IFS=$'\n' wordsArray=($(<$1))
numWords=${#wordsArray[#]}
sizeOfNumWords=${#numWords}
while [ True ]
do
for ((i=0; i<$sizeOfNumWords; i++))
do
let ranNumArray[$i]=$(( ( $RANDOM % 10 ) + 1 ))-1
ranNumStr="$ranNumStr${ranNumArray[$i]}"
done
if [ $ranNumStr -le $numWords ]
then
break
fi
ranNumStr=""
done
noLeadZeroStr=$((10#$ranNumStr))
echo ${wordsArray[$noLeadZeroStr]}
Here is what I discovery since my Mac OS doesn't use all the easy answers. I used the jot command to generate a number since the $RANDOM variable solutions seems not to be very random in my test. When testing my solution I had a wide variance in the solutions provided in the output.
RANDOM1=`jot -r 1 1 235886`
#range of jot ( 1 235886 ) found from earlier wc -w /usr/share/dict/web2
echo $RANDOM1
head -n $RANDOM1 /usr/share/dict/web2 | tail -n 1
The echo of the variable is to get a visual of the generated random number.
Using only vanilla sed and awk, and without using $RANDOM, a simple, space-efficient and reasonably fast "one-liner" for selecting a single line pseudo-randomly from a file named FILENAME is as follows:
sed -n $(awk 'END {srand(); r=rand()*NR; if (r<NR) {sub(/\..*/,"",r); r++;}; print r}' FILENAME)p FILENAME
(This works even if FILENAME is empty, in which case no line is emitted.)
One possible advantage of this approach is that it only calls rand() once.
As pointed out by #AdamKatz in the comments, another possibility would be to call rand() for each line:
awk 'rand() * NR < 1 { line = $0 } END { print line }' FILENAME
(A simple proof of correctness can be given based on induction.)
Caveat about rand()
"In most awk implementations, including gawk, rand() starts generating numbers from the same starting number, or seed, each time you run awk."
-- https://www.gnu.org/software/gawk/manual/html_node/Numeric-Functions.html