Bash for loop through array - bash

I'm writing a little bash script for work. But now I'm stuck. Let me just show you the code and explain:
# I have an `array` with names
NAMES=(Skypper Lampart Shepard Ryan Dean Jensen)
Now I wanna iterate trough the names
for (( i = 0; i < 6; i++ )); do
COMMAND="sed -i ${i+2}s/.*/${NAMES[${i}]}"
${COMMAND} config.txt
done
config.txt is a file with 2 numbers and names and I just wanna replace the names.
1
2
Name 1
Name 2
Name 3
Name 4
Name 5
Name 6
My problem is in the for-Loop how can I make $i + 2? So if I $i is 1 it should be 3.
Expected output:
1
2
Skypper
Lampart
Shepard
Ryan
Dean
Jensen

Bash is good at reading arrays (something you could have easily searched for).
Try something like:
for idx in "${!NAMES[#]}"
do
sed -i "$((idx + 2))s/.*/${NAMES[idx]} $idx/" config.txt
done
You will find that placing commands inside variables can also come unstuck unless you know what you are doing, so just use the command as intended :)
You might also need to remember that indexes start at zero and not 1

If I understood what you want to accomplish (Replace "Name" with a string from NAMES array, problem being index in array starts from 0 and you want to start on the 3rd line) - dirty and quick solution is to add 2 empty strings to beginning of your array and start your loop from the position you want.

Use this:
NAMES=(Skypper Lampart Shepard Ryan Dean Jensen)
line=2 # Need to skip first 2 lines
for name in "${NAMES[#]}"
do
((line++))
sed -i "${line}s/.*/$name/g" config.txt
done

You can try something around like this:
NAMES=(Skypper Lampart Shepard Ryan Dean Jensen)
for (( i = 0; i < 6; i++ )); do
b=$(( $i+2 ))
COMMAND="sed -i $b s/.*/${NAMES[${i}]}"
echo $COMMAND
# ${COMMAND} config.txt
done
Which gives me something like the following output:
# sh test.sh
sed -i 2 s/.*/Skypper
sed -i 3 s/.*/Lampart
sed -i 4 s/.*/Shepard
sed -i 5 s/.*/Ryan
sed -i 6 s/.*/Dean
sed -i 7 s/.*/Jensen

A bit late answer... :)
In your code you calling the sed n-times. This is inefficient. Therefore me proposing different solution, using ed instead of the sed. (as in good old times 30 years ago in BSD 2.9 :) ).
For this, approach:
first creating commands for the ed
executing them in one editor invocation
# it is good practice not using UPPERCASE variables
# as theycould collide with ENV variables
names=(Skypper Lampart Shepard Ryan Dean Jensen)
file="config.txt"
#create an array of commands for the "ed"
declare -a cmd
for name in "${names[#]}"; do
cmd+=("/Name/s//$name/")
done
cmd+=(w q)
echo "=== [$file before] ==="
cat "$file"
echo "=== [commands for execution ]==="
printf "%s\n" "${cmd[#]}"
#execute the prepared command in the "ed"
printf "%s\n" "${cmd[#]}" | ed -s "$file"
echo "===[ $file after ]==="
cat "$file"
output from the above
=== [config.txt before] ===
1
2
Name 1
Name 2
Name 3
Name 4
Name 5
Name 6
=== [commands for execution ]===
/Name/s//Skypper/
/Name/s//Lampart/
/Name/s//Shepard/
/Name/s//Ryan/
/Name/s//Dean/
/Name/s//Jensen/
w
q
===[ config.txt after ]===
1
2
Skypper 1
Lampart 2
Shepard 3
Ryan 4
Dean 5
Jensen 6
a variant which replaces by the line-numbers
names=(Skypper Lampart Shepard Ryan Dean Jensen)
file="config.txt"
#create an array of commands for the "ed"
declare -a cmd
n=3
for name in "${names[#]}"; do
cmd+=("${n}s/.*/$name/")
let n++
done
cmd+=(w q)
echo "=== [$file before] ==="
cat "$file"
echo "=== [commands for execution ]==="
printf "%s\n" "${cmd[#]}"
#execute the prepared command in the "ed"
printf "%s\n" "${cmd[#]}" | ed -s "$file"
echo "===[ $file after ]==="
cat "$file"
output
=== [config.txt before] ===
1
2
Name 1
Name 2
Name 3
Name 4
Name 5
Name 6
=== [commands for execution ]===
3s/.*/Skypper/
4s/.*/Lampart/
5s/.*/Shepard/
6s/.*/Ryan/
7s/.*/Dean/
8s/.*/Jensen/
w
q
===[ config.txt after ]===
1
2
Skypper
Lampart
Shepard
Ryan
Dean
Jensen

Related

Multiple lines added to vim line by line

Can you please help add multiple lines of txt to the file via bash script through vim?
I tried this:
vim -c "3 s/^/
add-this-line1
add-this-line2
add-this-line3/" -c "wq" /var/www/html/webserver/output_file.txt
But, the output of the file looks like this:
3 add-this-line1 add-this-line2 add-this-line3
What I want to do is, add the lines one by one FROM the line 3 in the output_file.txt not at the line 3 one next to another.
This is more of a job for ed, IMO
seq 10 > file
ed file <<END_ED
3a
first
second
third
.
wq
END_ED
For those new to ed, the line with the dot signals the end of "insert mode".
file now contains:
1
2
3
first
second
third
4
5
6
7
8
9
10
if you really want to do it via vim, I believe you need to insert new lines in your substitution:
vim -c "3 s/^/add-this-line1\radd-this-line2\radd-this-line3\r/" -c "wq" /var/www/html/webserver/output_file.txt
With ex or ed if available/acceptable.
printf '%s\n' '3a' 'foo' 'bar' 'baz' 'more' . 'w output_file.txt' | ex -s input_file.txt
Replace ex with ed and it should be the same output.
Using a bash array to store the data that needs to be inserted.
to_be_inserted=(foo bar baz more)
printf '%s\n' '3a' "${to_be_inserted[#]}" . 'w output_file.txt' | ex -s inputfile.txt
Again change ex to ed should do the same.
If the input file needs to be edited in-place then remove the output_file.txt just leave the w.
Though It seems you want to insert from the beginning of the line starting from line number "3 s/^/
Give the file.txt that was created by running
printf '%s\n' {1..10} > file.txt
A bit of shell scripting would do the trick.
#!/usr/bin/env bash
start=3
to_be_inserted=(
foo
bar
baz
more
)
for i in "${to_be_inserted[#]}"; do
printf -v output '%ds/^/%s/' "$start" "$i"
ed_array+=("$output")
((start++))
done
printf '%s\n' "${ed_array[#]}" ,p Q | ed -s file.txt
Output
1
2
foo3
bar4
baz5
more6
7
8
9
10
Change Q to w if in-place editing is needed.
Remove the ,p if you don't want to see the output.

How to reduce run time of shell script? [duplicate]

This question already has answers here:
Take nth column in a text file
(6 answers)
Closed 2 years ago.
I have written a simple code that takes data from a text file( which has space-separated columns and 1.5 million rows) gives the output file with the specified column. But this code takes more than an hr to execute. Can anyone help me out to optimize runtime
a=0
cat 1c_input.txt/$1 | while read p
do
IFS=" "
for i in $p
do
a=`expr $a + 1`
if [ $a -eq $2 ]
then
echo "$i"
fi
done
a=0
done >> ./1.c.$2.column.freq
some lines of sample input:
1 ib Jim 34
1 cr JoHn 24
1 ut MaRY 46
2 ti Jim 41
2 ye john 6
2 wf JoHn 22
3 ye jOE 42
3 hx jiM 21
some lines of sample output if the second argument entered is 3:
Jim
JoHn
MaRY
Jim
john
JoHn
jOE
jiM
I guess you are trying to print just 1 column, then do something like
#! /bin/bash
awk -v c="$2" '{print $c}' 1c_input.txt/$1 >> ./1.c.$2.column.freq
If you just want something faster, use a utility like cut. So to
extract the third field from a single space delimited file bigfile
do:
cut -d ' ' -f 3 bigfile
To optimize the shell code in the question, using only builtin shell
commands, do something like:
while read a b c d; echo "$c"; done < bigfile
...if the field to be printed is a command line parameter, there are
several shell command methods, but they're all based on that line.

Return the read cursor to the start of the file

I'm trying to read into two file (name,number) at the same time and get value of each possible pair.
The two file are like this:
*name1
John
*name2
Paul
*number1
25
*number2
45
What i'm trying to obtain are label and result like:
*name1 *number1 John 25
*name2 *number2 John 45
*name2 *number1 Paul 25
*name2 *number2 Paul 45
Since i come from python i've tried to do it with two loop like this:
name=/home/davide/name.txt
number=/home/davide/number.txt
while read name; do
if [[ ${name:0:1} == "*" ]]; then
n=$(echo $name)
else
while read number; do
if [[ ${number:0:1} == "*" ]]; then
echo $number $n
else
echo $name $number
fi
done < $number
fi
done < $name
I have the first two pair so my guess it's that i need a command to start from the beginning of number again (like seek(0) on python) but i haven't found a similar one for bash.
I also get an "ambiguous redirect" error and i don't understand why.
After setting up your input files:
printf >name.txt '%s\n' '*name1' John '*name2' Paul
printf >number.txt '%s\n' '*number1' 25 '*number2' 45
...the following code:
#!/usr/bin/env bash
name_file=name.txt
number_file=number.txt
while IFS= read -r name1 && IFS= read -r value1; do
while IFS= read -r name2 && IFS= read -r value2; do
printf '%s\n' "$name1 $name2 $value1 $value2"
done <"$number_file"
done <"$name_file"
...properly outputs:
*name1 *number1 John 25
*name1 *number2 John 45
*name2 *number1 Paul 25
*name2 *number2 Paul 45
What changed?
We stopped using name and number both for the filenames and for the values read from them. Because of this, when you ran <$number, it no longer had the filename number.txt in it after the first iteration; likewise for $name.
We started quoting all expansions ("$foo", not $foo). See the http://shellcheck.net/ warning SC2086, and BashPitfalls #14, explaining why even echo $foo is buggy.
Running read with the -r argument and IFS set to an empty value prevents it from consuming literal backslashes or pruning leading and trailing newlines.
Using two reads inside the condition of each while loop lets us read two lines at a time from each file (as is appropriate, given the intent to process content in pairs).
Bash operates more easly on "streams", not like, on the data itself.
first substitute every second newline starting from the first for a tabulation or a space or other separator
then "paste" the files together
Then rearange columns, from *name1 John *number1 25 to *name1 *number1 John 25
cat >name.txt <<EOF
*name1
John
*name2
Paul
EOF
cat <<EOF >number.txt
*number1
25
*number2
45
EOF
paste <(<name.txt sed 'N;s/\n/\t/') <(<number.txt sed 'N;s/\n/\t/') |
awk '{print $1,$3,$2,$4}'
will output:
*name1 *number1 John 25
*name2 *number2 Paul 45
First, in your example you overwrite the variable $number. So you have issues on reading file $number beginning from the second loop-run.
Solution with paste
Command paste can combine multiple files, and with option -d line-by-line.
#!/usr/bin/env bash
name=/home/davide/name.txt
number=/home/davide/number.txt
# combine both files linb-by-line
paste $'-d\n' "$name" "$number" |
while read nam
do
#after reading name to var 'nam', read number to var 'num':
read num
# print both
echo "$nam $num"
done
if you want TABS or any other separator and no other processing, you don't need the while loop. Examples
paste "$name" "$number"
paste -d: "$name" "$number"
paste -d\| "$name" "$number"
$ cat tst.awk
NR==FNR {
if ( NR%2 ) {
tags[++numPairs] = $0
}
else {
vals[numPairs] = $0
}
next
}
!(NR%2) {
for (pairNr=1; pairNr<=numPairs; pairNr++) {
print prev, tags[pairNr], $0, vals[pairNr]
}
}
{ prev = $0 }
$ awk -f tst.awk number.txt name.txt
*name1 *number1 John 25
*name1 *number2 John 45
*name2 *number1 Paul 25
*name2 *number2 Paul 45
In your script, you use the variable name for both the file path and the while-loop variable. This causes the "ambiguous redirect" error. Two lines need fix e.g.:
name_file=/home/davide/name.txt
done < $name_file
No need to for seek(0) in shell scripts. Just process the file again, e.g:
while read line ; do
echo "== $line =="
done < /some/file
while read line ; do
echo "--> ${line:0:1}"
done < /some/file
This is less efficient and less flexible than a more real programming language where you can seek(). But that's about differences, advantages and disadvantages between shell scripting and programming.
By the way, this line:
n=$(echo $name)
... is merely a awkward way of just doing:
n=$name
This can cause your script to behave quite unpredictable when $name contains special character like *. And since $name is read from a text file, this not unlikely to happen. (thanks Charles Duffy for making this point)

for loop control in bash using a string

I want to use a string to control a for loop in bash. My first test code produces what I would expect and what I want:
$ aa='1 2 3 4'
$ for ii in $aa; do echo $ii; done
1
2
3
4
I'd like to use something like the following instead. This doesn't give the output I'd like (I can see why it does what it does).
$ aa='1..4'
$ for ii in $aa; do echo $ii; done
1..4
Any suggestions on how I should modify the second example to give the same output as the first?
Thanks in advance for any thoughts. I'm slowly learning bash but still have a lot to learn.
Mike
The notation could be written out as:
for ii in {1..4}; do echo "$ii"; done
but the {1..4} needs to be written out like that, no variables involved, and not as the result of variable substitution. That is brace expansion in the Bash manual, and it happens before string expansions, etc. You'll probably be best off using:
for ii in $(seq 1 4); do echo "$ii"; done
where either the 1 or the 4 or both can be shell variables.
You could use seq command (see man seq).
$ aa='1 4'
$ for ii in $(seq $aa); do echo $ii; done
Bash won't do brace expansion with variables, but you can use eval:
$ aa='1..4'
$ for ii in $(eval echo {$aa}); do echo $ii; done
1
2
3
4
You could also split aa into an array:
IFS=. arr=($aa)
for ((ii=arr[0]; ii<arr[2]; ii++)); do echo $ii; done
Note that IFS can only be a single character, so the .. range places the numbers into indexes 0 and 2.
Note There are certainly more elegant ways of doing this, as Ben Grimm's answer, and this is not pure bash, as uses seq and awk.
One way of achieving this is by calling seq. It would be trivial if you knew the numbers in the string beforehand, so there would be no need to do any conversion, as you could simple do seq 1 4 or seq $a $b for that matter.
I assume, however, that your input is indeed a string in the format you mentioned, that is, 1..4 or 20..100. For this purpose you could convert the string into 2 numbers ans use them as parameters for seq.
One of possibly many ways of achieving this is:
$ `echo "1..4" | sed -e 's/\.\./ /g' | awk '{print "seq", $1, $2}'`
1
2
3
4
Note that this will work the same way for any input in the given format. If desired, sed can be changed by tr with similar results.
$ x="10..15"
$ `echo $x | tr "." " " | awk '{print "seq", $1, $2}'`
10
11
12
13
14
15

BASH: Iterating two v ariables in a for loop

I am having two files numbers.txt(1 \n 2 \n 3 \n 4 \n 5 \n) and alpha.txt (a \n n \n c \n d \n e \n)
Now I want to iterate both the files at the same time something like.
for num in `cat numbers.txt` && alpha in `cat alpha.txt`
do
echo $num "blah" $alpha
done
Or other idea I was having is
for num in `cat numbers.txt`
do
for alpha in `cat alpha.txt`
do
echo $num 'and' $alpha
break
done
done
but this kind of code always take the first value of $alpha.
I hope my problem is clear enough.
Thanks in advance.
Here it is what I actually intended to do. (Its just an example)
I am having one more file say template.txt having content.
variable1= NUMBER
variable2= ALPHA
I wanted to take the output from two files i.e numbers.txt and alpha.txt(one line from both at a time) and want to replace the NUMBER and ALPHA with the respective content from those two files.
so here it what I did as i got to know how to iterate both files together.
paste number.txt alpha.txt | while read num alpha
do
cp template.txt temp.txt
sed -i "{s/NUMBER/$num/g}" temp.txt
sed -i "{s/ALPHA/$alpha/g}" temp.txt
cat temp.txt >> final.txt
done
Now what i am having in final.txt is:
variable1= 1
variable2= a
variable1= 2
variable2= b
variable1= 3
variable2= c
variable1= 4
variable2= d
variable1= 5
variable2= e
variable1= 6
variable2= f
variable1= 7
variable2= g
variable1= 8
variable2= h
variable1= 9
variable2= i
variable1= 10
variable2= j
Its very simple and stupid approach. I wanted to know is there any other way to do this??
Any suggestion will be appreciated.
No, your question isn't clear enough. Specifically, the way you wish to iterate through your files is unclear, but assuming you want to have an output such as:
1 blah a
2 blah b
3 blah c
4 blah d
5 blah e
you can use the paste utility, like this:
paste number.txt alpha.txt | while read alpha num ; do
echo "$num and $alpha"
done
or even:
paste -d# alpha num | sed 's/#/ blah /'
Your first loop is impossible in bash. Your second one, without the break, would combine each line from numbers.txt with each line from alpha.txt, like this:
1 AND a
1 AND n
1 AND c
...
2 AND a
...
3 AND a
...
4 AND a
...
Your break makes it skip all lines from the alpha.txt, except the 1st one (bmk has already explained it in his answer)
It should be possible to organize the correct loop using the while loop construction, but it would be rather ugly.
There're lots of easier alternatives which maybe a better choice, depending on specifics of your task. For example, you could try this:
paste numbers.txt alpha.txt
or, if you really want your "AND"s, then, something like this:
paste numbers.txt alpha.txt | sed 's/\t/ AND /'
And if your numbers are really sequential (and you can live without 'AND'), you can simply do:
cat -n alpha.txt
Here is an alternate solution according to the first model you suggested:
while read -u 5 a && read -u 6 b
do
echo $a $b
done 5<numbers.txt 6<alpha.txt
The notation 5<numbers.txt tells the shell to open numbers.txt using file descriptor 5. read -u 5 a means read from a value for a from file descriptor 5, which has been associated with numbers.txt.
The advantage of this approach over paste is that it gives you fine-grain control over how you merge the two files. For example you could read one line from the first file and twice from the second file.
In your second example the inner loop is executed only once because of the break. It will simply jump out of the loop, i.e. you will always only get the first element of alpha.txt. Therefore I think you should remove it:
for num in `cat numbers.txt`
do
for alpha in `cat alpha.txt`
do
echo $num 'and' $alpha
done
done
If multiple loop isn't specifically your requirement but getting corresponding lines is then you may try the following code:
for line in `cat numbers.txt`
do
echo $line "and" $(cat alpha.txt| head -n$line | tail -n1 )
done
The head gets you the number of lines equal to the value of line and tail gets you the last element.
#tollboy, I think the answer you are looking for is this:
count=1
for item in $(paste number.txt alpha.txt); do
if [[ "${item}" =~ [a-zA-Z] ]]; then
echo "variable${count}= ${item}" >> final.txt
elif [[ "${item}" =~ [0-9] ]]; then
echo "variable${count}= ${item}" >> final.txt
fi
count=$((count+1))
done
When you type paste number.txt alpha.txt in your console, you see:
1 a
2 b
3 c
4 d
5 e
6 f
7 g
8 h
9 i
10 j
From bash's point of view $(paste number.txt alpha.txt) it looks like this:
1 a 2 b 3 c 4 d 5 e 6 f 7 g 8 h 9 i 10 j
So for each item in that list, figure out if it is alpha or numeric, and print it to the output file.
Lastly, increment the count.

Resources