I have a file/string containing the following:
[1-9]
[11-12]
[10-15]
I then want to expand that to become this:
1 2 3 4 5 6 7 8 9
11 12
10 11 12 13 14 15
I know how to do it in a very long way (first capture the two numbers and then expand them using a for loop).
I would like to know if there is a faster/smarter way of achieve the same.
One way:(Pure bash solution)
while IFS=- read l1 l2
do
eval echo ${l1/[/{}".."${l2/]/}}
done < file
There are several solutions.
Solution 1:
sed 's/^/echo /; s/[[]/{/; s/]/}/; s/-/../' | bash
Example:
$ cat 1.txt | sed 's/^/echo /; s/[[]/{/; s/]/}/; s/-/../' | bash
1 2 3 4 5 6 7 8 9
11 12
10 11 12 13 14 15
Solution 2:
tr '[]-' ' ' | sed "s/^/seq -s' '/" | bash
Example:
$ cat 1.txt | tr '[]-' ' ' | sed "s/^/seq -s' '/" | bash
1 2 3 4 5 6 7 8 9
11 12
10 11 12 13 14 15
If you're confident that your input all matches that pattern:
while read a; do
seq -s' ' $(echo "$a" | tr '[]-' ' ')
done
Add error checking as appropriate.
Here's a one-liner:
cat lines | sed -E -e 's/\[|]//g' -e 's/-/ /g' | xargs -n 2 seq -s ' ' -t '\n'
As in:
$ cat <<EOF | sed -E -e 's/\[|]//g' -e 's/-/ /g' | xargs -n 2 seq -s ' ' -t '\n'
> [1-9]
> [11-12]
> [10-15]
> EOF
1 2 3 4 5 6 7 8 9
11 12
10 11 12 13 14 15
Related
Im stuck on some homework. The requirements of the assignment are to accept an input file and perform some statistics on the values. The user may specify whether to calculate the statistics by row or by value. The shell script must be pure bash script so I can't use awk, sed, perl, python etc.
sample input:
1 1 1 1 1 1 1
39 43 4 3225 5 2 2
6 57 8 9 7 3 4
3 36 8 9 14 4 3
3 4 2 1 4 5 5
6 4 4814 7 7 6 6
I can't figure out how to sort and process the data by column. My code for processing the rows works fine.
# CODE FOR ROWS
while read -r line
echo $(printf "%d\n" $line | sort -n) | tr ' ' \\t > sorted.txt
....
#I perform the stats calculations
# for row line by working with the temp file sorted.txt
done
How could I process this data by column? I've never worked with shell script so I've been staring at this for hours.
If you wanted to analyze by columns you'll need the cols value first (number of columns). head -n 1 gives you the first row, and NF counts the number of fields, giving us the number of columns.
cols=$(head -n 1 test.txt | awk '{print NF}');
Then you can use cut with the '\t' delimiter to grab every column from input.txt, and run it through sort -n, as you did in your original post.
$ for i in `seq 2 $((cols+1))`; do cut -f$i -d$'\t' input.txt; done | sort -n > output.txt
For rows, you can use the shell built-in printf with the format modifier %dfor integers. The sort command works on lines of input, so we replace spaces ' ' with newlines \n using the tr command:
$ cat input.txt | while read line; do echo $(printf "%d\n" $line); done | tr ' ' '\n' | sort -n > output.txt
Now take the output file to gather our statistics:
Min: cat output.txt | head -n 1
Max: cat output.txt | tail -n 1
Sum: (courtesy of Dimitre Radoulov): cat output.txt | paste -sd+ - | bc
Mean: (courtesy of porges): cat output.txt | awk '{ $total += $2 } END { print $total/NR }'
Median: (courtesy of maxschlepzig): cat output.txt | awk ' { a[i++]=$1; } END { print a[int(i/2)]; }'
Histogram: cat output.txt | uniq -c
8 1
3 2
4 3
6 4
3 5
4 6
3 7
2 8
2 9
1 14
1 36
1 39
1 43
1 57
1 3225
1 4814
I have the following test file:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
I want to separate it in a way that each file contains the last line of the previous file as the first line. The example would be:
file 1:
1
2
3
4
5
file2:
5
6
7
8
9
file3:
9
10
11
12
13
file4:
13
14
15
16
17
file5:
17
18
19
20
That would make 4 files with 5 lines and 1 file with 4 lines.
As a first step, I tried to test the following commands I wrote to get only the first file which contains the first 5 lines. I can't figure out why the awk command in the if statement, instead of printing the first 5 lines, it prints the whole 20?
d=$(wc test)
a=$(echo $d | cut -f1 -d " ")
lines=$(echo $a/5 | bc -l)
integer=$(echo $lines | cut -f1 -d ".")
for i in $(seq 1 $integer); do
start=$(echo $i*5 | bc -l)
var=$((var+=1))
echo start $start
echo $var
if [[ $var = 1 ]]; then
awk 'NR<=$start' test
fi
done
Thanks!
Why not just use the split util available from your POSIX toolkit. It has an option to split on number of lines which you can give it as 5
split -l 5 input-file
From the man split page,
-l, --lines=NUMBER
put NUMBER lines/records per output file
Note that, -l is POSIX compliant also.
$ ls
$
$ seq 20 | awk 'NR%4==1{ if (out) { print > out; close(out) } out="file"++c } {print > out}'
$
$ ls
file1 file2 file3 file4 file5
.
$ cat file1
1
2
3
4
5
$ cat file2
5
6
7
8
9
$ cat file3
9
10
11
12
13
$ cat file4
13
14
15
16
17
$ cat file5
17
18
19
20
If you're ever tempted to use a shell loop to manipulate text again, make sure to read https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice first to understand at least some of the reasons to use awk instead. To learn awk, get the book Effective Awk Programming, 4th Edition, by Arnold Robbins.
oh. and wrt why your awk command awk 'NR<=$start' test didn't work - awk is not shell, it has no more access to shell variables (or vice-versa) than a C program does. To init an awk variable named awkstart with the value of a shell variable named start and then use that awk variable in your script you'd do awk -v awkstart="$start" 'NR<=awkstart' test. The awk variable can also be named start or anything else sensible - it is completely unrelated to the name of the shell variable.
You could improve your code by removing the unneccesary echo cut and bc and do it like this
#!/bin/bash
for i in $(seq $(wc -l < test) ); do
(( i % 4 != 1 )) && continue
tail +$i test | head -5 > "file$(( 1+i/4 ))"
done
But still the awk solution is much better. Reading the file only once and taking actions based on readily available information (like the linenumber) is the way to go. In shell you have to count the lines, there is no way around it. awk will give you that (and a lot of other things) for free.
Use split:
$ seq 20 | split -l 5
$ for fn in x*; do echo "$fn"; cat "$fn"; done
xaa
1
2
3
4
5
xab
6
7
8
9
10
xac
11
12
13
14
15
xad
16
17
18
19
20
Or, if you have a file:
$ split -l test_file
I have 5 variables and each variables contains five values.I want to print five lines with the five values from five variables one by one
For example
$a=1 2 3 4 5
$b=4 2 3 4 5
$c=8 9 7 6 5
$d= 8 7 6 5 4
$e=5 6 7 3 3
I want to print five lines in this format
My options was a=1,b=4,c=8,d=8and e=5
My options was a=2,b=2,c=9,d=7 and e=6
and so on upto five values.
I got confused in using the loops.Can anyone help me to provide loops in script to obtain the following output.
a="1 2 3 4 5"
b="4 2 3 4 5"
c="8 9 7 6 5"
d="8 7 6 5 4"
e="5 6 7 3 3"
for i in $(seq 1 5); do
echo -e "My options was \c"
echo -e "a=$(echo $a | cut -f$i -d' ')\c"
echo -e "b=$(echo $b | cut -f$i -d' ')\c"
echo -e "c=$(echo $c | cut -f$i -d' ')\c"
echo -e "d=$(echo $d | cut -f$i -d' ') and \c"
echo -e "e=$(echo $e | cut -f$i -d' ')"
done
Using this awk command with a bash loop:
for i in {1..5}; do
awk '{printf "My options was a=%d, b=%d, c=%d, d=%d and e=%d\n", $1, $2, $3, $4, $5}' <<< $(awk '{print $'$i'}' <(echo -e "$a\n$b\n$c\n$d\n$e") | tr $'\n' ' '); done
Output:
$ a='1 2 3 4 5'
$ b='4 2 3 4 5'
$ c='8 9 7 6 5'
$ d='8 7 6 5 4'
$ e='5 6 7 3 3'
$ for i in {1..5}; do
awk '{printf "My options was a=%d, b=%d, c=%d, d=%d and e=%d\n", $1, $2, $3, $4, $5}' <<< $(awk '{print $'$i'}' <(echo -e "$a\n$b\n$c\n$d\n$e") | tr $'\n' ' '); done
My options was a=1, b=4, c=8, d=8 and e=5
My options was a=2, b=2, c=9, d=7 and e=6
My options was a=3, b=3, c=7, d=6 and e=7
My options was a=4, b=4, c=6, d=5 and e=3
My options was a=5, b=5, c=5, d=4 and e=3
If you transpose the matrix, this is really simple, portable, and idiomatic.
while read -r a b c d e; do
: stuff with "$a", "$b", etc
done <<____
1 4 8 8 5
2 2 9 7 6
3 3 7 6 7
4 4 6 5 3
5 5 5 4 3
____
Notice how the first column enumerates the a values, the second, the bs, etc.
I'm trying to use the sort command to sort integers in string separated by a space. For example 8 6 5 7 9 56 -20 - 10. I receive the string on the standard output. I tried all of these but nothing works :
sort -t' '
sort -t ' '
sort -t " "
sort -t" "
sort -t=" "
echo "8 6 5 7 9 56 -20 - 10" | tr ' ' '\n' | sort -n
Sort can only sort lines.
You can first read string into an array with space as delimiter then use sort with process substitution:
s='8 6 5 7 9 56 -20 - 10'
read -ra arr <<< "$s"
sort -n <(printf "%s\n" "${arr[#]}")
Output:
-20
-10
5
6
7
8
9
56
To store output in string again:
read -r str < <(sort -n <(printf "%s\n" "${arr[#]}") | tr '\n' ' ')
And check output:
declare -p str
declare -- str="-20 -10 5 6 7 8 9 56"
I am trying to look for a way to delete every 3rd and 5th line but not the 15th using sed, but this is the thing: you can't make use of the ~ way (GNU). It has to be something like
sed 'n;n;d' test
but I can't figure out how to combine the 3...
Example input
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Example output:
1
2
4
7
8
11
13
14
15
It'll need to be in sed, no awk or perl
awk command is easier to understand for this requirement:
awk 'NR==15 || (NR%3 && NR%5)' file
1
2
4
7
8
11
13
14
15
ugh:
$ seq 15 | sed -n 'p;n;p;n;n;p;n;n;n;p;n;p;n;n;n;p;n;n;p;n;p;n;p'
1
2
4
7
8
11
13
14
15
This might work for you (GNU sed):
sed '0~15b;0~3d;0~5d' file
Using gnu sed:
sed '15p;0~3d;0~5d' test
here is the test result from above awk/sed commands:
seq 99 |awk 'NR==15 || (NR%3 && NR%5)' > anubhava.txt
seq 99 |sed -n 'p;n;p;n;n;p;n;n;n;p;n;p;n;n;n;p;n;n;p;n;p;n;p' > glenn.jackman.txt
seq 99 |sed '0~15b;0~3d;0~5d' > potong.txt
seq 99 |sed '15p;0~3d;0~5d' > bmw.txt
$diff anubhava.txt glenn.jackman.txt
17a18
> 30
25a27
> 45
33a36
> 60
41a45
> 75
49a54
> 90
$ diff -q anubhava.txt potong.txt
Files anubhava.txt and potong.txt differ # same problem that can't delete line 30, 45, 60, etc.
$ diff -q anubhava.txt bmw.txt
$