Given a file integers that contains integers separated by new lines. For instance:
1
39
77
109
137
169
197
229
261
293
One can iterate over the file using the following code:
while read a
do
echo "$a"
done < integers
I'm looking however for an elegant solution such that the loop takes two integers at once and always updates by one step, such that:
while #some funny commands
do
echo "$a | $b"
done < integers
results in:
1 | 39
39 | 77
77 | 109
109 | 137
137 | 169
169 | 197
197 | 229
229 | 261
261 | 293
{
read a
while read b; do
echo "$a | $b"
a=$b
done
} < file
Output:
1 | 39
39 | 77
77 | 109
109 | 137
137 | 169
169 | 197
197 | 229
229 | 261
261 | 293
Use a variable to store the previous value:
prev=
while read line; do
[[ ! -z $prev ]] && echo $prev "|" $line;
prev=$line;
done <file
Related
Here's the file:
90 88 80 70
80 90 67 89
70 81 78 85
88 79 90 78
67 66 80 90
80 60 90 89
so for example, I would need to add 90, 88, 80, and 70 together from the top row.
You can achieve this quite easily with awk:
$ cat numbers.txt
90 88 80 70
80 90 67 89
70 81 78 85
88 79 90 78
67 66 80 90
80 60 90 89
$ awk '{print $1+$2+$3+$4}' numbers.txt
328
326
314
335
303
319
Here comes a funky pure Bash solution:
while IFS=$'\n' read -ra line; do
numbers=($line)
echo $(IFS=+; echo "$((${numbers[*]}))")
done < foo.txt
As suggested by another user, this can be shortened to:
while read -ra numbers; do
echo $(IFS=+; echo "$((${numbers[*]}))")
done < foo.txt
As suggested by another user, one echo can be removed by using a compound statement instead of a command substitution statement:
while read -ra numbers; do
(IFS=+; echo "$((${numbers[*]}))")
done < foo.txt
Since this is tagged only Bash it might be a worthwhile solution.
As per suggestion by #anishsane a variant without modify IFS in a subshell:
while read -r line; do echo $(( 0 $(printf "+%s" $line) )); done < file
The below sums regardless of the number of columns.
while IFS="" read -r p || [ -n "$p" ]
do
read -a pa <<< "$p"
{
echo "0"
for i in "${pa[#]}"
do
echo "$i"
echo '+'
done
echo "p"
} | dc
done < filename.dat
I have a tab file with two columns like that
5 6 14 22 23 25 27 84 85 88 89 94 95 98 100 6 94
6 8 17 20 193 205 209 284 294 295 299 304 305 307 406 205 284 307 406
2 10 13 40 47 58 2 13 40 87
and the desired output should be
5 6 14 22 23 25 27 84 85 88 89 94 95 98 100 14 27
6 8 17 20 193 205 209 284 294 295 299 304 305 307 406 6 209 299 305
2 10 13 23 40 47 58 87 10 23 40 58
I would like to change the numbers in 2nd column for random numbers in 1st column resulting in an output in 2nd column with the same number of numbers. I mean e.g. if there are four numbers in 2nd column for x row, the output must have four random numbers from 1st column for this row, and so on...
I'm try to create two arrays by AWK and split and replace every number in 2nd column for numbers in 1st column but not in a randomly way. I have seen the rand() function but I don't know exactly how joint these two things in a script. Is it possible to do in BASH environment or are there other better ways to do it in BASH environment? Thanks in advance
awk to the rescue!
$ awk -F'\t' 'function shuf(a,n)
{for(i=1;i<n;i++)
{j=i+int(rand()*(n+1-i));
t=a[i]; a[i]=a[j]; a[j]=t}}
function join(a,n,x,s)
{for(i=1;i<=n;i++) {x=x s a[i]; s=" "}
return x}
BEGIN{srand()}
{an=split($1,a," ");
shuf(a,an);
bn=split($2,b," ");
delete m; delete c; j=0;
for(i=1;i<=bn;i++) m[b[i]];
# pull elements from a upto required sample size,
# not intersecting with the previous sample set
for(i=1;i<=an && j<bn;i++) if(!(a[i] in m)) c[++j]=a[i];
cn=asort(c);
print $1 FS join(c,cn)}' file
5 6 14 22 23 25 27 84 85 88 89 94 95 98 100 85 94
6 8 17 20 193 205 209 284 294 295 299 304 305 307 406 20 205 294 295
2 10 13 23 40 47 58 87 10 13 47 87
shuffle (standard algorithm) the input array, sample required number of elements, additional requirement is no intersection with the existing sample set. Helper structure map to keep existing sample set and used for in tests. The rest should be easy to read.
Assuming that there is a tab delimiting the two columns, and each column is a space delimited list:
awk 'BEGIN{srand()}
{n=split($1,a," ");
m=split($2,b," ");
printf "%s\t",$1;
for (i=1;i<=m;i++)
printf "%d%c", a[int(rand() * n) +1], (i == m) ? "\n" : " "
}' FS=\\t input
Try this:
# This can be an external file of course
# Note COL1 and COL2 seprated by hard TAB
cat <<EOF > d1.txt
5 6 14 22 23 25 27 84 85 88 89 94 95 98 100 6 94
6 8 17 20 193 205 209 284 294 295 299 304 305 307 406 205 284 307 406
2 10 13 40 47 58 2 13 40 87
EOF
# Loop to read each line, not econvert TAB to:, though could have used IFS
cat d1.txt | sed 's/ /:/' | while read LINE
do
# Get the 1st column data
COL1=$( echo ${LINE} | cut -d':' -f1 )
# Get col1 number of items
NUM_COL1=$( echo ${COL1} | wc -w )
# Get col2 number of items
NUM_COL2=$( echo ${LINE} | cut -d':' -f2 | wc -w )
# Now split col1 items into an array
read -r -a COL1_NUMS <<< "${COL1}"
COL2=" "
# THis loop runs once for each COL2 item
COUNT=0
while [ ${COUNT} -lt ${NUM_COL2} ]
do
# Generate a random number to use as teh random index for COL1
COL1_IDX=${RANDOM}
let "COL1_IDX %= ${NUM_COL1}"
NEW_NUM=${COL1_NUMS[${COL1_IDX}]}
# Check for duplicate
DUP_FOUND=$( echo "${COL2}" | grep ${NEW_NUM} )
if [ -z "${DUP_FOUND}" ]
then
# Not a duplicate, increment loop conter and do next one
let "COUNT = COUNT + 1 "
# Add the random COL1 item to COL2
COL2="${COL2} ${COL1_NUMS[${COL1_IDX}]}"
fi
done
# Sort COL2
COL2=$( echo ${COL2} | tr ' ' '\012' | sort -n | tr '\012' ' ' )
# Print
echo ${COL1} :: ${COL2}
done
Output:
5 6 14 22 23 25 27 84 85 88 89 94 95 98 100 :: 88 95
6 8 17 20 193 205 209 284 294 295 299 304 305 307 406 :: 20 299 304 305
2 10 13 40 47 58 :: 2 10 40 58
I wrote this in response to Reddit's daily programmer challenge, and I would like to get some of your feedback on it to improve the code (it seems to work). The challenge is as follows:
We are given a list of numbers in a "short-hand" range notation where only the significant part of the next number is written because we know the numbers are always increasing (ex. "1,3,7,2,4,1" represents [1, 3, 7, 12, 14, 21]). Some people use different separators for their ranges (ex. "1-3,1-2", "1:3,1:2", "1..3,1..2" represent the same numbers [1, 2, 3, 11, 12]) and they sometimes specify a third digit for the range step (ex. "1:5:2" represents [1, 3, 5]).
NOTE: For this challenge range limits are always inclusive.
Our job is to return a list of the complete numbers.
The possible separators are: ["-", ":", ".."]
Sample input:
104..02
545,64:11
Sample output:
104 105 106...200 201 202 # truncated for simplicity
545 564 565 566...609 610 611 # truncated for simplicity
My solution:
BEGIN { FS = "," }
function next_value(current_value, previous_value) {
regexp = current_value "$"
while(current_value <= previous_value || !(current_value ~ regexp)) {
current_value += 10
}
return current_value;
}
{
j = 0
delete number_list
for(i = 1; i <= NF; i++) {
# handle fields with ranges
if($i ~ /-|:|\.\./) {
split($i, range, /-|:|\.\./)
if(range[1] > range[2]) {
if(j != 0) {
range[1] = next_value(range[1], number_list[j-1])
range[2] = next_value(range[2], range[1])
}
else
range[2] = next_value(range[2], range[1]);
}
if(range[3] == "")
number_to_iterate_by = 1;
else
number_to_iterate_by = range[3];
range_iterator = range[1]
while(range_iterator <= range[2]) {
number_list[j] = range_iterator
range_iterator += number_to_iterate_by
j++
}
}
else {
number_list[j] = $i
j++
}
}
# apply increasing range logic and print
for(i = 0; i < j; i++ ) {
if(i == 0) {
if(NR != 1) printf "\n"
current_value = number_list[i]
}
else {
previous_value = current_value
current_value = next_value(number_list[i], previous_value)
}
printf "%s ", current_value
}
}
END { printf "\n" }
This is BASH (Not AWK).
I believe it is a valid answer because the original challenge doesn't specify a language.
#!/bin/bash
mkord(){ local v=$1 dig base
max=$2
(( dig=10**${#v} , base=max/dig*dig , v+=base ))
while (( v < max )); do (( v+=dig )); done
max=$v
}
while read line; do
line="${line//[,\"]/ }" line="${line//[:-]/..}"
IFS=' ' read -a arr <<<"$line"
max=0 a='' res=''
for val in "${arr[#]//../ }"; do
IFS=" " read v1 v2 v3 <<<"$val"
(( a==0 )) && max=$v1
[[ $v1 ]] && mkord "$v1" "$max" && v1=$max
[[ $v2 ]] && mkord "$v2" "$max" && v2=$max
res=$res${a:+,}${v2:+\{}$v1${v2:+\.\.}$v2${v3:+\.\.}$v3${v2:+\}}
a=1
done
(( ${#arr[#]} > 1 )) && res={$res}
eval set -- $res
echo "\"$*\""
done <"infile"
If the source of the tests is:
$ cat infile
"1,3,7,2,4,1"
"1-3,1-2"
"1:5:2"
"104-2"
"104..02"
"545,64:11"
The result will be:
"1 3 7 12 14 21"
"1 2 3 11 12"
"1 3 5"
"104 105 106 107 108 109 110 111 112"
"104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202"
"545 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611"
This gets the list done in 7 milliseconds.
My solution using gawk, RT (It contains the input text that matched the text denoted by RS) and next_n function uses modulo operation for to find the next number based on the last
cat range.awk
BEGIN{
RS="\\.\\.|,|:|-"
start = ""
end = 0
temp = ""
}
function next_n(n, last){
mod = last % (10**length(n))
if(mod < n) return last - mod + n
return last + ((10**length(n))-mod) + n
}
{
if(RT==":" || RT==".." || RT=="-"){
if(start=="") start = next_n($1,end)
else temp = $1
}else{
if(start != ""){
if(temp==""){
end = next_n($1,start)
step = 1
}else {
end = next_n(temp,start)
step = $1
}
for(i=start; i<=end; i+=step) printf "%s ", i
start = ""
temp = ""
}else{
end = next_n($1,end)
printf "%s ", end
}
}
}
END{
print ""
}
TEST 1
echo "104..02" | awk -f range.awk
OUTPUT 1
104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202
TEST 2
echo "545,64:11" | awk -f range.awk
OUTPUT 2
545 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611
TEST 3
echo "2..5,7,2-1,2:1,0-3,2-7,8..0,4,4,2..1" | awk -f range.awk
OUTPUT 3
2 3 4 5 7 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 40 41 42 43 52 53 54 55 56 57 58 59 60 64 74 82 83 84 85 86 87 88 89 90 91
TEST 4 with step
echo "1:5:2,99,88..7..3" | awk -f range.awk"
OUTPUT 4
1 3 5 99 188 191 194 197
I need to prepare a simple script to generate all the permutations possible of a set of elements stored in a variable in groups of n elements (being n parameterizable), the easiest solution which came to mind was using several loops depending on the selected length of the group. But I thought that it would be more elegant taking advantage of the ability of echo command to generate combinations, that is
echo {1,2}{1,2}
11 12 21 22
So using this method, I'm trying to achieve a general way to do it, using as input parameters the list of elements (for example {1,2}) and the number of elements. It would be something like it:
set={1,2,3,4}
group=3
for ((i=0; i<$group; i++));
do
repetition=$set$repetition
done
So in this particular case, at the end of the loop the repetition variable has the value {1,2,3,4}{1,2,3,4}{1,2,3,4}. But I'm not able to find the way to use this variable to produce the combinations using the echo command. I've tried, several things like:
echo $repetition
echo $(echo $repetition)
I'm stucked on it, I'd appreciate any tip or help on that.
You can use:
bash -c "echo "$repetition""
111 112 113 114 121 122 123 124 131 132 133 134 141 142 143 144 211 212 213 214 221 222 223 224 231 232 233 234 241 242 243 244 311 312 313 314 321 322 323 324 331 332 333 334 341 342 343 344 411 412 413 414 421 422 423 424 431 432 433 434 441 442 443 444
Or else use eval instead of bash -c
If you need k-combinations for all k, this combination script can help:
#!/bin/bash
POWER=$((2**$#))
BITS=`seq -f '0' -s '' 1 $#`
while [ $POWER -gt 1 ];do
POWER=$(($POWER-1))
BIN=`bc <<< "obase=2; $POWER"`
MASK=`echo $BITS | sed -e "s/0\{${#BIN}\}$/$BIN/" | grep -o .`
POS=1; AWK=`for M in $MASK;do
[ $M -eq 1 ] && echo -n "print \\$\${POS};"
POS=$(($POS+1))
done;echo`
awk -v ORS=" " "{$AWK}" <<< "$#" | sed 's/ $//'
done
Example:
./combination ⚪ ⛔ ⚫
⚪ ⛔ ⚫
⚪ ⛔
⚪ ⚫
⚪
⛔ ⚫
⛔
⚫
The empty set is there too, trust me.
Is there anywhere a precise makefile grammar definition? Or at least some common subset since I guess that there are some flavors. Such a grammar that could be used for writing a parser.
GNU Make manual doesn't seem to be that precise. It would require some guessing and trial-and-error to write a parser for makefile based on that document.
I also found a similar question on ANTLR mail list. But it remained unanswered which kind of suggests the answer...
I made GNU Make's grammar for myself, you can find the grammar and overview of lexer below. It's not perfect, but it can serve as a starting point for someone who wants something better. Some additional and background information is in the post.
Implementation: lexer and parser.
Bison's dump of the grammar
/* Assign lower precedence to NL. */
/* %precedence NL */
/* %precedence COMMENT "ifdef" "ifndef" "ifeq" "ifneq" */
1 makefile: statements "end of file"
2 | "end of file"
3 statements: br
4 | statement
5 | statements br
6 | statements statement
7 conditional: if_eq_kw condition statements_opt "endif" comment_opt br
8 | if_eq_kw condition statements_opt "else" statements_opt "endif" comment_opt br
9 | if_eq_kw condition statements_opt "else" conditional
10 | if_def_kw identifier statements_opt "endif" comment_opt br
11 | if_def_kw identifier statements_opt "else" statements_opt "endif" comment_opt br
12 | if_def_kw identifier statements_opt "else" conditional
13 conditional_in_recipe: if_eq_kw condition recipes_opt "endif" comment_opt
14 | if_eq_kw condition recipes_opt "else" recipes_opt "endif" comment_opt
15 | if_eq_kw condition recipes_opt "else" conditional_in_recipe
16 | if_def_kw identifier recipes_opt "endif" comment_opt
17 | if_def_kw identifier recipes_opt "else" recipes_opt "endif" comment_opt
18 | if_def_kw identifier recipes_opt "else" conditional_in_recipe
19 condition: '(' expressions_opt ',' expressions_opt ')'
20 | SLIT SLIT
21 define: "define" pattern definition "endef" br
22 | specifiers "define" pattern definition "endef" br
23 | "define" pattern ASSIGN_OP definition "endef" br
24 | specifiers "define" pattern ASSIGN_OP definition "endef" br
25 definition: comment_opt br
26 | comment_opt br exprs_in_def br
27 include: "include" expressions br
28 statements_opt: comment_opt br
29 | comment_opt br statements
30 if_def_kw: "ifdef"
31 | "ifndef"
32 if_eq_kw: "ifeq"
33 | "ifneq"
34 statement: COMMENT
35 | assignment br
36 | function br
37 | rule
38 | conditional
39 | define
40 | include
41 | export br
42 export: "export"
43 | "unexport"
44 | assignment_prefix
45 | assignment_prefix WS targets
46 assignment: pattern ASSIGN_OP comment_opt
47 | pattern ASSIGN_OP exprs_in_assign comment_opt
48 | assignment_prefix ASSIGN_OP comment_opt
49 | assignment_prefix ASSIGN_OP exprs_in_assign comment_opt
50 assignment_prefix: specifiers pattern
51 specifiers: "override"
52 | "export"
53 | "unexport"
54 | "override" "export"
55 | "export" "override"
56 | "undefine"
57 | "override" "undefine"
58 | "undefine" "override"
59 expressions_opt: %empty
60 | expressions
61 expressions: expression
62 | expressions WS expression
63 exprs_nested: expr_nested
64 | exprs_nested WS expr_nested
65 exprs_in_assign: expr_in_assign
66 | exprs_in_assign WS expr_in_assign
67 exprs_in_def: first_expr_in_def
68 | br
69 | br first_expr_in_def
70 | exprs_in_def br
71 | exprs_in_def WS expr_in_recipe
72 | exprs_in_def br first_expr_in_def
73 first_expr_in_def: char_in_def expr_in_recipe
74 | function expr_in_recipe
75 | char_in_def
76 | function
77 exprs_in_recipe: expr_in_recipe
78 | exprs_in_recipe WS expr_in_recipe
79 expression: expression_text
80 | expression_function
81 expr_nested: expr_text_nested
82 | expr_func_nested
83 expr_in_assign: expr_text_in_assign
84 | expr_func_in_assign
85 expr_in_recipe: expr_text_in_recipe
86 | expr_func_in_recipe
87 expression_text: text
88 | expression_function text
89 expr_text_nested: text_nested
90 | expr_func_nested text_nested
91 expr_text_in_assign: text_in_assign
92 | expr_func_in_assign text_in_assign
93 expr_text_in_recipe: text_in_recipe
94 | expr_func_in_recipe text_in_recipe
95 expression_function: function
96 | '(' exprs_nested ')'
97 | expression_text function
98 | expression_function function
99 expr_func_nested: function
100 | '(' exprs_nested ')'
101 | expr_func_nested function
102 | expr_text_nested function
103 expr_func_in_assign: function
104 | expr_func_in_assign function
105 | expr_text_in_assign function
106 expr_func_in_recipe: function
107 | expr_func_in_recipe function
108 | expr_text_in_recipe function
109 function: VAR
110 | "$(" function_name ")"
111 | "$(" function_name WS arguments ")"
112 | "$(" function_name ',' arguments ")"
113 | "$(" function_name ':' expressions ")"
114 | "$(" function_name ASSIGN_OP expressions ")"
115 function_name: function_name_text
116 | function_name_function
117 function_name_text: function_name_piece
118 | function_name_function function_name_piece
119 function_name_piece: CHARS
120 | function_name_piece CHARS
121 function_name_function: function
122 | function_name_text function
123 arguments: %empty
124 | argument
125 | arguments ','
126 | arguments ',' argument
127 argument: expressions
128 rule: targets colon prerequisites NL
129 | targets colon prerequisites recipes NL
130 | targets colon assignment NL
131 target: pattern
132 pattern: pattern_text
133 | pattern_function
134 pattern_text: identifier
135 | pattern_function identifier
136 pattern_function: function
137 | pattern_text function
138 | pattern_function function
139 prerequisites: %empty
140 | targets
141 targets: target
142 | targets WS target
143 recipes: recipe
144 | recipes recipe
145 recipes_opt: comment_opt NL
146 | comment_opt recipes NL
147 recipe: LEADING_TAB exprs_in_recipe
148 | NL conditional_in_recipe
149 | NL COMMENT
150 identifier: CHARS
151 | ','
152 | '('
153 | ')'
154 | identifier CHARS
155 | identifier keywords
156 | identifier ','
157 | identifier '('
158 | identifier ')'
159 text: char
160 | text char
161 text_nested: char_nested
162 | text_nested char_nested
163 text_in_assign: char_in_assign
164 | text_in_assign char_in_assign
165 text_in_recipe: char_in_recipe
166 | text_in_recipe char_in_recipe
167 char: CHARS
168 | SLIT
169 | ASSIGN_OP
170 | ':'
171 char_nested: char
172 | ','
173 char_in_assign: char_nested
174 | '('
175 | ')'
176 | keywords
177 char_in_def: char
178 | '('
179 | ')'
180 | ','
181 | COMMENT
182 | "include"
183 | "override"
184 | "export"
185 | "unexport"
186 | "ifdef"
187 | "ifndef"
188 | "ifeq"
189 | "ifneq"
190 | "else"
191 | "endif"
192 | "define"
193 | "undefine"
194 char_in_recipe: char_in_assign
195 | COMMENT
196 keywords: "include"
197 | "override"
198 | "export"
199 | "unexport"
200 | "ifdef"
201 | "ifndef"
202 | "ifeq"
203 | "ifneq"
204 | "else"
205 | "endif"
206 | "define"
207 | "endef"
208 | "undefine"
209 br: NL
210 | LEADING_TAB
211 colon: ':'
212 | ':' ':'
213 comment_opt: %empty
214 | COMMENT
Some lexer details
All in ' and almost all in " quotes are literals.
')' ::= <unpaired )>
'}' ::= <unpaired }>
"$(" ::= "$(" | "${" – beginning of an expansion
")" ::= ")" | "}" – ending of an expansion
"end of file" ::= <end of file>
COMMENT ::= <# comment (can be multiline)>
ASSIGN_OP ::= "=" | "?=" | ":=" | "::=" | "+=" | "!="
CHARS ::= <sequence of non-whitespace>
WS ::= <sequence of whitespace>
NL ::= "\n" | "\r" | "\r\n"
VAR ::= /\$./
SLIT ::= <single- or double-quote literal>
LEADING_TAB ::= <tabulation at the first position in a line (eats NL)>
Whitespace handling is tricky: lexer should ignore it unless previously
returned token needs that whitespace after it.
Unpaired closing parts of expansions () or }) are just themselves.
While contents of $(...) or ${...} is never a keyword.
Addressing confusion
Two comments express the same question:
On the other hand thou make somehow does parse the makefiles and allows some and rejects others...
You don't need a grammar to parse a languages. However, it's a nice thing to have.
In general, there are more possible languages than there are formal grammars, because grammars have constraints which exclude some languages.
I guess that make reads input and processes it at the same time while managing state, such that at every point it knows what kind of input it can accept next and what to do with it. Formal languages lean to being context-free, which means that they don't keep much record about their state. It's this gap, which makes it hard to formalize make's language, even though implementing it without a grammar is still possible.
It seems there is no official grammar for gnu make, and ...
It would be tricky to write a grammar for make, since the grammar is
extremely context-dependent.
As said by Paul D. Smith in a message in the gnu make mailing list. Paul D. Smith is the official maintainer of gnu make.