shell script run multiple files - bash
I want to use a shell scrip in a for-loop that run 100 files in parallel.
Currently, I have a shell script of the following format:
#!/bin/bash
NUM=10
python a1.py $((NUM + 0)) &
python a2.py $((NUM + 2)) &
python a3.py $((NUM + 4)) &
python a4.py $((NUM + 6)) &
python a5.py $((NUM + 8)) &
Now, if I have a1.py, a2.py, a3.py .... a100.py, and I want to run them in parallel, how do I do it in for-loop?
If you have bash version 4 and run this:
echo {10..208..2}
You will get this:
10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60
62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 100 102 104 106 108
110 112 114 116 118 120 122 124 126 128 130 132 134 136 138 140 142 144 146
148 150 152 154 156 158 160 162 164 166 168 170 172 174 176 178 180 182 184
186 188 190 192 194 196 198 200 202 204 206 208
which looks like your series. Then, if you want to run lots of jobs in parallel, I would use GNU Parallel. That offers you {#} as a placeholder for the job number. So, if you run this:
parallel -k echo {#} {} ::: {10..208..2}
You will get this:
1 10
2 12
3 14
4 16
5 18
So, to run your actual scripts, something like:
parallel -k --dry-run 'python a{#}.py {}' ::: {10..208..2}
Sample Output
python a1.py 10
python a2.py 12
python a3.py 14
python a4.py 16
...
...
If that looks good, run again without the --dry-run and without the -k which keeps the output in order to make it easier to debug.
TLDR;
My most concise answer, with GNU Parallel is:
parallel python a{#}.py {} ::: {10..208..2}
Or if you don't have bash version 4:
parallel python a{#}.py {} ::: $(seq 10 2 208)
NUM=10
for ((i=0; i<100; i++)); do echo python a$(($i+1)).py $(($NUM+$i*2)); done
Output:
python a1.py 10
python a2.py 12
python a3.py 14
.
.
.
python a98.py 204
python a99.py 206
python a100.py 208
If this looks okay, use:
NUM=10
for ((i=0; i<100; i++)); do python a$(($i+1)).py $(($NUM+$i*2)) & done
Related
Can't generate any alignments in MCScanX
I'm trying to find collinearity between a group of genes from two different species using MCScanX. But I don't know what I could be possibly doing wrong anymore. I've checked both input files countless times (.gff and .blast), and they seem to be in line with what the manual says. Like, for the first species, I've downloaded the gff file from figshare. I already had the fasta file containing only the proteins of interest (that I also got from figshare), so gene ids matched. Then, I downloaded both the gff and the protein fasta file from coffee genome hub. I used the coffee proteins fasta file as the reference genome in rBLAST to align the first specie's genes against it. After blasting (and keeping only the first five best alignments with e-values greater than 1e-10), I filtered both gff files so they only contained genes that matched those in the blast file, and then concatenated them. So the final files look like this: View (test.blast) #just imagine they're tab separated values sp1.id1 sp2.id1 44.186 43 20 1 369 411 206 244 0.013 37.4sp1.id1 sp2.id2 25.203 123 80 4 301 413 542 662 0.00029 43.5sp1.id1 sp2.id3 27.843 255 130 15 97 333 458 676 1.75e-05 47.8sp1.id1 sp2.id4 26.667 105 65 3 301 396 329 430 0.004 39.7sp1.id1 sp2.id5 27.103 107 71 3 301 402 356 460 0.000217 43.5sp1.id2 sp2.id6 27.368 95 58 2 40 132 54 139 0.41 32sp1.id2 sp2.id7 27.5 120 82 3 23 138 770 888 0.042 35sp1.id2 sp2.id8 38.596 57 35 0 21 77 126 182 0.000217 42sp1.id2 sp2.id9 36.17 94 56 2 39 129 633 725 1.01e-05 46.6sp1.id2 sp2.id10 37.288 59 34 2 75 133 345 400 0.000105 43.1sp1.id3 sp2.id11 33.846 65 42 1 449 512 360 424 0.038 37.4sp1.id3 sp2.id12 40 50 16 2 676 725 672 707 6.7 30sp1.id3 sp2.id13 31.707 41 25 1 370 410 113 150 2.3 30.4sp1.id3 sp2.id14 31.081 74 45 1 483 550 1 74 3.3 30sp1.id3 sp2.id15 35.938 64 39 1 377 438 150 213 0.000185 43.5 View (test.gff) #just imagine they're tab separated values ex0 sp2.id1 78543527 78548673ex0 sp2.id2 97152108 97154783ex1 sp2.id3 16555894 16557150ex2 sp2.id4 3166320 3168862ex3 sp2.id5 7206652 7209129ex4 sp2.id6 5079355 5084496ex5 sp2.id7 27162800 27167939ex6 sp2.id8 5584698 5589330ex6 sp2.id9 7085405 7087405ex7 sp2.id10 1105021 1109131ex8 sp2.id11 24426286 24430072ex9 sp2.id12 2734060 2737246ex9 sp2.id13 179361 183499ex10 sp2.id14 893983 899296ex11 sp2.id15 23731978 23733073ts1 sp1.id1 5444897 5448367ts2 sp1.id2 28930274 28935578ts3 sp1.id3 10716894 10721909 So I moved both files to the test folder inside MCScanX directory and ran MCScan (using Ubuntu 20.04.5 LTS, the WSL feature) with: ../MCScanX ./test I've also tried ../MCScanX -b 2 ./test (since "-b 2" is the parameter for inter-species patterns of syntenic blocks) but all I ever get is 255 matches imported (17 discarded)85 pairwise comparisons0 alignments generated What am I missing???? I should be getting a test.synteny file that, as per the manual's example, looks like this: ## Alignment 0: score=9171.0 e_value=0 N=187 at1&at1 plus 0- 0: AT1G17240 AT1G72300 0 0- 1: AT1G17290 AT1G72330 0 ... 0-185: AT1G22330 AT1G78260 1e-63 0-186: AT1G22340 AT1G78270 3e-174 ##Alignment 1: score=5084.0 e_value=5.6e-251 N=106 at1&at1 plus
How to replace list of numbers in column for random numbers in other column in BASH environment
I have a tab file with two columns like that 5 6 14 22 23 25 27 84 85 88 89 94 95 98 100 6 94 6 8 17 20 193 205 209 284 294 295 299 304 305 307 406 205 284 307 406 2 10 13 40 47 58 2 13 40 87 and the desired output should be 5 6 14 22 23 25 27 84 85 88 89 94 95 98 100 14 27 6 8 17 20 193 205 209 284 294 295 299 304 305 307 406 6 209 299 305 2 10 13 23 40 47 58 87 10 23 40 58 I would like to change the numbers in 2nd column for random numbers in 1st column resulting in an output in 2nd column with the same number of numbers. I mean e.g. if there are four numbers in 2nd column for x row, the output must have four random numbers from 1st column for this row, and so on... I'm try to create two arrays by AWK and split and replace every number in 2nd column for numbers in 1st column but not in a randomly way. I have seen the rand() function but I don't know exactly how joint these two things in a script. Is it possible to do in BASH environment or are there other better ways to do it in BASH environment? Thanks in advance
awk to the rescue! $ awk -F'\t' 'function shuf(a,n) {for(i=1;i<n;i++) {j=i+int(rand()*(n+1-i)); t=a[i]; a[i]=a[j]; a[j]=t}} function join(a,n,x,s) {for(i=1;i<=n;i++) {x=x s a[i]; s=" "} return x} BEGIN{srand()} {an=split($1,a," "); shuf(a,an); bn=split($2,b," "); delete m; delete c; j=0; for(i=1;i<=bn;i++) m[b[i]]; # pull elements from a upto required sample size, # not intersecting with the previous sample set for(i=1;i<=an && j<bn;i++) if(!(a[i] in m)) c[++j]=a[i]; cn=asort(c); print $1 FS join(c,cn)}' file 5 6 14 22 23 25 27 84 85 88 89 94 95 98 100 85 94 6 8 17 20 193 205 209 284 294 295 299 304 305 307 406 20 205 294 295 2 10 13 23 40 47 58 87 10 13 47 87 shuffle (standard algorithm) the input array, sample required number of elements, additional requirement is no intersection with the existing sample set. Helper structure map to keep existing sample set and used for in tests. The rest should be easy to read.
Assuming that there is a tab delimiting the two columns, and each column is a space delimited list: awk 'BEGIN{srand()} {n=split($1,a," "); m=split($2,b," "); printf "%s\t",$1; for (i=1;i<=m;i++) printf "%d%c", a[int(rand() * n) +1], (i == m) ? "\n" : " " }' FS=\\t input
Try this: # This can be an external file of course # Note COL1 and COL2 seprated by hard TAB cat <<EOF > d1.txt 5 6 14 22 23 25 27 84 85 88 89 94 95 98 100 6 94 6 8 17 20 193 205 209 284 294 295 299 304 305 307 406 205 284 307 406 2 10 13 40 47 58 2 13 40 87 EOF # Loop to read each line, not econvert TAB to:, though could have used IFS cat d1.txt | sed 's/ /:/' | while read LINE do # Get the 1st column data COL1=$( echo ${LINE} | cut -d':' -f1 ) # Get col1 number of items NUM_COL1=$( echo ${COL1} | wc -w ) # Get col2 number of items NUM_COL2=$( echo ${LINE} | cut -d':' -f2 | wc -w ) # Now split col1 items into an array read -r -a COL1_NUMS <<< "${COL1}" COL2=" " # THis loop runs once for each COL2 item COUNT=0 while [ ${COUNT} -lt ${NUM_COL2} ] do # Generate a random number to use as teh random index for COL1 COL1_IDX=${RANDOM} let "COL1_IDX %= ${NUM_COL1}" NEW_NUM=${COL1_NUMS[${COL1_IDX}]} # Check for duplicate DUP_FOUND=$( echo "${COL2}" | grep ${NEW_NUM} ) if [ -z "${DUP_FOUND}" ] then # Not a duplicate, increment loop conter and do next one let "COUNT = COUNT + 1 " # Add the random COL1 item to COL2 COL2="${COL2} ${COL1_NUMS[${COL1_IDX}]}" fi done # Sort COL2 COL2=$( echo ${COL2} | tr ' ' '\012' | sort -n | tr '\012' ' ' ) # Print echo ${COL1} :: ${COL2} done Output: 5 6 14 22 23 25 27 84 85 88 89 94 95 98 100 :: 88 95 6 8 17 20 193 205 209 284 294 295 299 304 305 307 406 :: 20 299 304 305 2 10 13 40 47 58 :: 2 10 40 58
ruby use upto with range variable?
I'm trying to use a variable range with ruby, but my code does not work; ruby -e ' input2=145..170 ; input3= input2.to_s.gsub(/(.*?)\.\.(.*?)/) { 5.upto($2.to_i) { |i| print i, " " } }; print input3' > zzmf But I obtained 5170 This part fails: 5.upto($2.to_i) { |i| print i, " " } I expected: 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 5170
I don't think gsub is what you need, try the match example below. [2] gets the second match from the regex /(\d+)..(\d+)/ applied to "147..170" 5.upto("147..170".match(/(\d+)\.\.(\d+)/)[2].to_i) { |i| print i, " "} gsub is intended for string substitution. https://ruby-doc.org/core-2.1.4/String.html#method-i-gsub
I see my code and I confuse in regular expression I use this .*? and the correct is this .* (.*)/ ruby -e ' input2=145..170 ; input3= input2.to_s.gsub(/(.*?)\.\.(.*)/) { 5.upto($2.to_i) { |i| print i, " " } }; print input3' > zzmf thanks for your responses
Print numbers 1 to 100 in 10 columns
I want to print the numbers 1 to 100 numbers, but in columns: first 1 to 10 in first column, 11 to 20 in 2nd column, then 21 to 30 in 3rd column, ..., 91 to 100 in 10th column. How can I achieve this in Bash? I have tried: #!/bin/bash for ((i=1; i <= 100 ; i++)) do echo " $i" done
Its a bit heavy since it spawns many subprocesses .. but posting it as a 1-liner paste <(seq 1 10) <(seq 11 20) <(seq 21 30) <(seq 31 40) <(seq 41 50) <(seq 51 60) <(seq 61 70) <(seq 71 80) <(seq 81 90) <(seq 91 100) for 1-10 in 1st column, 11-20 in 2nd column and so on.. and seq 1 100 | paste - - - - - - - - - - for 1-10 in 1st row, 11-20 in 2nd row and so on.. Note: There are 10 hypens in 2nd command and in the 1st one, <(command) means process substitution i.e substitutes the the output of the process Edit: Approach purely using for loop for ((i=1;i<=10;i++)); do for ((j=i;j<=(i+90);j+=10)); do printf "%2d " $j done echo done for 1-10 in 1st column, 11-20 in 2nd column and so on.. and for ((i=0;i<10;i++)); do for ((j=1;j<=10;j++)); do printf "%2d " $[$i*10+$j] done echo done for 1-10 in 1st row, 11-20 in 2nd row and so on..
There is an easier way! $ seq 100 | pr -10t 1 11 21 31 41 51 61 71 81 91 2 12 22 32 42 52 62 72 82 92 3 13 23 33 43 53 63 73 83 93 4 14 24 34 44 54 64 74 84 94 5 15 25 35 45 55 65 75 85 95 6 16 26 36 46 56 66 76 86 96 7 17 27 37 47 57 67 77 87 97 8 18 28 38 48 58 68 78 88 98 9 19 29 39 49 59 69 79 89 99 10 20 30 40 50 60 70 80 90 100
convert comma separated list in text file into columns in bash
I've managed to extract data (from an html page) that goes into a table, and I've isolated the columns of said table into a text file that contains the lines below: [30,30,32,35,34,43,52,68,88,97,105,107,107,105,101,93,88,80,69,55], [28,6,6,50,58,56,64,87,99,110,116,119,120,117,114,113,103,82,6,47], [-7,,,43,71,30,23,28,13,13,10,11,12,11,13,22,17,3,,-15,-20,,38,71], [0,,,3,5,1.5,1,1.5,0.5,0.5,0,0.5,0.5,0.5,0.5,1,0.5,0,-0.5,-0.5,2.5] Each bracketed list of numbers represents a column. What I'd like to do is turn these lists into actual columns that I can work with in different data formats. I'd also like to be sure to include that blank parts of these lists too (i.e., "[,,,]") This is basically what I'm trying to accomplish: 30 28 -7 0 30 6 32 6 35 50 43 3 34 58 71 5 43 56 30 1.5 52 64 23 1 . . . . . . . . . . . . I'm parsing data from a web page, and ultimately planning to make the process as automated as possible so I can easily work with the data after I output it to a nice format. Anyone know how to do this, have any suggestions, or thoughts on scripting this?
Since you have your lists in python, just do it in python: l=[["30", "30", "32"], ["28","6","6"], ["-7", "", ""], ["0", "", ""]] for i in zip(*l): print "\t".join(i) produces 30 28 -7 0 30 6 32 6
awk based solution: awk -F, '{gsub(/\[|\]/, ""); for (i=1; i<=NF; i++) a[i]=a[i] ? a[i] OFS $i: $i} END {for (i=1; i<=NF; i++) print a[i]}' file 30 28 -7 0 30 6 32 6 35 50 43 3 34 58 71 5 43 56 30 1.5 52 64 23 1 .......... ..........
Another solution, but it works only for file with 4 lines: $ paste \ <(sed -n '1{s,\[,,g;s,\],,g;s|,|\n|g;p}' t) \ <(sed -n '2{s,\[,,g;s,\],,g;s|,|\n|g;p}' t) \ <(sed -n '3{s,\[,,g;s,\],,g;s|,|\n|g;p}' t) \ <(sed -n '4{s,\[,,g;s,\],,g;s|,|\n|g;p}' t) 30 28 -7 0 30 6 32 6 35 50 43 3 34 58 71 5 43 56 30 1.5 52 64 23 1 68 87 28 1.5 88 99 13 0.5 97 110 13 0.5 105 116 10 0 107 119 11 0.5 107 120 12 0.5 105 117 11 0.5 101 114 13 0.5 93 113 22 1 88 103 17 0.5 80 82 3 0 69 6 -0.5 55 47 -15 -0.5 -20 2.5 38 71 Updated: or another version with preprocessing: $ sed 's|\[||;s|\][,]\?||' t >t2 $ paste \ <(sed -n '1{s|,|\n|g;p}' t2) \ <(sed -n '2{s|,|\n|g;p}' t2) \ <(sed -n '3{s|,|\n|g;p}' t2) \ <(sed -n '4{s|,|\n|g;p}' t2)
If a file named data contains the data given in the problem (exactly as defined above), then the following bash command line will produce the output requested: $ sed -e 's/\[//' -e 's/\]//' -e 's/,/ /g' <data | rs -T Example: cat data [30,30,32,35,34,43,52,68,88,97,105,107,107,105,101,93,88,80,69,55], [28,6,6,50,58,56,64,87,99,110,116,119,120,117,114,113,103,82,6,47], [-7,,,43,71,30,23,28,13,13,10,11,12,11,13,22,17,3,,-15,-20,,38,71], [0,,,3,5,1.5,1,1.5,0.5,0.5,0,0.5,0.5,0.5,0.5,1,0.5,0,-0.5,-0.5,2.5] $ sed -e 's/[//' -e 's/]//' -e 's/,/ /g' <data | rs -T 30 28 -7 0 30 6 43 3 32 6 71 5 35 50 30 1.5 34 58 23 1 43 56 28 1.5 52 64 13 0.5 68 87 13 0.5 88 99 10 0 97 110 11 0.5 105 116 12 0.5 107 119 11 0.5 107 120 13 0.5 105 117 22 1 101 114 17 0.5 93 113 3 0 88 103 -15 -0.5 80 82 -20 -0.5 69 6 38 2.5 55 47 71