I know this question has been answered so many times. However, I still have some points need to be clarified. First let me paste my code snippet:
1 #!/bin/bash
2 declare -a test
3 declare -i counter
4
5 while read x;
6 do
7 test[counter]=$x
9 ((counter++))
10 done < reg.txt
11 echo "---------------------"
12 echo ${test[0]}
13 echo ${test[1]}
And the data in reg.txt is
1.1.1.1
2.2.5.6
45.25.12.45
1.1.2.3
1.1.3.4
I know that to put data in array test properly, I have to use '<' to turn file "reg.txt" into input data. However, how am I supposed to pick out ip address contains "1.1".
At line 10, I tried different things such as:
done < reg.txt|grep "1.1" #Using this way makes the 'test' array empty.
Or this:
done < <(reg.txt | grep "1.1")
The grammar is incorrect. (A lot of people suggest this and I don't know why).
In summary, I mean, is there a way to re-construct file before being read by while loop?
Using this syntax:
done < reg.txt|grep "1.1"
doesn't do what you want it to do; instead, it applies the grep command to the output of the while loop.
The test array is does get populated with 5 values, but those values aren't remembered after the while loop completes - as explained in the answers to this question: Modifying a variable inside while loop is not remembered
What you're looking for is this:
done < <(cat reg.txt | grep "1\.1")
Note that the part within the parenthesis is a pipeline, and it needs to be a valid bash command. (You were missing the "cat" command.) You can test that part separately and verify that it selects the input data that you want.
Related
This question already has answers here:
Read lines from a file into a Bash array [duplicate]
(6 answers)
Closed 5 months ago.
I have 20 files from which I want to grep all the lines that have inside a given id (id123), and save them in a new text file. So, in the end, I would have several txt files, as much as ids we have.
If you have a small number of Ids, you can create a script with the list inside. E.g:
list=("id123" "id124" "id125" "id126")
for i in "${list[#]}"
do
zgrep -Hx $i *.vcf.gz > /home/Roy/$i.txt
done
This would give us 4 txt files (id123.txt...) etc.
However, this list is around 500 ids, so it's much easier to read the txt file that stores the ids and iterate through it.
I was trying to do something like:
list = `cat some_data.txt`
for i in "${list[#]}"
do
zgrep -Hx $i *.vcf.gz > /home/Roy/$i.txt
done
However, this only provides the last id of the file.
If each id in the file is on a distinct line, you can do
while read i; do ...; done < panel_genes_cns.txt
If that is not the case, you can simply massage the file to make it so:
tr -s '[[:space:]]' \\n < panel_genes_cns.txt | while read i; do ...; done
There are a few caveats to be aware of. In each, the commands inside the loop are also reading from the same input stream that while reads from, and this may consume ids unexpectedly. In the second, the pipeline will (depending on the shell) run in a subshell, and any variables defined in the loop will be out of scope after the loop ends. But for your simple case, either of these should work without worrying too much about these issues.
I did not check whole code, but from initally I can see you are using wrong redirection.
You have to use >> instead of >.
> is overwrites and >> is append.
list = `cat pannel_genes_cns.txt`
for i in "${list[#]}"
do
zgrep -Hx $i *.vcf.gz >> /home/Roy/$i.txt
done
This question already has answers here:
How do I iterate over a range of numbers defined by variables in Bash?
(20 answers)
Closed 3 years ago.
I am still learning how to shell script and I have been given a challenge to make it easier for me to echo "Name1" "Name2"..."Name15" and I'm not too sure where to start, I've had ideas but I don't want to look silly if I mess it up. Any help?
I haven't actually tried anything just yet it's all just been mostly thought.
#This is what I wrote to start
#!/bin/bash
echo "Name1"
echo "Name2"
echo "Name3"
echo "Name4"
echo "Name5"
echo "Name6"
echo "Name7"
echo "Name8"
echo "Name9"
echo "Name10"
echo "Name11"
echo "Name12"
echo "Name13"
echo "Name14"
echo "Name15"
My expected results are obviously just for it to output "Name1" "Name2" etc. But I'm looking for a more creative way to do it. If possible throw in a few ways to do it so I can learn. Thank you.
The easiest (possibly not the most creative) way to do this is to use printf:
printf "%s\n" name{1..15}
This relies on bash brace expansion {1..15} to have the 15 strings.
Use a for loop
for i in {1..15};do echo "Name$i";done
A few esoteric solutions, from the least to the most unreasonable :
base64 encoded string :
base64 -d <<<TmFtZTEKTmFtZTIKTmFtZTMKTmFtZTQKTmFtZTUKTmFtZTYKTmFtZTcKTmFtZTgKTmFtZTkKTmFtZTEwCk5hbWUxMQpOYW1lMTIKTmFtZTEzCk5hbWUxNApOYW1lMTUK
The weird chain is your expected result encoded in base64, an encoding generally used to represent binary data as text. base64 -d <<< weirdChain is passing the weird chain as input to the base64 tool and asking it to decode it, which displays your expected result
generate an infinite stream of "Name", truncate it, use line numbers :
yes Name | awk 'NR == 16 { exit } { printf("%s%s\n", $0, NR) }'
yes outputs an infinite stream of what it's passed as argument (or y by default, used to automatize interactive scripts asking for [y/n] confirmation). The awk command exits once it reaches the 16th line, and otherwise prints its input (provided by yes) followed by the line number. The truncature could as easily be done with head -15, and I've tried using the nl "number line" utility or grep -n to number lines, but they always added the line numbers as prefix which required an extra re-formatting step.
read random binary data and hope to stumble on all the lines you want to output :
timeout 1d strings /dev/urandom | grep -Eo "Name(1[0-5]|[1-9])" | sort -uV
strings /dev/urandom will extract ascii sequences from the binary random source /dev/urandom, grep will filter those which respect the format of a line of your expected output and sort will reorder those lines in the correct order. Since sort needs to have a received its whole input before it reorders it and /dev/urandom won't stop producing data, we use timeout 1d to stop reading from /dev/urandom after a whole day in hope it has sifted through enough random data to find your 15 lines (I'm not sure that's even remotely likely).
use an HTTP client to retrieve this page, extract the bash script you posted and execute it.
my_old_script=$(curl "https://stackoverflow.com/questions/57818680/" | grep "#This is what I wrote to start" -A 18 | tail -n+4)
eval "$my_old_script"
curl is a command line tool that can be used as an HTTP client, grep with its -A 18 parameter will select the "This is what I wrote to start" text and the 18 lines that follow, tail will remove the first 3 lines, and eval will execute your script.
While it will be much more efficient than the previous solution, it's an even less reasonable solution because high-rep users can edit your question to make this solution execute arbitrary code on your computer. Ideally you'd be using an HTML-aware parser rather than basic string manipulation to extract the code, but we're not talking about best practices here...
How can I to compare the output in stdout of a program with a model output in an output file? I ask because I am trying to make a grading script. Also, I am thinking of using -q grep, but am not sure how I would use it still.
Please make answer simple because I am a noob at bash.
Important edit:
I want to use this in an if statement. For example:
if(modle_file.txt is identical to stdout when running program); then
echo "Great!"
else
echo "Wrong output. You loose 1 point."
Edit:
The program takes an input. So for example if we do:
%Python3 Program.py
Enter a number: 5
The first 5 (arbitrary) things are:
2, 5, etc (program output)
%
If your file is called example.txt, do
diff example.txt <(program with all its options)
The <() syntax takes the output of the program in parentheses and passes it to the diff command as if it was a text file.
EDIT:
If you just want to check whether the text file and the output of the program are the same or not in an if-clause, you can do:
if [ "$(diff example.txt <(program with all its options))" == "" ]; then
echo 'the outputs are identical'
else
echo 'the outputs differ'
fi
I.e. diff only generates output if the files differ, so an empty string as answer means the files are identical.
EDIT 2:
In principle you can re-direct stdin to a file, like so:
program < input.txt
Now, without further testing, I don't know whether this will work with your python script, but suppose you can put all the input the program expects into such a file, you could do
if [ "$(diff example.txt <(program < input.txt))" == "" ]; then
echo 'Great!'
else
echo 'Wrong output. You loose 1 point.'
fi
EDIT 3::
I wrote a simple test program in python (let's call it program.py):
x = input('type a number: ')
print(x)
y = input('type another number: ')
print(y)
If you run it interactively in the shell with python program.py, and give 5 and 7 as answers, you get the following output:
type a number: 5
5
type another number: 7
7
If you create a file, say input.txt, which contains all the desired input,
5
7
and pipe that into your file like so:
python program.py < input.txt
you get the following output:
type a number: 5
type another number: 7
The reason for the difference is that python (and many other shell programs) treat input differently depending on whether it comes from an interactive shell, a pipe, or a redirected stdin. In this case, input is not echoed as the input comes from input.txt. However, if you run both your code and the student's code using input.txt, the two outputs should still be comparable.
EDIT 4:
As one of the comments states below, it is not necessary to compare the entire output of the diff command against an empty string ("") if you only want to know whether they differ, the return status is enough. It's best to write a small test script in bash (let's call it code_checker.sh),
if diff example.txt <(python program.py < input.txt) > /dev/null; then
echo "Great!"
else
echo "Wrong output. You loose 1 point."
fi
the >/dev/null part in the if-clause re-directs the output of diff to a special device, effectively ignoring it. If you have a lot of output, it might be better to use cmp like mentioned by user1934428.
One way to do this would be to redirect the stdout output to a file, like this:
myCommand > /folder/file.txt
And then run a diff command to compare the two files
diff /folder/file.txt /folder/model_output.txt
Edit: To use this on an if statement, you could do the following:
if [ -z "$(diff /folder/file.txt /folder/model_output.txt 2>&1)" ]; then echo "Great!"; else echo "Wrong output. You loose 1 point."; fi
If the files are equal, it will print Great!, otherwise it will print Wrong output. You loose 1 point.
Since you are not interested in the actual differences, but only in whether they are identical, I think cmp is the best choice (and faster if the files are large):
if cmp -s example.txt <(your program goes here)
then
echo identical
fi
Note that this compares only stdout (as you requested), not stderr.
So, I am building a bash script which iterates through folders named by numbers from 1 to 9. The script depends on getting the folder names by user input. My intention is to use a for loop using read input to get a folder name or a range of folder names and then do some stuff.
Example:
Let's assume I want to make a backup with rsync -a of a certain range of folders. Usually I would do:
for p in {1..7}; do
rsync -a $p/* backup.$p
done
The above would recursively backup all content in the directories 1 2 3 4 5 6 and 7 and put them into folders named as 'backup.{index-number}'. It wouldn't catch folders/files with a leading . but that is not important right now.
Now I have a similar loop in an interactive bash script. I am using select and case statements for this task. One of the options in case is this loop and it shall somehow get a range of numbers from user input. This now becomes a problem.
Problem:
If I use read to get the range then it fails when using {1..7} as input. The input is taken literally and the output is just:
{1..7}
I really would like to know why this happens. Let me use a more descriptive example with a simple echo command.
var={1..7} # fails and just outputs {1..7}
for p in $var; do echo $p;done
read var # Same result as above. Just outputs {1..7}
for p in $var; do echo $p;done
for p in {1..7}; do echo $p;done # works fine and outputs the numbers 1-7 seperated with a newline.
I've found a workaround by storing the numbers in an array. The user can then input folder names seperated by a space character like this: 1 2 3 4 5 6 7
read -a var # In this case the output is similar to the 3rd loop above
for p in ${var[#]}; do echo $p; done
This could be a way to go but when backing up 40 folders ranging from 1-40 then adding all the numbers one-by-one completely makes my script redundant. One could find a solution to one of the millennium problems in the same time.
Is there any way to read a range of numbers like {1..9} or could there be another way to get input from terminal into the script so I can iterate through the range within a for-loop?
This sounds like a question for google but I am obviously using the wrong patterns to get a useful answer. Most of similar looking issues on SO refer to brace and parameter expansion issues but this is not exactly the problem I have. However, to me it feels like the answer to this problem is going in a similar direction. I fail to understand why when a for-loop for assigning {1..7} to a variable works but doing the same like var={1..7} doesn't. Plz help -.-
EDIT: My bash version:
$ echo $BASH_VERSION
4.2.25(1)-release
EDIT2: The versatility of a brace expansion is very important to me. A possible solution should include the ability to define as many ranges as possible. Like I would like to be able to choose between backing up just 1 folder or a fixed range between f.ex 4-22 and even multiple options like folders 1,2,5,6-7
Brace expansion is not performed on the right-hand side of a variable, or on parameter expansion. Use a C-style for loop, with the user inputing the upper end of the range if necessary.
read upper
for ((i=1; i<=$upper; i++)); do
To input both a lower and upper bound separated by whitespace
read lower upper
for (i=$lower; i <= $upper; i++)); do
For an arbitrary set of values, just push the burden to the user to generate the appropriate list; don't try to implement your own parser to process something like 1,2,20-22:
while read p; do
rsync -a $p/* backup.$p
done
The input is one value per line, such as
1
2
20
21
22
Even if the user is using the shell, they can call your script with something like
printf '%s\n' 1 2 20..22 | backup.sh
It's easier for the user to generate the list than it is for you to safely parse a string describing the list.
The evil eval
$ var={1..7}
$ for i in $(eval echo $var); do echo $i; done
this also works,
$ var="1 2 {5..9}"
$ for i in $(eval echo $var); do echo $i; done
1
2
5
6
7
8
9
evil eval was a joke, that is, as long as you know what you're evaluating.
Or, with awk
$ echo "1 2 5-9 22-25" |
awk -v RS=' ' '/-/{split($0,a,"-"); while(a[1]<=a[2]) print a[1]++; next}1'
1
2
5
6
7
8
9
22
23
24
25
First off, I appreciate any and all help in answering this question.
I have a command in a bash script that will output the following:
255 254 253 252 ... 7 6 5 4 3 2 1
It is a specific list of numbers, beginning with the largest (which is what I would like), then going to the smallest. The dataset is space-delimited. The output above (except including all numbers), is what you would see if you ran this command in the terminal on a linux machine, or through a bash script.
I have configured my apache2 server to allow for cgi/bash through the cgi-bin directory. When I run this command in a bash file from the web, I get the expected output.
What I'm looking for is for a way to be able to put these numbers each as a separate entry in a drop-down box for selection, meaning the user can select one point of data (254, for example) from the drop down menu.
I'm not sure what I'm doing with this, so any help would be appreciated. I'm not sure if I need to convert the data into an array, or what. The drop down menu can be on the same page of the bash script, but wherever it is, it has to update it's list of numbers from the command every time it is run.
Thank you for your help.
I've always found this site useful when fiddling with shell scripts: http://tldp.org/LDP/abs/html/
you'll have to get your output into an array using some sort of string manipulation using the spaces as delimiters, then loop over that to build some html output - so the return value will basically just output your select box on the page where you execute your cgi/bash script.
-sean
Repeating the answer (since the original question was marked as duplicate):
you can write a bash for loop to do everything. This just prints out the elements:
for i in `seq 1 "${#x[*]}"`; do
echo "|${x[i]} |"
done
To get the alignment correct, you need to figure out the max length (one loop) and then print out the terms:
# w will be the length
w=0
for i in `seq 1 "${#x[*]}"`; do
if [ $w -lt ${#x[$i]} ]; then w=${#x[$i]}; fi
done
for i in `seq 1 $((w+2))`; do printf "%s" "-"; done
printf "\n"
for i in `seq 1 "${#x[*]}"`; do
printf "|%-$ws |\n" ${#x[$i]}
done
for i in `seq 1 $((w+2))`; do printf "%s" "-"; done
printf "\n"