I have been looking and couldnt find clear clues to verify what I am deducing from a script given to me.
so file.txt is an opened file (by the file descriptor 3) and is constantly adding a new line by a script that records timestamp. Does the following piece make into the while loop each time a new line is added to the file?
exec 3 < /path/file.txt
while read <&3
command
command..
done
So as long as I dont close the file descriptor, a new line added to my file.txt will always activate the while loop, right?
Please help me clear this up. Thanks
To read from file descriptor 3, use read -u 3 (see Bash builtins). Don't forget to specify the variable name into which the value should be read.
Once read detects EOF, it stays at EOF; it won't spot additions to the file after that. So, if the code adding lines to the file is slower than the code in this script, you will reach and end point and the loop will terminate. If you don't want that, consider using tail -f /path/file.txt, and maybe process substitution too:
while read -u 3 line
do
command1
command2
done 3< <(tail -f /path/file.txt)
Or, if you want to do the exec:
exec 3< <(tail -f /path/file.txt)
while read -u 3 line
do
command1
command2
done
Note that the tail -f loops will never finish until you interrupt the script in some way.
So as long as I dont close the file descriptor, a new line added to my
file.txt will always activate the while loop, right?
Answer: wrong.
Redirecting exec 3 < /path/file.txt gives you the ability to read from /path/file.txt using the file descriptor, but does nothing to allow any type of triggering from /path/file.txt to your code. Think about it this way. If there is a new line in /path/file.txt, you can read it, but the redirection provides no way of knowing whether or not a new line has been added to the file for your code to respond to. It's still up to your code to check.
Related
Each line in a given file 'a.txt' contains the directory/path to another unique file. Suppose we want to parse 'a.txt' line-by-line, extract the path in string format, and then use a tool such as vim to process the file at this path, and so on.
After going through this thread - Read a file line by line assigning the value to a variable, I wrote the following script, say 'open-file.sh' on bash (I'm new to it)
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
vim -c ":q" -cq $line # Just open the file and close it using :q
done < "$1"
We would then run the above script as -
./open-file.sh a.txt
The problem is that although the path to a new file is correctly specified by $line, when vim opens the file, vim continues to receive the text contained in 'a.txt' as a command. How can I write a script where I can correctly obtain the path from 'a.txt', open it using vim, and then continue parsing the remaining lines in 'a.txt' ?
Replace:
vim -c ":q" -cq $line
With:
vim -c ":q" -cq "$line" </dev/tty
The redirection </dev/tty tells vim to take its standard input from the terminal. Without that, the standard input for vim is "$1".
Also, it is good practice to put $line in double-quotes to protect it from word splitting, etc.
Lastly, while vim is excellent for interactive work, if your end-goal is fully automated processing of each file, you might want to consider tools such as sed or awk.
Although I'm not sure of your ultimate goal, this shell command will execute vim once per line in a.txt:
xargs -o -n1 vim -c ':q' < a.txt
As explained in the comments to Read a file line by line assigning the value to a variable, the issue you're encountering is due to the fact that vim is an interactive program and thus continues to read input from $line.
The problem was already mentioned in a comment under the answer you based your script on.
vim is consuming stdin which is given to the loop by done < $1. We can observe the same behavior in the following example:
$ while read i; do cat; done < <(seq 3)
2
3
<(seq 3) simulates a file with the three lines 1, 2, and 3. Instead of three silent iterations we get only one iteration and the output 2 and 3.
stdin is not only passed to read in the head of the loop, but also to cat in the body of the loop. Therefore read reads one line, the loop is entered, cat reads all remaining lines, stdin is empty, read has nothing to read anymore, the loop exits.
You could circumvent the problem by redirecting something to vim, however there is an even better way. You don't need the loop at all:
< "$1" xargs -d\\n -n1 vim -c :q -cq
xargs will execute vim once for every line in the file given by $1.
I have a file a.txt and each line contains a parameter. Now I want to use mpiexec to call my program such as a.out to calculate with each parameter. So I use linux shell script to handle this. The code is sample
cat a.txt | while read line
do
mpiexec -v -hostfile hosts -np 16 ./a.out ${line}
done
Unexpectedly, the script end after processing only one line of file a.txt. So, it is because of the wrong use of pipe? How can I tackle with this problem?
#!/bin/bash
for LINE in `cat a.txt | xargs -r`; do
mpiexec -v -hostfile hosts -np 16 ./a.out $LINE
done
I had this issue too. Claudio's solution helped set me on the right path to understanding why the loop exits after the first iteration. First off, here is a solution which is pretty close to what you wrote:
cat a.txt | while read line; do
</dev/null mpiexec -np 16 ./a.out ${line}
done
Note that I am just using mpiexec on a local computer, (python's threading situation is bad enough to need this) so I can't test if this works with separate hosts. You can try adding that back in yourself.
The reason that your script didn't work is that mpiexec seems to gobble up whatever is attached to the standard input. I assume it does this so that in case a.out needs that input, it would gobble all the input and send it along with the command to run a.out that gets sent to the other servers. The result is that on the first iteration, read reads the first line from your file. Then mpiexec reads the rest of the lines, even though a.out probably doesn't use them in your case. Then on the second iteration, read tries to read more lines, but since mpiexec already read the rest, read is told that the end of file has been reached, so the loop exits.
Since we want to prevent mpiexec from reading the standard in, we redirect mpiexec's standard in to come from /dev/null. Since /dev/null always contains nothing, mpiexec will read nothing and leave the standard input alone.
I have an external device that I need to power up and then wait for it to get started properly. The way I want to do this is by connecting to it via serial port (via Plink which is a command-line tool for PuTTY) and read all text lines that it prints and try to find the text string that indicates that it has been started properly. When that text string is found, the script will proceed.
The problem is that I need to read these text lines in real-time. So far, I have only seen methods for calling a command and then process its output when that command is finished. Alternatively, I could let Plink run in the background by appending an & to the command and redirecting the output to a file. But the problem is that this file will be empty from the beginning so the script will just proceed directly. Is there maybe a way to wait for a new line of a certain file and read it once it comes? Or does anyone have any other ideas how to accomplish this?
Here is the best solution I have found so far:
./plink "connection_name" > new_file &
sleep 10 # Because I know that it will take a little while before the correct text string pops up but I don't know the exact time it will take...
while read -r line
do
# If $line is the correct string, proceed
done < new_file
However, I want the script to proceed directly when the correct text string is found.
So, in short, is there any way to access the output of a command continously before it has finished executing?
This might be what you're looking for:
while read -r line; do
# do your stuff here with $line
done < <(./plink "connection_name")
And if you need to sleep 10:
{
sleep 10
while read -r line; do
# do your stuff here with $line
done
} < <(./plink "connection_name")
The advantage of this solution compared to the following:
./plink "connection_name" | while read -r line; do
# do stuff here with $line
done
(that I'm sure someone will suggest soon) is that the while loop is not run in a subshell.
The construct <( ... ) is called Process Substitution.
Hope this helps!
Instead of using a regular file, use a named pipe.
mkfifo new_file
./plink "connection_name" > new_file &
while read -r line
do
# If $line is the correct string, proceed
done < new_file
The while loop will block until there is something to read from new_file, so there is no need to sleep.
(This is basically what process substitution does behind the scenes, but doesn't require any special shell support; POSIX shell does not support process substitution.)
Newer versions of bash (4.2 or later) also support an option to allow the final command of a pipeline to execute in the current shell, making the simple solution
shopt +s lastpipe
./plink "connection_name" | while read -r line; do
# ...
done
possible.
I'm using a "while" loop within a shell script (BASH) to read line of a file (one by one) -- "Fortunately", its not working the No. of times the file has lines in it.
Here's the summary:
$ cat inputfile.txt
1
2
3
4
5
Now, the shell script content is pretty simple as shown below:
#!/bin/bash
while read line
do
echo $line ----------;
done < inputfile.txt;
The above script code works just fine..... :). It shows all the 5 lines from inputfile.txt.
Now, I have another script whose code is like:
#!/bin/bash
while read line
do
echo $line ----------;
somevariable="$(ssh sshuser#sshserver "hostname")";
echo $somevariable;
done < inputfile.txt;
Now, in this script, while loop just shows only line "1 ---------" and exits out from the loop after showing valid value for "$somevariable"
Any idea, what I'm missing here. I didn't try using some number N < inputfile.txt and using done <&N way (i.e. to change the input redirector by using a file pointed by N descriptor)
.... but I'm curious why this simple script is not working for N no. of times, when I just added a simple variable declaration which is doing a "ssh" operation in a child shell.
Thanks.
You might want to add the -n option to the ssh command. This would prevent it to "swallow" your inputfile.txt as its standard input.
Alternatively, you might just redirect ssh stdin from /dev/null, eg:
somevariable="$(ssh sshuser#sshserver "hostname" </dev/null)";
I have a file file.txt with contents like
i love this world
I hate stupid managers
I love linux
I have MS
When I do the following:
for line in `cat file.txt`; do
echo $line
done
It gives output like
I
love
this
world
I
..
..
But I need the output as entire lines like below — any thoughts ?
i love this world
I hate stupid managers
I love linux
I have MS
while read -r line; do echo "$line"; done < file.txt
As #Zac noted in the comments, the simplest solution to the question you post is simply cat file.txt so i must assume there is something more interesting going on so i have put the two options that solve the question as asked as well:
There are two things you can do here, either you can set IFS (Internal Field Separator) to a newline and use existing code, or you can use the read or line command in a while loop
IFS="
"
or
(while read line ; do
//do something
done) < file.txt
I believe the question was how to read in an entire line at a time. The simple script below will do this. If you don't specify a variable name for "read" it will stuff the entire line into the variable $REPLY.
cat file.txt|while read; do echo $REPLY; done
Dave..
You can do it by using read if the file is coming into stdin. If you need to do it in the middle of a script that already uses stdin for other purposes, you can temporarily reassign the stdin file descriptor.
#!/bin/bash
file=$1
# save stdin to usually unused file descriptor 3
exec 3<&0
# connect the file to stdin
exec 0<"$file"
# read from stdin
while read -r line
do
echo "[$line]"
done
# when done, restore stdin
exec 0<&3
Try
(while read l; do echo $l; done) < temp.txt
read: Read a line from the standard
input and split it into fields.
Reads a single line from the standard input, or from file
descriptor FD
if the -u option is supplied. The line is split into fields as with word
splitting, and the first word is assigned to the first NAME, the second
word to the second NAME, and so on, with any leftover words assigned
to
the last NAME. Only the characters found in $IFS are
recognized as word
delimiters.