I'm trying to use the contents of a text file as the input for a command. I know how to read a file just fine. However, when I pass the read line to the command I want to execute, the script starts skipping every other line.
Given a plain text file named queue:
one
two
three
four
This prints out each line as expected:
queue=`pwd`/queue
while read input; do
echo $input
done < $queue
output:
one
two
three
four
However, when I pass $input off to the command, every other line is skipped:
queue=`pwd`/queue
while read input; do
echo $input
transcode-video --dry-run $input
done < $queue
output (transcode-video outputs a bunch of stuff, but I omitted that for brevity. I don't believe it is relevant):
one
three
I managed to get my script working by first reading the whole file into an array and then iterating over the array, but I still don't understand why directly looping over the file doesn't work. I'm assuming the file pointer is getting advanced somehow, but I cannot figure out why. transcode-video is a ruby gem. Is there something I'm not aware of going on behind the scenes when the ruby program is executed? The author of the gem provided a sample script that actually strips lines out of the file using a sed command, and that works fine.
Can someone explain what is going on here?
The launched app tries to process stdin, and reads a line. Try:
transcode-video --dry-run $input </dev/null
Or check the manual for a command-line flag that does the job.
Related
Sorry if this has been asked before but I couldn't find anything.
I have an issue with a (example) script that needs to do 2 things:
echo the output of the tree command, and echo the same again, but this time passing the output through a pipe to another script.
The first part works fine, I get the expected tree of of directories and files, output on the terminal.
However when passing the output through the pipe, all I get on the other side is the first line. Why is this?
I have tried assigning the output to a temporary file and then using cat on this before passing through, but with no success.
Thanks
example_script:
tree Folder
tree Folder > test.pipe
...
#The other script reads from the pipe like so:
read thisthing < test.pipe
echo $thisthing #I have also tried cat $thisthing
I have a simple bash script that, so far, just reads the each line of a file and prints it. Simple enough:
while read i
do
echo $i
#otherViewDef=`grep -i $currentView $viewssqlfile`
done <$viewsdeffile
This script works as expected, unless the commented line is uncommented. If this is this case, the loop exits after echoing the first line of the file. I understand that this should not work as both currentView and viewsqlfile are unset, but what is the justification for this behavior as opposed to reporting an error and giving a non-zero return signal?
I think there's something different; this can't be the actual script, because the errors would be different. Assuming $currentView is set but $viewssqlfile is not, the assignment executes
grep -i $currentView
which reads from stdin, which means it greps the contents of $viewsdeffile. It finds no matches, so prints nothing. After that, the read i has nothing to read, returns false, and the loop exits.
In other words, if the controlling read of a loop reads from a redirected stdin, make sure no program in the loop body attempts to reads from stdin as well; they all share the same stdin.
Placing set -x near the top is likely to provide some insight.
I have a bash script that prints a line of text into a file, and then calls a second script that prints some more data into the same file. Lets call them script1.sh and script2.sh. The reason it's split into two scripts, is because I have different versions of script2.sh.
script1.sh:
rm -f output.txt
echo "some text here" > output.txt
source script2.sh
script2.sh:
./read_time >> output.txt
./run_program
./read_time >> output.txt
Variations on the three lines in script2.sh are repeated.
This seems to work most of the time, but every once in a while the file output.txt does not contain the line "some text here". At first I thought it was because I was calling script2.sh like this: ./script2.sh. But even using source the problem still occurs.
The problem is not reproducible, so even when I try to change something I don't know if it's actually fixed.
What could be causing this?
Edit:
The scripts are very simple. script1 is exactly as you see here, but with different file names. script 2 is what I posted, but then the same 3 lines repeated, and ./run_program can have different arguments. I did a grep for the output file, and for > but it doesn't show up anywhere unexpected.
The way these scripts are used is that script1 is created by a program (the only difference between the versions is the source script2.sh line. This script1.sh is then run on a different computer (linux on an FPGA actually) using ssh. Before that is done, the output file is also deleted using ssh. I don't know why, but I didn't write all of this. Also, I've checked the code running on the host. The only mention of the output file is when it is deleted using ssh, and when it is copied back to the host after the script1 is done.
Edit 2:
I finally managed to make the problem reproducible at a reasonable rate by stripping script2.sh of everything but a single line printing into the file. This also let me do the testing a bit faster. Once I had this I got the problem between 1 and 4 times for every 10 runs. Removing the command that was deleting the file over ssh before the script was run seems to have solved the problem. I will test it some more to be sure, but I think it's solved. Although I'm still not sure why it would be a problem. I thought that the ssh command would not exit before all the remove commands were executed.
It is hard to tell without seeing the real code. Most likely explanation is that you have a typo, > instead of >>, somewhere in one of the script2.sh files.
To verify this, set noclobber option with set -o noclobber. The shell will then terminate when trying to write to existing file with >.
Another possibility, is that the file is removed under certain rare conditions. Or it is damaged by some command which can have random access to it - look for commands using this file without >>. Or it is used by some command both as input and output which step on each other - look for the file used with <.
Lastly, you can have a racing condition with a command outputting to the file in background, started before that echo.
Can you grep all your scripts for 'output.txt'? What about scripts called inside read_time and run_program?
It looks like something in one of the script2.sh scripts must be either overwriting, truncating or doing a substitution on output.txt.
For example,there could be a '> output.txt' burried inside a conditional for a condition that rarely obtains. Just a guess, but it would explain why you don't always see it.
This is an interesting problem. Please post the solution when you find it!
I wrote the following code
var=0
cat $file | while read line do
var=$line
done
echo $var
Now as I understand it the pipe (|) will cause a sub shell to be created an therefore the variable var on line 1 will have the same value on the last line.
However this will solve it:
var=0
while read line do
var=$line
done < $file
echo $line
My question is why does the redirect not cause a subshell to be created, or if you like why does pipe cause one to be created?
Thanks
The cat command is a command which means it needs its own process and has its own STDIN and STDOUT. You're basically taking the STDOUT produced by the cat command and redirecting it into the process of the while loop.
When you use redirection, you're not using a separate process. Instead, you're merely redirecting the STDIN of the while loop from the console to the lines of the file.
Needless to say, the second way is more efficient. In the old Usenet days before all of you little whippersnappers got ahold of our Internet (_Hey you kids! Get off of my Internet!) and destroyed it with your fancy graphics and all them web page, some people use to give out the Useless Use of Cat award for people who contributed to the comp.unix.shell group and had a spurious cat command because the use of cat is almost never necessary and is usually more inefficient.
If you're using a cat in your code, you probably don't need it. The cat command comes from concatenate and is suppose to be used only to concatenate files together. For example, when we use to use SneakerNet on 800K floppies, we would have to split up long files with the Unix split command and then use cat to merge them back together.
A pipe is there to hook the stdout of one program to the stdin or another one. Two processes, possibly two shells. When you do redirection (> and <), all you're doing remapping stdin (or stdout) to a file. reading/writing a file can be done without another process or shell.
To process a bunch of data and get it ready to be inserted into our database, we generate a bunch of shell scripts. Each of them has about 15 lines, one for each table that the data is going. One a recent import batch, some of the import files failed going into one particular table. So, I have a bunch of shell scripts (about 600) where I need to comment out the first 7 lines, then rerun the file. There are about 6000 shell scripts in this folder, and nothing about a particular file can tell me if it needs the edit. I've got a list of which files that I pulled from the database output.
So how do I write a bash script (or anything else that would work better) to take this list of file names and for each of them, comment out the first 7 lines, and run the script?
EDIT:
#!/usr/bin/env sh
cmd1
cmd2
cmd3
cmd4
cmd5
cmd6
cmd7
cmd8
Not sure how readable that is. Basically, the first 7 lines (not counting the first line) need to have a # added to the beginning of them. Note: the files have been edited to make each line shorter and partially cut off copying out of VIM. But in the main part of each file, there is a line starting with echo, then a line starting with sqlldr
Using sed, you can specify a line number range in the file to be changed.
#!/bin/bash
while read line
do
# add a comment to beginning of lines 1 - 7 and rename the script
sed '3,9 s/^/#/' $line > $line.new
exec $line.new
done < "filelist.txt"
You may wish to test this before running it on all of those scripts...
EDIT: changed the lines numbers to reflect comments.
Roughly speaking:
#!/bin/sh
for file in "$#"
do
out=/tmp/$file.$$
sed '2,8s/^/#/' < $file > $out
$SHELL $out
rm -f $out
done
Assuming you don't care about checking for race conditions etc.
ex seems made for what you want to do.
For instance, for editing one file, with a here document:
#!/bin/sh
ex test.txt << END
1,12s/^/#/
wq
END
That'll comment out the first 12 lines in "test.txt". For your example you could try "$FILE" or similar (including quotes!).
Then run them the usual way, i.e. ./"$FILE"
edit: $SHELL "$FILE" is probably a better approach to run them (from one of the above commenters).
Ultimately you're going to want to use the linux command sed. Whatever logic you need to place in the script, you know. But your script will ultimately call sed. http://lowfatlinux.com/linux-sed.html