This question already has answers here:
While loop stops reading after the first line in Bash
(5 answers)
Closed 6 years ago.
I am writing a Bash file to execute two PhantomJS tasks.
I have two tasks written in external JS files: task1.js & task2.js.
Here's my Bash script so far:
#!/bin/bash
url=$1
cd $(cd $(dirname ${BASH_SOURCE}); pwd -P)
dir=../temp
mkdir -p $dir
file=$dir/file.txt
phantomjs "taks1.js" $url > $file
while IFS="" read -r line || [[ -n $line ]]; do
dir=../build
file=$dir/$line.html
mkdir -p $(dirname $file)
phantomjs "task2.js" $url $line > $file
done < $file
For some unknown reason task2 is being run only once, then the script stops.
If I remove the PhantomJS command, the while loop runs normally until all lines are read from the file.
Maybe someone knows why is that?
Cheers.
Your loop is reading contents from stdin. If any other program you run consumes stdin, the loop will terminate.
Either fix any program that may be consuming stdin to read from /dev/null, or use a different FD for the loop.
The first approach looks like this:
phantomjs "task2.js" "$url" "$line" >"$file" </dev/null
The second looks like this (note the 3< on establishing the redirection, and the <&3 to read from that file descriptor):
while IFS="" read -r line <&3 || [[ -n $line ]]; do
dir=../build
file=$dir/$line.html
mkdir -p "$(dirname "$file")"
phantomjs "task2.js" "$url" "$line" >"$file"
done 3< $file
By the way, consider taking file out of the loop altogether, by having the loop read directly from the first phantomjs program's output:
while IFS="" read -r line <&3 || [[ -n $line ]]; do
dir=../build
file=$dir/$line.html
mkdir -p "$(dirname "$file")"
phantomjs "task2.js" "$url" "$line" >"$file"
done 3< <(phantomjs "task1.js" "$url")
Related
This question already has answers here:
While loop stops reading after the first line in Bash
(5 answers)
Closed 6 years ago.
I use this bash-code to upload files to a remote server, for normal files this works fine:
for i in `find devel/ -newer $UPLOAD_FILE`
do
echo "Upload:" $i
if [ -d $i ]
then
echo "Creating directory" $i
ssh $USER#$SERVER "cd ${REMOTE_PATH}; mkdir -p $i"
continue
fi
if scp -Cp $i $USER#$SERVER:$REMOTE_PATH/$i
then
echo "$i OK"
else
echo "$i NOK"
rm ${UPLOAD_FILE}_tmp
fi
done
The only problem is that for files with a space in the name, the for-loop fails, so I replaced the first line like this:
find devel/ -newer $UPLOAD_FILE | while read i
do
echo "Upload:" $i
if [ -d $i ]
then
echo "Creating directory" $i
ssh $USER#$SERVER "cd ${REMOTE_PATH}; mkdir -p $i"
continue
fi
if scp -Cp $i $USER#$SERVER:$REMOTE_PATH/$i
then
echo "$i OK"
else
echo "$i NOK"
rm ${UPLOAD_FILE}_tmp
fi
done
For some strange reason, the ssh-command breaks out of the while-loop, therefore the first missing directory is created fine, but all subsequent missing files/directories are ignored.
I guess this has something to do with ssh writing something to stdout which confuses the "read" command. Commenting out the ssh-command makes the loop work as it should.
Does anybody know why this happens and how one can prevent ssh from breaking the while-loop?
The problem is that ssh reads from standard input, therefore it eats all your remaining lines. You can just connect its standard input to nowhere:
ssh $USER#$SERVER "cd ${REMOTE_PATH}; mkdir -p $i" < /dev/null
You can also use ssh -n instead of the redirection.
Another approach is to loop over a FD other than stdin:
while IFS= read -u 3 -r -d '' filename; do
if [[ -d $filename ]]; then
printf -v cmd_str 'cd %q; mkdir -p %q' "$REMOTE_PATH" "$filename"
ssh "$USER#$SERVER" "$cmd_str"
else
printf -v remote_path_str '%q#%q:%q/%q' "$USER" "$SERVER" "$REMOTE_PATH" "$filename"
scp -Cp "$filename" "$remote_path_str"
fi
done 3< <(find devel/ -newer "$UPLOAD_FILE" -print0)
The -u 3 and 3< operators are critical here, using FD 3 rather than the default FD 0 (stdin).
The approach given here -- using -print0, a cleared IFS value, and the like -- is also less buggy than the original code and the existing answer, which can't handle interesting filenames correctly. (Glenn Jackman's answer is close, but even that can't deal with filenames with newlines or filenames with trailing whitespace).
The use of printf %q is critical to generate commands which can't be used to attack the remote machine. Consider what would happen with a file named devel/$(rm -rf /)/hello with code which didn't have this paranoia.
This question already has answers here:
Read user input inside a loop
(6 answers)
How to read from user within while-loop read line?
(3 answers)
Closed 5 years ago.
For my purpose, I need to execute a shell command, achieve the output, and for each line ask user for prompt.
The problem is that on the read prompt, stdin buffer isn't empty
this is my code:
#!/bin/sh
git branch -a | sed 's/remotes\/origin\///g'
echo "############################"
git branch -a | sed 's/remotes\/origin\///g' | while read line
do
if [[ "$line" != *develop* ]] \
&& [[ "$line" != *master ]] \
&& [[ "$line" != *release/* ]] \
&& [[ "$line" != *hotfix* ]]
then
read -r -p "Do you want to delete branch $line <y/N>?" prompt
echo $prompt
fi
done
The line:
read -r -p "Do you want to delete branch $line <y/N>?" prompt
does not even display to video, and prompt variable show the result of line variable above.
How can I solve this problem?
Use a FD other than 0 (stdin), to leave the original stdin free for input from the user:
#!/usr/bin/env bash
# ^^^^- NOT /bin/sh; also, do not run with "sh scriptname"
while read -r line <&3; do
line=${line#remotes/origin/} # trim remotes/origin/ w/o needing sed
case $line in
*develop*|*master|*release/*|*hotfix*) continue ;;
*) read -r -p "Do you want to delete branch $line <y/N>?" prompt
echo "$prompt" ;;
esac
done 3< <(git branch -a)
Here, we're using FD 3 for output from git, such that FD 0 is still stdin, available to read from the user; and then redirecting <&3 on the explicit read where we want content from git.
I have a bash script that I want to expand to support piping json into.
Example:
echo '{}' | myscript store
So, I tried the following:
local value="$1"
if [[ -z "$value" ]]; then
while read -r piped; do
value=$piped
done;
fi
Which works in a simple case above, but doing:
cat input.json | myscript store
Only get's the last line of the file input.json, it does not handle every line.
How can I support all cases of piping?
The following works:
if [[ -z "$value" && ! -t 0 ]]; then
while read -r piped; do
value+=$piped
done;
fi
The trick was using += and also checking ! -t 0 which checks if we are piping.
If you want to behave like cat, why not use it?
#! /bin/bash
value="$( cat "$#" )"
This question already has answers here:
While loop stops reading after the first line in Bash
(5 answers)
Closed 6 years ago.
I use this bash-code to upload files to a remote server, for normal files this works fine:
for i in `find devel/ -newer $UPLOAD_FILE`
do
echo "Upload:" $i
if [ -d $i ]
then
echo "Creating directory" $i
ssh $USER#$SERVER "cd ${REMOTE_PATH}; mkdir -p $i"
continue
fi
if scp -Cp $i $USER#$SERVER:$REMOTE_PATH/$i
then
echo "$i OK"
else
echo "$i NOK"
rm ${UPLOAD_FILE}_tmp
fi
done
The only problem is that for files with a space in the name, the for-loop fails, so I replaced the first line like this:
find devel/ -newer $UPLOAD_FILE | while read i
do
echo "Upload:" $i
if [ -d $i ]
then
echo "Creating directory" $i
ssh $USER#$SERVER "cd ${REMOTE_PATH}; mkdir -p $i"
continue
fi
if scp -Cp $i $USER#$SERVER:$REMOTE_PATH/$i
then
echo "$i OK"
else
echo "$i NOK"
rm ${UPLOAD_FILE}_tmp
fi
done
For some strange reason, the ssh-command breaks out of the while-loop, therefore the first missing directory is created fine, but all subsequent missing files/directories are ignored.
I guess this has something to do with ssh writing something to stdout which confuses the "read" command. Commenting out the ssh-command makes the loop work as it should.
Does anybody know why this happens and how one can prevent ssh from breaking the while-loop?
The problem is that ssh reads from standard input, therefore it eats all your remaining lines. You can just connect its standard input to nowhere:
ssh $USER#$SERVER "cd ${REMOTE_PATH}; mkdir -p $i" < /dev/null
You can also use ssh -n instead of the redirection.
Another approach is to loop over a FD other than stdin:
while IFS= read -u 3 -r -d '' filename; do
if [[ -d $filename ]]; then
printf -v cmd_str 'cd %q; mkdir -p %q' "$REMOTE_PATH" "$filename"
ssh "$USER#$SERVER" "$cmd_str"
else
printf -v remote_path_str '%q#%q:%q/%q' "$USER" "$SERVER" "$REMOTE_PATH" "$filename"
scp -Cp "$filename" "$remote_path_str"
fi
done 3< <(find devel/ -newer "$UPLOAD_FILE" -print0)
The -u 3 and 3< operators are critical here, using FD 3 rather than the default FD 0 (stdin).
The approach given here -- using -print0, a cleared IFS value, and the like -- is also less buggy than the original code and the existing answer, which can't handle interesting filenames correctly. (Glenn Jackman's answer is close, but even that can't deal with filenames with newlines or filenames with trailing whitespace).
The use of printf %q is critical to generate commands which can't be used to attack the remote machine. Consider what would happen with a file named devel/$(rm -rf /)/hello with code which didn't have this paranoia.
This question already has answers here:
While loop stops reading after the first line in Bash
(5 answers)
Closed 6 years ago.
I use this bash-code to upload files to a remote server, for normal files this works fine:
for i in `find devel/ -newer $UPLOAD_FILE`
do
echo "Upload:" $i
if [ -d $i ]
then
echo "Creating directory" $i
ssh $USER#$SERVER "cd ${REMOTE_PATH}; mkdir -p $i"
continue
fi
if scp -Cp $i $USER#$SERVER:$REMOTE_PATH/$i
then
echo "$i OK"
else
echo "$i NOK"
rm ${UPLOAD_FILE}_tmp
fi
done
The only problem is that for files with a space in the name, the for-loop fails, so I replaced the first line like this:
find devel/ -newer $UPLOAD_FILE | while read i
do
echo "Upload:" $i
if [ -d $i ]
then
echo "Creating directory" $i
ssh $USER#$SERVER "cd ${REMOTE_PATH}; mkdir -p $i"
continue
fi
if scp -Cp $i $USER#$SERVER:$REMOTE_PATH/$i
then
echo "$i OK"
else
echo "$i NOK"
rm ${UPLOAD_FILE}_tmp
fi
done
For some strange reason, the ssh-command breaks out of the while-loop, therefore the first missing directory is created fine, but all subsequent missing files/directories are ignored.
I guess this has something to do with ssh writing something to stdout which confuses the "read" command. Commenting out the ssh-command makes the loop work as it should.
Does anybody know why this happens and how one can prevent ssh from breaking the while-loop?
The problem is that ssh reads from standard input, therefore it eats all your remaining lines. You can just connect its standard input to nowhere:
ssh $USER#$SERVER "cd ${REMOTE_PATH}; mkdir -p $i" < /dev/null
You can also use ssh -n instead of the redirection.
Another approach is to loop over a FD other than stdin:
while IFS= read -u 3 -r -d '' filename; do
if [[ -d $filename ]]; then
printf -v cmd_str 'cd %q; mkdir -p %q' "$REMOTE_PATH" "$filename"
ssh "$USER#$SERVER" "$cmd_str"
else
printf -v remote_path_str '%q#%q:%q/%q' "$USER" "$SERVER" "$REMOTE_PATH" "$filename"
scp -Cp "$filename" "$remote_path_str"
fi
done 3< <(find devel/ -newer "$UPLOAD_FILE" -print0)
The -u 3 and 3< operators are critical here, using FD 3 rather than the default FD 0 (stdin).
The approach given here -- using -print0, a cleared IFS value, and the like -- is also less buggy than the original code and the existing answer, which can't handle interesting filenames correctly. (Glenn Jackman's answer is close, but even that can't deal with filenames with newlines or filenames with trailing whitespace).
The use of printf %q is critical to generate commands which can't be used to attack the remote machine. Consider what would happen with a file named devel/$(rm -rf /)/hello with code which didn't have this paranoia.