I'm trying to debug an larger osascript 'do shell script' program and have narrowed the problem down to a much shorter repro case that I don't understand at all. Here's the repro case:
osascript -e 'do shell script "echo 123; echo 45; echo 6"'
Gives me the output:
653
Can anybody explain what's going on? It's almost like 'do shell script' does not properly handle line endings, or tries to print everything on one line.
Applescript replaces \n characters with \r characters when reading the output of a shell script. You can tell this is happening by running the output through od -c:
$ osascript -e 'do shell script "echo 123; echo 45; echo 6"' | od -c
0000000 1 2 3 \r 4 5 \r 6 \n
0000011
To turn this off, use the without altering line endings parameter.
osascript -e 'do shell script "echo 123; echo 45; echo 6" without altering line endings'
See Technical Note TN2065 for more.
Related
Summary
Using read loop that runs a windows executable in WSL shell script causes it to exit the loop after the first iteration.
Details
I've been quite baffled by what appears to be an interoperability issue with running windows executables from a shell script in WSL2. The following while loop should print 3 lines but it will only print "line 1". It has been tested on Ubuntu 20.04 in dash, bash, and zsh.
while read -r line; do
powershell.exe /C "echo \"${line}\""
done << EOF
line 1
line 2
line 3
EOF
This issue also occurs when reading lines from a file instead of a heredoc even if that file has windows line endings. Note that if powershell were changed to /bin/bash or any other native executable this would print 3 lines. Also powershell could be replaced with any windows executable cmd.exe, explorer.exe, etc and it would still only run the first iteration. This appears to be a problem with read specifically since this loop will work fine.
for line in "line 1" "line 2" "line 3"
do
powershell.exe /C "echo \"${line}\""
done
Work-around
Thanks to this post I have discovered a work around is to pipe through a dummy command: echo "" | cmd.exe /C "echo \"${line}\"". A note about this fix is that only piping seems to work. Redirecting the output or running it through another layer of bash does not: /bin/bash -c "cmd.exe /C \"echo ${line}\"". I am partially posting this for improved visibility for anyone having this issue in the future, but I am still curious if anyone has any insight as to why this issue exists (perhaps due to line endings?). Thank you!
Short Answer:
A slightly-improved solution over echo "" | is to do a second-redirection from /dev/null. This avoids potential newline issues from the echo, but there are other solutions as well:
while read -r line; do
powershell.exe /C "echo \"${line}\"" < /dev/null
done << EOF
line 1
line 2
line 3
EOF
Explanation:
Well, you already had a solution, but what you really wanted was the explanation.
Jetchisel and MarkPlotnick are on the right track in the comments. This appears to be the same root cause (and solution) as in this question about ssh. To replicate your example with ssh (assuming a key in ssh-agent so that no password prompt is generated):
while read -r line; do
ssh hostname echo ${line}
done << EOF
line 1
line 2
line 3
EOF
You will see the same results as with PowerShell -- Only "line 1" displays.
In both cases, the first line goes to the read statement, but the subsequent lines are stdin which are consumed by powershell.exe (or ssh) itself.
You can see this "proven" in PowerShell through a slight modification to your script:
while read -r line; do
powershell.exe -c "echo \"--- ${line} ---\"; \$input"
done << EOF
line 1
line 2
line 3
EOF
Results in:
--- line 1 ---
line 2
line 3
The follow-up question is, IMHO, why bash doesn't have this issue. The answer is that PowerShell seems to always consume whatever stdin is available at the time of invocation and adds it to the $input magic variable. Bash, on the other hand, does not consume the additional stdin until explicitly asked:
while read -r line; do
bash -c "echo --- \"${line}\" ---; cat /dev/stdin"
done << EOF
line 1
line 2
line 3
EOF
Generates the same results as the previous PowerShell example:
--- line 1 ---
line 2
line 3
Ultimately, the main solution with PowerShell is to force a second indirection which is consumed before your desired input. echo "" | can do this, but be careful:
while read -r line; do
echo "" | powershell.exe -c "echo \"--- ${line} ---\"; \$input"
done << EOF
line 1
line 2
line 3
EOF
Results in:
--- line 1 ---
--- line 2 ---
--- line 3 ---
< /dev/null doesn't have this issue, but you could also handle it with echo -n "" | instead.
#!/bin/bash
if [ $# -ne 1 ]
then
echo "USAGE:vitest filename"
else
FILENAME=$1
exec vi $FILENAME <<EOF
i
Line 1.
Line 2.
^[
ZZ
EOF
fi
exit 0
I'm trying to input the Line 1. and Line 2. with Exec vi using the here doc, and commands.
When running the script it gives me the following:
Vim(?):Warning: Input is not from a terminal
Vim: Error reading input, exiting...
Press ENTER or type command to continueVim: Finished.
Vim: Error reading input, exiting...
Vim: Finished.
You want to start vi in ex mode, with a few minor changes to the script.
vi -e "$FILENAME" <<EOF
i
Line 1.
Line 2.
.
wq
EOF
exec is almost certainly unnecessary, especially since you have an exit command following vi. exec is used to replace the current script with the given command; it is not needed simply to execute a command.
A brief history of UNIX text editors:
ed was the original editor, designed to work with a teletype rather than a video terminal.
ex was an extended version of ed, designed to take advantage of a video terminal.
vi was an editor that provided ex with a full-screen visual mode, in contrast with the line-oriented interface employed by ed and ex.
As suggested, ed
ed file << END
1i
line1
line2
.
wq
END
The "dot" line means "end of input".
It can be written less legibly as a one-liner
printf "%s\n" 1i "line1" "line2" . wq | ed file
Use cat.
$ cat file1.txt file2.txt | tee file3.txt
Line 1
Line 2
aaaa
bbbb
cccc
Using sed
If I understand correctly, you want to add two lines to the beginning of a file. In that case, as per Cyrus' suggestion, run:
#!/bin/bash
if [ $# -ne 1 ]
then
echo "USAGE:vitest filename"
exit 1
fi
sed -i.bak '1 s/^/line1\nline2\n/' "$1"
Notes:
When a shell variable is used, it should be in double-quotes unless you want word splitting and pathname expansion to be performed. This is important for file names, for example, as it is now common for them to contain whitespace.
It is best practice to use lower or mixed case names for shell variables. The system uses upper case names for its variables and you don't want to overwrite one of them accidentally.
In the check for the argument, the if statement should include an exit to prevent the rest of the script from being run in the case that no argument was provided. In the above, we added exit 1 which sets the exit code to 1 to signal an error.
Using vi
Let's start with this test file:
$ cat File
some line
Now, let's run vi and see what is in File afterward:
$ vi -s <(echo $'iline1\nline2\n\eZZ') File
$ cat File
line1
line2
some line
The above requires bash or similar.
This question already has answers here:
Are shell scripts sensitive to encoding and line endings?
(14 answers)
Closed 6 months ago.
Any idea of what the problem could be?
My code is:
#!/bin/bash
while :
do
echo "Press [CTRL+C] to stop.."
sleep 1
done
Saved it as .sh and ran bash file.sh
CentOS 6 32-bit
What is the issue? First time EVER using BASH, need it for a simple infinite loop on something.
Run cat -v file.sh.
You most likely have a carriage return or no-break space in your file. cat -v will show them as ^M and M-BM- or M- respectively. It will similarly show any other strange characters you might have gotten into your file.
Remove the Windows line breaks with
tr -d '\r' < file.sh > fixedfile.sh
I was getting the same error on Cygwin; I did the following (one of them fixed it):
Converted TABS to SPACES
ran dos2unix on the .(ba)sh file
What is the error you're getting?
$ bash file.sh
test.sh: line 8: syntax error: unexpected end of file
If you get that error, you may have bad line endings. Unix uses <LF> at the end of the file while Windows uses <CR><LF>. That <CR> character gets interpreted as a character.
You can use od -a test.sh to see the invisible characters in the file.
$ od -a test.sh
0000000 # ! / b i n / b a s h cr nl # sp cr
0000020 nl w h i l e sp : cr nl d o cr nl sp sp
0000040 sp sp e c h o sp " P r e s s sp [ C
0000060 T R L + C ] sp t o sp s t o p " cr
0000100 nl sp sp sp sp s l e e p sp 1 cr nl d o
0000120 n e cr nl
0000124
The sp stands for space, the ht stands for tab, the cr stands for <CR> and the nl stands for <LF>. Note that all of the lines end with cr followed by a nl character.
You can also use cat -v test.sh if your cat command takes the -v parameter.
If you have dos2unix on your box, you can use that command to fix your file:
$ dos2unix test.sh
There's a way you can get this problem without having mixed newline problems (at least, in my shell, which is GNU bash v4.3.30):
#!/bin/bash
# foo.sh
function foo() {
echo "I am quoting a thing `$1' inside a function."
}
while [ "$input" != "y" ]; do
read -p "Hit `y' to continue: " -n 1 input
echo
done
foo "What could possibly go wrong?"
$ ./foo.sh
./foo.sh: line 11: syntax error near unexpected token `done'
./foo.sh: line 11: `done'
This is because bash expands backticks inside double-quoted strings (see the bash manual on quoting and command substitution), and before finding a matching backtick, will interpret any additional double quotes as part of the command substitution:
$ echo "Command substitution happens inside double-quoted strings: `ls`"
Command substitution happens inside double-quoted strings: foo.sh
$ echo "..even with double quotes: `grep -E "^foo|wrong" foo.sh`"
..even with double quotes: foo "What could possibly go wrong?"
You can get around this by escaping the backticks in your string with a backslash, or by using a single-quoted string.
I'm not really sure why this only gives the one error message, but I think it has to do with the function definition:
#!/bin/bash
# a.sh
function a() {
echo "Thing's `quoted'"
}
a
while true; do
echo "Other `quote'"
done
#!/bin/bash
# b.sh
echo "Thing's `quoted'"
while true; do
echo "Other `quote'"
done
$ ./a.sh
./a.sh: line 10: syntax error near unexpected token `done'
./a.sh: line 10: `done'
$ ./b.sh
./b.sh: command substitution: line 6: unexpected EOF while looking for matching `''
./b.sh: command substitution: line 9: syntax error: unexpected end of file
Thing's quote'
./b.sh: line 7: syntax error near unexpected token `done'
./b.sh: line 7: `done'
Might help someone else : I encountered the same kind of issues while I had done some "copy-paste" from a side Microsoft Word document, where I took notes, to my shell script(s).
Re-writing, manually, the exact same code in the script just solved this.
It was quite un-understandable at first, I think Word's hidden characters and/or formatting were the issue. Obvious but not see-able ... I lost about one hour on this (I'm no shell expert, as you might guess ...)
Sometimes this error happens because of unexpected CR characters in file, usually because the file was generated on a Windows system which uses CR line endings. You can fix this by running os2unix or tr, for example:
tr -d '\015' < yourscript.sh > newscript.sh
This removes any CR characters from the file.
Open new file named foobar
nano -w foobar
Input script
#!/bin/bash
while [ 0 = 0 ]; do
echo "Press [CTRL+C] to stop.."
sleep 1
done;
Exit and save
CTRL+X then Y and Enter
Set script executable and run
chmod +x foobar
./foobar
Had similar problems just now and these are two separate instances and solutions that worked for me:
Case 1. Basically, had a space after the last command within my newline-separated for-loop, eg. (imagining that | here represents the carat in a text editor showing where you are writing), this is what I saw when clicking around the end of the line of the last command in the loop:
for f in $pathToFiles
do
$stuff |
done
Notice the space before before the carat (so far as I know, this is something cat has no option do display visually (one way you could test is with something like od -bc yourscript.sh)). Changing the code to
for f in $pathToFiles
do
$stuff| <--- notice the carat shows no ending space before the newline
done
fixed the problem.
Case 2. Was using a pseudo try-catch block for the for-loop (see https://stackoverflow.com/a/22010339/8236733) like
{
for f in $pathToFiles
do
{ $stuff } || { echo "Failed to complete stuff"; exit 255; }
done
} || { echo "Failed to complete loop"; exit 255; }
and apparently bash did not like the nested {}s. Changing to
{
for f in $pathToFiles
do
$stuff
done
} || { echo "Failed to complete loop"; exit 255; }
fixed the problem in this case. If anyone can further explain either of these cases, please let me know more about them in the comments.
I had same problem, but solved.
I removed the following line in .bashrc
alias do="docker.exe" # this line caused the problem
I use WSL(windows subsystem for linux)
In my case, what was causing the problem was an if else statement. After re-writing the conditions, the error 'near done' got away.
Edit your code in any linux environment then you won't face this problem. If edit in windows
notepad any space take it as ^M.
I have exactly the same issue as above, and took me the whole day to discover that it doesn't like my newline approach. Instead I reused the same code with semi-colon approach instead.
For example my initial code using the newline (which threw the same error as yours):
Y=1
while test "$Y" -le "20"
do
echo "Number $Y"
Y=$[Y+1]
done
And using code with semicolon approach with worked wonder:
Y=1 ; while test "$Y" -le "20"; do echo "Number $Y"; Y=$[Y+1] ; done
I notice the same problem occurs for other commands as well using the newline approach, so I think I am gonna stick to using semicolon for my future code.
For me you had to have it do something between the do and done.
#!/bin/bash
echo "Endless Loop, to STOP press ctrl C or ctrl Z"
for ((; ;))
do
done
echo "Loop Ended"
gave me the error, but
#!/bin/bash
echo "Endless Loop, to STOP press ctrl C or ctrl Z"
for ((; ;))
do
sleep 1
done
echo "Loop Ended"
fixed it.
i read about \a escape character.
its description tells that it is basically used for alert or BEL
can anyone tell me that
is it creates some sound effect?
or please tell me any command that makes CPU to beep a sound
please tell me its necessary for me
Thanx in advanced
it echoes ASCI code 7:
$ echo -n $'\a' > file
$ od file
0000000 000007
0000001
I'm currently using the following to capture everything that goes to the terminal and throw it into a log file
exec 4<&1 5<&2 1>&2>&>(tee -a $LOG_FILE)
however, I don't want color escape codes/clutter going into the log file. so i have something like this that sorta works
exec 4<&1 5<&2 1>&2>&>(
while read -u 0; do
#to terminal
echo "$REPLY"
#to log file (color removed)
echo "$REPLY" | sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' >> $LOG_FILE
done
unset REPLY #tidy
)
except read waits for carriage return which isn't ideal for some portions of the script (e.g. echo -n "..." or printf without \n).
Follow-up to Jonathan Leffler's answer:
Given the example script test.sh:
#!/bin/bash
LOG_FILE="./test.log"
echo -n >$LOG_FILE
exec 4<&1 5<&2 1>&2>&>(tee -a >(sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' > $LOG_FILE))
##### ##### #####
# Main
echo "starting execution"
printf "\n\n"
echo "color test:"
echo -e "\033[0;31mhello \033[0;32mworld\033[0m!"
printf "\n\n"
echo -e "\033[0;36mEnvironment:\033[0m\n foo: cat\n bar: dog\n your wife: hot\n fix: A/C"
echo -n "Before we get started. Is the above information correct? "
read YES
echo -e "\n[READ] $YES" >> $LOG_FILE
YES=$(echo "$YES" | sed 's/^\s*//;s/\s*$//')
test ! "$(echo "$YES" | grep -iE '^y(es)?$')" && echo -e "\nExiting... :(" && exit
printf "\n\n"
#...some hundreds of lines of code later...
echo "Done!"
##### ##### #####
# End
exec 1<&4 4>&- 2<&5 5>&-
echo "Log File: $LOG_FILE"
The output to the terminal is as expected and there is no color escape codes/clutter in the log file as desired. However upon examining test.log, I do not see the [READ] ... (see line 21 of test.sh).
The log file [of my actual bash script] contains the line Log File: ... at the end of it even after closing the 4 and 5 fds. I was able to resolve the issue by putting a sleep 1 before the second exec - I assume there's a race condition or fd shenanigans to blame for it. Unfortunately for you guys, I am not able to reproduce this issue with test.sh but I'd be interested in any speculation anyone may have.
Consider using the pee program discussed in Is it possible to distribute stdin over parallel processes. It would allow you to send the log data through your sed script, while continuing to send the colours to the actual output.
One major advantage of this is that it would remove the 'execute sed once per line of log output'; that is really diabolical for performance (in terms of number of processes executed, if nothing else).
I know it's not a perfect solution, but cat -v will make non visible chars like \x1B to be converted into visible form like ^[[1;34m. The output will be messy, but it will be ascii text at least.
I use to do stuff like this by setting TERM=dumb before running my command. That pretty much removed any control characters except for tab, CR, and LF. I have no idea if this works for your situation, but it's worth a try. The problem is that you won't see color encodings on your terminal either since it's a dumb terminal.
You can also try either vis or cat (especially the -v parameter) and see if these do something for you. You'd simply put them in your pipeline like this:
exec 4<&1 5<&2 1>&2>&>(tee -a | cat -v | $LOG_FILE)
By the way, almost all terminal programs have an option to capture the input, and most clean it up for you. What platform are you on, and what type of terminal program are you using?
You could attempt to use the -n option for read. It reads in n characters instead of waiting for a new line. You could set it to one. This would increase the number of iteration the code runs, but it would not wait for newlines.
From the man:
-n NCHARS read returns after reading NCHARS characters rather than waiting for a complete line of input.
Note: I have not tested this
You can use ANSIFilter to strip or transform console output with ANSI escape sequences.
See http://www.andre-simon.de/zip/download.html#ansifilter
Might not screen -L or the script commands be viable options instead of this exec loop?