I've got a text file composed of different commands:
set +e; false; echo $?
(false; echo foo); echo \ $?
for i in .; do (false; echo foo); echo \ 1:$?; done; echo \ 2:$?
and so on.
I want to test these commands line-by-line, as a whole, in a bash child process using set -e, then check the error code from the parent bash process to see if the child exited with an error. I'm stuck on how to process the simpler commands (e.g. the first one listed above). Here's one of my recent attempts:
mapfile file < "$dir"/errorData
bash -c "${file[0]}"
echo $?
returns
bash: set +e: command not found
bash: false: command not found
bash: echo 127: command not found
I've tried several variants without luck. Using bash -c \'"${file[1]}"\' leads to one error message about command not found, with the above three strings now combined into one long string. Of course, simply wrapping the file commands in strong quotes and feeding them directly to bash in an interactive shell works fine.
Edit: I'm using bash 4.4.12(1)
For anyone who finds that the shell is parsing their text in ways that seem to violate the reference manual, make sure to check any copied input data for invisible control characters. E.g. I found a non-breaking space in place of what looked like simple whitespace, in the input file with commands to be executed. This is especially common when copying from web pages to an editor or the command line. bash uses whitespace, tabs, and newlines to tokenize, not the non-breaking space (U+0020 in the Unicode standard). A hex editor, e.g. hexfiend, can help troubleshoot similar issues.
Related
I am new to bash scripting and i'm just trying out new things and getting to grips with it.
Basically I am writing a small script to store the content of a file in a variable and then use that variable in an if statement.
Through step by step i have figured out the ways to store variables and then store content of files as variables. I am now working on if statements.
My test if statement if very VERY basic. I just wanted to grasp the syntax before moving onto more complicated if statement for my program.
My if statement is:
if [ "test" = "test" ]
then
echo "This is the same"
fi
Simple right? however when i run the script i am getting the error:
syntax error near unexpected token `fi'
I have tried a number of things from this site as well as others but i am still getting this error and I am unsure what is wrong. Could it be an issue on my computer stopping the script from running?
Edit for ful code. Note i also deleted all the commented out code and just used the if statement, still getting same error.
#!/bin/bash
#this stores a simple variable with the content of the file testy1.txt
#DATA=$(<testy1.txt)
#This echos out the stored variable
#echo $DATA
#simple if statement
if [ "test" = "test" ]
then
echo "has value"
fi
If a script looks ok (you've balanced all your quotes and parentheses, as well as backticks), but issues an error message, this may be caused by funny characters, i.e. characters that are not displayed, such as carriage returns, vertical tabs and others. To diagnose these, examine your script with
od -c script.sh
and look for \r, \v or other unexpected characters. You can get rid of \r for example with the dos2unix script.sh command.
BTW, what operating system and editor are you using?
To complement Jens's helpful answer, which explains the symptoms well and offers a utility-based solution (dos2unix). Sometimes installing a third-party utility is undesired, so here's a solution based on standard utility tr:
tr -d '\r' < script > script.tmp && mv script.tmp script
This removes all \r (CR) characters from the input, saves the output to a temporary file, and then replaces the original file.
While this blindly removes \r instances even if they're not part of \r\n (CRLF) pairs, it's usually safe to assume that \r instances indeed only occur as part of such pairs.
Solutions with other standard utilities (awk, sed) are possible too - see this answer of mine.
If your sed implementation offers the -i option for in-place updating, it may be the simpler choice.
To diagnose the problem I suggest using cat -v script, as its output is easy to parse visually: if you see ^M (which represents \r) at the end of the output lines, you know you're dealing with a file with Window line endings.
Why Your Script Failed So Obscurely
Normally, a shell script that mistakenly has Windows-style CRLF line endings, \r\n, (rather than the required Unix-style LF-only endings, \n) and starts with shebang line #!/bin/bash fails in a manner that does indicate the cause of the problem:
/bin/bash^M: bad interpreter
as a quick SO search can attest. The ^M indicates that the CR was considered part of the interpreter path, which obviously fails.
(If, by contrast, the script's shebang line is env-based, such as #!/usr/bin/env bash, the error message differs, but still points to the cause: env: bash\r: No such file or directory)
The reason you did not see this problem is that you're running in the Windows Unix-emulation environment Cygwin, which - unlike on Unix - allows a shebang line to end in CRLF (presumably to also support invoking other interpreters on Windows that do expect CRLF endings).
The CRLF problem therefore didn't surface until later in your script, and the fact that you had no empty lines after the shebang line further obfuscated the problem:
An empty CRLF-terminated line would cause Bash (4.x) to complain as follows: "bash: line <n>: $'\r': command not found, because Bash tries to execute the CR as a command (since it doesn't recognize it as part of the line ending).
The comment lines directly following the shebang lines are unproblematic, because a comment line ending in CR is still syntactically valid.
Finally, the if statement broke the command, in an obscure manner:
If your file were to end with a line break, as is usually the case, you would have gotten syntax error: unexpected end of file:
The line-ending then and if tokens are seen as then\r and if\r (i.e., the CR is appended) by Bash, and are therefore not recognized as keywords. Bash therefore never sees the end of the if compound command, and complains about encountering the end of the file before seeing the if statement completed.
Since your file did not, you got syntax error near unexpected token 'fi':
The final fi, due to not being followed by a CR, is recognized as a keyword by Bash, whereas the preceding then wasn't (as explained). In this case, Bash therefore saw keyword fi before ever seeing then, and complained about this out-of-place occurrence of fi.
Optional Background Information
Shell scripts that look OK but break due to characters that are either invisible or only look the same as the required characters are a common problem that usually has one of the following causes:
Problem A: The file has Windows-style CRLF (\r\n) line endings rather than Unix-style LF-only (\n) line endings - which is the case here.
Copying a file from a Windows machine or using an editor that saves files with CRLF sequences are among the possible causes.
Problem B: The file has non-ASCII Unicode whitespace and punctuation that looks like regular whitespace, but is technically distinct.
A common cause is source code copied from websites that use non-ASCII whitespace and punctuation for formatting code for display purposes;
an example is use of the no-break space Unicode character (U+00A0; UTF-8 encoding 0xc2 0xa0), which is visually indistinguishable from a normal (ASCII) space (U+0020).
Diagnosing the Problem
The following cat command visualizes:
all normally invisible ASCII control characters, such as \r as ^M.
all non-ASCII characters (assuming the now prevalent UTF-8 encoding), such as the non-break space Unicode char. as M-BM- .
^M is an example of caret notation, which is not obvious, especially with multi-byte characters, but, beyond ^M, it's usually not necessary to know exactly what the notation stands for - you just need to note if the ^<letter> sequences are present at all (problem A), or are present in unexpected places (problem B).
The last point is important: non-ASCII characters can be a legitimate part of source code, such as in string literals and comments. They're only a problem if they're used in place of ASCII punctuation.
LC_ALL=C cat -v script
Note: If you're using GNU utilities, the LC_ALL=C prefix is optional.
Solutions to Problem A: translating line endings from CRLF to LF-only
For solutions based on standard or usually-available-by-default utilities (tr, awk, sed, perl), see this answer of mine.
A more robust and convenient option is the widely used dos2unix utility, if it is already installed (typically, it is not), or installing it is an option.
How you install it depends on your platform; e.g.:
on Ubuntu: sudo apt-get install dos2unix
on macOs, with Homebrew installed, brew install dos2unix
dos2unix script would convert the line endings to LF and update file script in place.
Note that dos2unix also offers additional features, such as changing the character encoding of a file.
Solutions to Problem B: translating Unicode punctuation to ASCII punctuation
Note: By punctuation I mean both whitespace and characters such as -
The challenge in this case is that only Unicode punctuation should be targeted, whereas other non-ASCII characters should be left alone; thus, use of character-transcoding utilities such as iconv is not an option.
nws is a utility (that I wrote) that offers a Unicode-punctuation-to-ASCII-punctuation translation mode while leaving non-punctuation Unicode chars. alone; e.g.:
nws -i --ascii script # translate Unicode punct. to ASCII, update file 'script' in place
Installation:
If you have Node.js installed, install it by simply running [sudo] npm install -g nws-cli, which will place nws in your path.
Otherwise: See the manual installation instructions.
nws has several other functions focused on whitespace handling, including CRLF-to-LF and vice-versa translations (--lf, --crlf).
syntax error near unexpected token `fi'
means that if statement not opened and closed correctly, you need to check from the beginning that every if, for or while statement was opened and closed correctly.
Don't forget to add in the beginning of script :
#!/bin/bash
Here, "$test" should be a variable that stores the content of that file.
if [ "$test" = "test" ]; then
echo "This is the same"
fi
Consider the following script:
#!/bin/bash
CMD="echo hello world > /tmp/hello.out"
${CMD}
The output for this is:
hello world > /tmp/hello.out
How can I modify CMD so that the output gets redirected to hello.out?
For my use case, it is not feasible to either do this:
${CMD} > /tmp/hello.out
or to add this at the top of the script:
exec > /tmp/hello.out
No, there is no way to make a redirection happen from a variable.
Why?
The first thing the shell does with a command line is:
Each line that the shell reads from the standard input or a script is called a pipeline; it contains one or more commands separated by zero or
more pipe characters (|). For each pipeline it reads, the shell breaks it up into commands, sets up the I/O for the pipeline, then does the following for each command (Figure 7-1):
From: Learning the bash Shell Unix Shell Programming . Chapter Preview / Figure . Pdf
That means that even before starting with the first word of a command line, the redirections are set up.
The "Parameter Expansion" happens quite a lot latter (in step 6 of the Figure).
There is no way to set up redirections after a variable is expanded.
Unless ...
The "command line is reprocessed" using eval.
eval "$CMD"
But this comes with a lot of danger.
The command line is changed by the first processing in the 12 steps detailed in the book (quotes are removed, variables expanded, words split, etc.).
It is usually quite difficult to estimate all the changes and consequences before the line is actually processed.
And then, it is processed again.
You can use eval to instruct the shell to reinterpret the variable content as a shell command:
eval $CMD
CMake's execute_process command seems to only let you, well, execute a process - not an arbitrary line you could feed a command shell. The thing is, I want to use pipes, file descriptor redirection, etc. - and that does not seem to be possible. The alternative would be very painful for me (I think)...
What should I do?
PS - CMake 2.8 and 3.x answer(s) are interesting.
You can execute any shell script, using your shell's support for taking in a script within a string argument.
Example:
execute_process(
COMMAND bash "-c" "echo -n hello | sed 's/hello/world/;'"
OUTPUT_VARIABLE FOO
)
will result in FOO containing world.
Of course, you would need to escape quotes and backslashes with care. Also remember that running bash would only work on platforms which have bash - i.e. it won't work on Windows.
execute_process command seems to only let you, well, execute a process - not an arbitrary line you could feed a command shell.
Yes, exactly this is written in documentation for that command:
All arguments are passed VERBATIM to the child process. No intermediate shell is used, so shell operators such as > are treated as normal arguments.
I want to use pipes
Different COMMAND within same execute_process invocation are actually piped:
Runs the given sequence of one or more commands with the standard output of each process piped to the standard input of the next.
file descriptor redirection, etc. - and that does not seem to be possible.
For complex things just prepare separate shell script and run it using execute_process. You can pass variables from CMake to this script using its parameters, or with prelimiary configure_file.
I needed to pipe two commands one after the other and actually learned that each COMMAND of the execute_process is piped already. So at least that much is resolved by simply adding commands one after the other:
execute_process(
COMMAND echo "Hello"
COMMAND sed -e 's/H/h/'
OUTPUT_VARIABLE GREETINGS
OUTPUT_STRIP_TRAILING_WHITESPACE)
Now the variable GREETINGS is set to hello.
If you indeed need a lot of file redirection (as you stated), you probably want to write an external script and then execute that script from CMakeLists.txt. It's really difficult to get all the escaping right in CMake.
If you can simplify your scripts to one command generating a file, then another handling that file, etc. then you can always use the INPUT_FILE and OUTPUT_FILE options. Or pass a filename to your command for the input.
It's often much cleaner to handle one file at a time. Although I understand that some commands may need multiple sources and destinations.
I have a number of files (more than a hundred) that I want to process using Vim. A sample of the files’ contents is as follows:
xyz.csv /home/user/mydocs/abc.txt
/home/user/waves/wav.wav , user_wav.wav
I want this to be replaced by:
xyz.csv /var/lib/mydir/abc.txt
/var/sounds/wav.wav , wav.wav
In each of the files, the changes I need to make are the same. My questions are:
Can I use Vim search and replace functionality by calling it from within a Bash script?
If so, how do I go about it?
P.S. I have searched StackOverflow for similar questions and found some answers using ex scripts, etc. I want to know how I can call an ex script from within a bash script.
While vim is quite powerful, this is not something I would normally use vim for. It can be done using a combination of common command line utilities instead.
I've assumed that the blank line in your example above is actually blank and does not contain spaces or any other whitespace characters. You can use the following to do what you want.
sed -e "s,/home/user/mydocs,/var/lib/mydir," -e "s,/home/user/waves,/var/sounds," -e "/^$/d" file1
You can use that command together with find and a for loop to do this for a bunch of files:
for file in `find . -maxdepth 1 -type f`
do
sed -e "s,/home/user/mydocs,/var/lib/mydir," -e "s,/home/user/waves,/var/sounds," -e "/^$/d" $file
done
In the for loop, the find command above limits the output to all files in the current directory (including dot files), assigning each line from the output of find to the file variable and then running the sed command posted earlier to transform the file the way you want it to be transformed.
This is how you'd invoke an ed script from bash:
ed filename <<END
/^$/d
%s|/home/user/mydocs|/var/lib/mydir|
%s|/home/user/waves|/var/sounds|
%s|, user_|, |
w
q
END
To answer with vim, you can do
vim -e 'bufdo!%s:\(xyz.csv \)/home/user/mydocs/\(abc.txt\n\)\n.*:\1/var/lib/mydir/\2/var/sounds/wav.wav , wav.wav:' -e 'xa' FILES
Note, I had assumed, that the second line is statically replaced, as it had looked like in the question.
If you don't like writing long lines in your script, you can create a file like:
s/FOO/BAR/
" several replacement and other commands
w " write the file
bd " if you want to
Then do:
vim -e "buffdo!source /your_scriptfile" -e "x" FILES
HTH
If all the editing consists in a series of substitutions, the most
idiomatic way of accomplishing it using Vim would be the following.
Open all the target files at once:
vim *.txt
Run the substitution commands on the loaded files:
:argdo %s#/home/user/mydocs#/var/lib/mydir#
:argdo %s#/home/user/waves#/var/sounds#
:argdo %s#, \zsuser_##
...
If changes are correctly made, save the files:
:wall
If the editing you want to automate could not be expressed only
in substitutions, record a macro and run it via the :normal
command:
:argdo norm!#z
(Here z is the name of the macro to be run.)
Lastly, if the editing should be performed from time to time and
needs to be stored in a script, try using the approach described
in the answer to a similar question.
Answer
While most vim users would be aware of the % motion command for executing inclusive commands on the whole document in the current buffer. Most modern versions of vim (ie 6.x+) support actions on regex searches for exclusive actions like so:
:/regex/substitute/match/replace/ # as vim command line
+"/reges/s/match/replace" # as bash cli parameter
This breaks down into vim doing the following,
search for regex and put the cursor at start of found point
call internal substitute function (see :help s or :help substitute) [ could be other commands ]
match string with regex for substitution
replace with new string value
Effectively it operates the same as the :global command.
Notes
Command after regex search can be any command, including '!"shell command"' filter commands.
Reference Help
:help global
:help substitute
:help filter
Okay, I am sure this is simple but it is driving me nuts. I recently went to work on a program where I have had to step back in time a bit and use Redhat 9. When I'm typing on the command line from a standard xterm running KornShell (ksh), and I reach the end of the line the screen slides to the right (cutting off the left side of my command) instead of wrapping the text around to a new line. This makes things difficult for me because I can't easily copy and paste from the previous command straight from the command line. I have to look at the history and paste the command from there. In case you are wondering, I do a lot of command-line awk scripts that cause the line to get quite long.
Is there a way to force the command line to wrap instead of shifting visibility to the right side of the command I am typing?
I have poured through man page options with no luck.
I'm running:
XFree86 4.2.99.903(174)
KSH 5.2.14.
Thanks.
Did you do man ksh?
You want to do a set -o multiline.
Excerpt from man ksh:
multiline:
The built-in editors will use multiple lines on the screen for
lines that are longer than the width of the screen. This may not
work for all terminals.
eval $(resize) should do it.
If possible, try to break the command down to multiple lines by adding \
ie:
$ mycommand -a foo \
-f bar \
-c dif
The simple answer is:
$ set -o multiline
ksh earlier than 5.12, like the ksh shipped with NetBSD 6.1, doesn't have this option. You will have to turn off current Interactive Input Line Editing mode, which is usually emacs:
$ set +o emacs
This turns off a lot of featuers altogether, like tab-completion or the use of 'Up-arrow' key to roll back the previous command.
If you decide to get used to emacs mode somehow, remember ^a goes to the begining of the line ("Home" key won't workk) and ^e goes to the end.
I don't know of a way of forcing the shell to wrap, but I would ask why you'd be writing lines that long. With awk scripts, I simply wrap the script in single quotes, and then break the lines where I want. It only gets tricky if you need single quotes in the script -- and diabolical if you need both single and double quotes. Actually, the rule is simple enough: use single quotes to wrap the whole script, and when you want a single quote in the script, write '\''. The first quote terminates the previous single-quoted string; the backslash-single quote yields a single quote; and the last single quote starts a new single quoted string. It really gets hairy if you need to escape those characters for an eval or something similar.
The other question is - why not launch into an editor. Since I'm a die-hard vim nutcase (ok - I've been using vi for over 20 years, so it is easier for me than the alternatives), I have Korn shell set to vi mode (set -o vi), and can do escape-v to launch the editor on whatever I've typed.
This is kind of a pragmatic answer, but when that's an issue for me I usually do something like:
strings ~/.history | grep COMMAND
or
strings ~/.history | tail
(The history file has a little bit of binary data in it, hence 'strings')