We write two similar scripts: one for bash (linux) and one for batch (dos/windows).
Even if the specific code is different we would like to visually compare (diff) both scripts and have the similar parts of code aligned side by side.
We use explicit comments with the same text to achieve this. But the beginning of the comments is different in both scripting (REM or :: in windows) and (# in linux).
Therefore there is a wrong alignment:
linux
windows
# first step
REM first step
foo.sh
foo.bat
# second step
REM second step
bar.sh
bar.bat
Is there a way to use a common character or sequence of characters to make the comments equal?
Is the use of : #; safe for both systems/scripts?
linux
windows
: #; first step
: #; first step
foo.sh
foo.bat
: #; second step
: #; second step
bar.sh
bar.bat
Are there any unwanted side effects?
: in bash is not exactly a comment. It is a void command.
A little bit like pass in some languages.
It helps, for example, to fill empty slots, if needed
if condition
then
:
else
doSomething
fi
So, you may use, somehow, as a sort of comment. That would works both in bash and batch (well, I know nothing of batch. But since you said that :: is a comment there). But beware that it is not exactly a comment. So there are some differences
For example
#!/bin/bash
echo one ||
:: foo
echo two
echo un ||
# bar
echo deux
Displays one, two and un but not deux.
Because echo one || prints one and then execute the following command only if it fails (which it doesn't). Here the following command is :: foo. Which is not executed (you wouldn't know, since it does nothing, but it is not executed). And the echo two is a brand new unrelated command that is executed.
Whereas, on the other hand, echo un || likewise prints un, and doesn't execute the next command, since echo un did not fail. But the next command here is echo deux. Because # bar doesn't count, since it is a comment.
And that is only one of the many examples one could probably find to show that : is not a comment.
But, well, if you use it being aware of that, I suppose you could use it to insert void comments in your bash code that would also be void in batch.
Edit:
I won't edit for each new example that comes to mind. But that one is pretty important
echo un # deux
echo one : two
prints
un
one : two
: is a command. So, as other commands, like ls not all occurrence of it is treated as so (no more than echo ls list the directory constant. ls is just a string here)
So, you can't use it as a replacement for inline comments. Only for full lines comments.
Related
I have recently started studying shell script and I'd like to be able to comment out a set of lines in a shell script. I mean like it is in case of C/Java :
/* comment1
comment2
comment3
*/`
How could I do that?
Use : ' to open and ' to close.
For example:
: '
This is a
very neat comment
in bash
'
Multiline comment in bash
: <<'END_COMMENT'
This is a heredoc (<<) redirected to a NOP command (:).
The single quotes around END_COMMENT are important,
because it disables variable resolving and command resolving
within these lines. Without the single-quotes around END_COMMENT,
the following two $() `` commands would get executed:
$(gibberish command)
`rm -fr mydir`
comment1
comment2
comment3
END_COMMENT
Note: I updated this answer based on comments and other answers, so comments prior to May 22nd 2020 may no longer apply. Also I noticed today that some IDE's like VS Code and PyCharm do not recognize a HEREDOC marker that contains spaces, whereas bash has no problem with it, so I'm updating this answer again.
Bash does not provide a builtin syntax for multi-line comment but there are hacks using existing bash syntax that "happen to work now".
Personally I think the simplest (ie least noisy, least weird, easiest to type, most explicit) is to use a quoted HEREDOC, but make it obvious what you are doing, and use the same HEREDOC marker everywhere:
<<'###BLOCK-COMMENT'
line 1
line 2
line 3
line 4
###BLOCK-COMMENT
Single-quoting the HEREDOC marker avoids some shell parsing side-effects, such as weird subsitutions that would cause crash or output, and even parsing of the marker itself. So the single-quotes give you more freedom on the open-close comment marker.
For example the following uses a triple hash which kind of suggests multi-line comment in bash. This would crash the script if the single quotes were absent. Even if you remove ###, the FOO{} would crash the script (or cause bad substitution to be printed if no set -e) if it weren't for the single quotes:
set -e
<<'###BLOCK-COMMENT'
something something ${FOO{}} something
more comment
###BLOCK-COMMENT
ls
You could of course just use
set -e
<<'###'
something something ${FOO{}} something
more comment
###
ls
but the intent of this is definitely less clear to a reader unfamiliar with this trickery.
Note my original answer used '### BLOCK COMMENT', which is fine if you use vanilla vi/vim but today I noticed that PyCharm and VS Code don't recognize the closing marker if it has spaces.
Nowadays any good editor allows you to press ctrl-/ or similar, to un/comment the selection. Everyone definitely understands this:
# something something ${FOO{}} something
# more comment
# yet another line of comment
although admittedly, this is not nearly as convenient as the block comment above if you want to re-fill your paragraphs.
There are surely other techniques, but there doesn't seem to be a "conventional" way to do it. It would be nice if ###> and ###< could be added to bash to indicate start and end of comment block, seems like it could be pretty straightforward.
After reading the other answers here I came up with the below, which IMHO makes it really clear it's a comment. Especially suitable for in-script usage info:
<< ////
Usage:
This script launches a spaceship to the moon. It's doing so by
leveraging the power of the Fifth Element, AKA Leeloo.
Will only work if you're Bruce Willis or a relative of Milla Jovovich.
////
As a programmer, the sequence of slashes immediately registers in my brain as a comment (even though slashes are normally used for line comments).
Of course, "////" is just a string; the number of slashes in the prefix and the suffix must be equal.
I tried the chosen answer, but found when I ran a shell script having it, the whole thing was getting printed to screen (similar to how jupyter notebooks print out everything in '''xx''' quotes) and there was an error message at end. It wasn't doing anything, but: scary. Then I realised while editing it that single-quotes can span multiple lines. So.. lets just assign the block to a variable.
x='
echo "these lines will all become comments."
echo "just make sure you don_t use single-quotes!"
ls -l
date
'
what's your opinion on this one?
function giveitauniquename()
{
so this is a comment
echo "there's no need to further escape apostrophes/etc if you are commenting your code this way"
the drawback is it will be stored in memory as a function as long as your script runs unless you explicitly unset it
only valid-ish bash allowed inside for instance these would not work without the "pound" signs:
1, for #((
2, this #wouldn't work either
function giveitadifferentuniquename()
{
echo nestable
}
}
Here's how I do multiline comments in bash.
This mechanism has two advantages that I appreciate. One is that comments can be nested. The other is that blocks can be enabled by simply commenting out the initiating line.
#!/bin/bash
# : <<'####.block.A'
echo "foo {" 1>&2
fn data1
echo "foo }" 1>&2
: <<'####.block.B'
fn data2 || exit
exit 1
####.block.B
echo "can't happen" 1>&2
####.block.A
In the example above the "B" block is commented out, but the parts of the "A" block that are not the "B" block are not commented out.
Running that example will produce this output:
foo {
./example: line 5: fn: command not found
foo }
can't happen
Simple solution, not much smart:
Temporarily block a part of a script:
if false; then
while you respect syntax a bit, please
do write here (almost) whatever you want.
but when you are
done # write
fi
A bit sophisticated version:
time_of_debug=false # Let's set this variable at the beginning of a script
if $time_of_debug; then # in a middle of the script
echo I keep this code aside until there is the time of debug!
fi
in plain bash
to comment out
a block of code
i do
:||{
block
of code
}
A config file that the last line contains data that I want to assign everything to the RIGHT of the = sign into a variable that I can display and call later in the script.
Example: /path/to/magic.conf:
foo
bar
ThisOption=foo.bar.address:location.555
What would be the best method in a bash shell script to read the last line of the file and assign everything to the right of the equal sign? In this case, foo.bar.address:location.555.
The last line always has what I want to target and there will only ever be a single = sign in the file that happens to be the last line.
Google and searching here yielded many close but non-relative results with using sed/awk but I couldn't come up with exactly what I'm looking for.
Use sed:
variable=$(sed -n 's/^ThisOption=//p' /path/to/magic.conf)
echo "The option is: $variable")
This works by finding and removing the ThisOption= marker at the start of the line, and printing the result.
IMPORTANT: This method absolutely requires that the file be trusted 100%. As mentioned in the comments, anytime you "eval" code without any sanitization there are grave risks (a la "rm -rf /" magnitude - don't run that...)
Pure, simple bash. (well...using the tail utility :-) )
The advantage of this method, is that it only requires you to know that it will be the last line of the file, it does not require you to know any information about that line (such as what the variable to the left of the = sign will be - information that you'd need in order to use the sed option)
assignment_line=$(tail -n 1 /path/to/magic.conf)
eval ${assignment_line}
var_name=${assignment_line%%=*}
var_to_give_that_value=${!var_name}
Of course, if the var that you want to have the value is the one that is listed on the left side of the "=" in the file then you can skip the last assignment and just use "${!var_name}" wherever you need it.
Today, I wanted to test if filenames can contain commas and stumbled upon something else while opening cmd and trying these three tests:
echo a,b>a
This works as supposed (writes a,b to the file named a)
echo a>a,b
Does just the same! What happens here gets a bit clearer with the third test:
echo a>file,b this is a test
This will create a file named file containing a,b this is a test.
Now, three questions arise for me:
What is the explanation for this? If someone asked me, I would've guessed the comma separates commands or filenames, e.g. I would've expected the second test to create two files named a and b.
Is this behaviour documented somewhere?
Is it a cmd specific Windows extension or has it been like this since good old DOS times?
It's expected behaviour as ,;=<space><tab> are delimiters for parameters.
If you put the code into a batch file without echo OFF you will see
test.bat
echo a,b>a
echo a>a,b
echo a>file,b this is a test
Output
C:\temp>test.bat
C:\temp>echo a,b 1>a
C:\temp>echo a,b 1>a
C:\temp>echo a,b this is a test 1>file
After a redirection, only the next token is relevant, the rest is part of the normal line content.
It's unimportant where the redirection occurs in a line.
But there is the rule that when more than one redirection exists for the same stream, the last one will win.
> file.txt echo hello> nul world > con
This will result in hello world at the console.
Btw. There is still an obscure behaviour with redirection and lines extended by carets (multilines).
echo one two three^
four
Result: one two three four
But
echo one two >con three^
four
Result: one two four
The comma is a standard delimiter in batch as well as ; <space> = <tab> and everything after the comma is taken as another parameter to echo and only one parameter is taken for the redirection. You can try to enclose a,b in quotes and this should change the behaviour of the output and produce a,b file. You can also escape the delimiters with ^ - echo a>a^,b
You can try also echo a>a=b - it will be the same.
I would like to write a script to execute the steps outlined below. If someone can provide simple examples on how to modify files and search through folders using a script (not necessarily solving my problem below), I will greatly appreciate it.
submit job MyJob in currentDirectory using myJobShellFile.sh to a queue
upon completion of MyJob, goto to currentDirectory/myJobDataFolder.
In myJobDataFolder, there are folders
myJobData.0000 myJobData.0001 myJobData.0002 myJobData.0003
I want to find the maximum number maxIteration of all the listed folders. Here it would be maxIteration=0003.\
In file myJobShellFile.sh, at the last line says
mpiexec ./main input myJobDataFolder
I want to append this line to
'mpiexec ./main input myJobDataFolder 0003'
I want to submit MyJob to the que while maxIteration < 10
Upon completion of MyJob, find the new maxIteration and change this number in myJobShellFile.sh and goto step 4.
I think people write python scripts typically to do this stuff, but am having a hard time finding out how. I probably don't know the correct terminology for this procedure. I am also aware that the script will vary slightly depending on the queing system, but any help will be greatly appreciated.
Quite a few aspects of your question are unclear, such as the meaning of “submit job MyJob in currentDirectory using myJobShellFile.sh to a que”, “append this line to
'mpiexec ./main input myJobDataFolder 0003'”, how you detect when a job is done, relevant parts of myJobShellFile.sh, and some other details. If you can list the specific shell commands you use in each iteration of job submission, then you can post a better question, with a bash tag instead of python.
In the following script, I put a ### at the end of any line where I am guessing what you are talking about. Lines ending with ### may be irrelevant to whatever you actually do, or may be pseudocode. Anyway, the general idea is that the script is supposed to do the things you listed in your items 1 to 5. This script assumes that you have modified myJobShellFile.sh to say
mpiexec ./main input $1 $2
instead of
mpiexec ./main input
because it is simpler to use parameters to modify what you tell mpiexec than it is to keep modifying a shell script. Also, it seems to me you would want to increment maxIter before submitting next job, instead of after. If so, remove the # from the t=$((1$maxIter+1)); maxIter=${t#1} line. Note, see the “Parameter Expansion” section of man bash re expansion of the ${var#txt} form, and the “Arithmetic Expansion” section re $((expression)) form. The 1$maxIter and similar forms are used to change text like 0018 (which is not a valid bash number because 8 is not an octal digit) to 10018.
#!/bin/sh
./myJobShellFile.sh MyJob ###
maxIter=0
while true; do
waitforjobcompletion ###
cd ./myJobDataFolder
maxFile= $(ls myJobData* | tail -1)
maxIter= ${maxFile#myJobData.} #Get max extension
# If you want to increment maxIter, uncomment next line
# t=$((1$maxIter+1)); maxIter=${t#1}
cd ..
if [[ 1$maxIter -lt 11000 ]] ; then
./myJobShellFile.sh MyJobDataFolder $maxIter
else
break
fi
done
Notes: (1) To test with smaller runs than 1000 submissions, replace 11000 by 10000+n; for example, to do 123 runs, replace it with 10123. (2) In writing the above script, I assumed that not-previously-known numbers of output files appear in the output directory from time to time. If instead exactly one output file appears per run, and you just want to do one run per value for the values 0000, 0001, 0002, 0999, 1000, then use a script like the following. (For testing with a smaller number than 1000, replace 1000 with (eg) 0020. The leading zeroes in these numbers tell bash to fill the generated numbers with leading zeroes.)
#!/bin/sh
for iter in {0000..1000}; do
./myJobShellFile.sh MyJobDataFolder $iter
waitforjobcompletion ###
done
(3) If the system has a command that sleeps while it waits for a job to complete on the supercomputing resource, it is reasonable to use that command in place of waitforjobcompletion in the above scripts. Otherwise, if the system has a command jobisrunning that returns true if a job is still running, replace waitforjobcompletion with something like the following:
while jobisrunning ; do sleep 15; done
This will run the jobisrunning command; if it returns true, the shell will sleep for 15 seconds and then retest. Here is an example that illustrates waiting for a file to appear and then for it to go away:
while [ ! -f abc ]; do sleep 3; echo no abc; done
while ls abc >/dev/null 2>&1; do sleep 3; echo an abc; done
The second line's test could be [ -f abc ] instead; I showed a longer example to illustrate how to suppress output and error messages by routing them to /dev/null. (4) To reverse the sense of a while statement's test, replace the word while with until. For example, while [ ! -f abc ]; ... is equivalent to until [ -f abc ]; ....
I am trying to read part of a file and stop and a particular line, using bash. I am not very familiar with bash, but I've been reading the manual and various references, and I don't understand why something like the following does not work (but instead produces a syntax error):
while { read -u 4 line } && (test "$line" != "$header_line")
do
echo in loop, line=$line
done
I think I could write a loop that tests a "done" variable, and then do my real tests inside the loop and set "done" appropriately, but I am curious as to 1) why the above does not work, and 2) is there some small correction that would make it work? I still fairly confused about when to use [, (, {, or ((, so perhaps some other combination would work, though I have tried several.
(Note: The "read -u 4 line" works fine when I call it above the loop. I have opened a file on file descriptor 4.)
I think what you want is more like this:
while read -u 4 line && test "$line" != "$header_line"
do
...
done
Braces (the {} characters) are used to separate variables from other parts of a string when whitespace cannot be used. For example, echo "${var}x" will print the value of the variable var followed by an x, but echo "$varx" will print the value of the variable varx.
Brackets (the [] characters) are used as a shortcut for the test program. [ is another name for test, but when test detects that it was called with [ it required a ] as its last argument. The point is clarity.
Parenthesis (the () characters) are used in a number of different situations. They generally start subshells, although not always (I'm not really certain in case #3 here):
Retrieving a single exit code from a series of processes, or a single output stream from a sequence of commands. For example, (echo "Hi" ; echo "Bye") | sed -e "s/Hi/Hello/" will print two lines, "Hello" and "Bye". It is the easiest way to get multiple echo statements to produce a single stream.
Evaluating commands as if they were variables: $(expr 1 + 1) will act like a variable, but will produce the value 2.
Performing math: $((5 * 4 / 3 + 2 % 1)) will evaluate like a variable, but will compute the result of that mathematical expression.
The && operator is a list operator - he seperates two commands and only executes when the first is true, but in this case the first is the while and he is expecting his do stuff. And then he reaches do and the while stuff is already history.
Your intention is to put it into the expression. So you put it together with (). E.g. this a solution with just a small change
while ( read -u 4 line && test "$line" != "$header_line" )