Unexpected command line behavior using commas in Windows/DOS batch file - windows

Today, I wanted to test if filenames can contain commas and stumbled upon something else while opening cmd and trying these three tests:
echo a,b>a
This works as supposed (writes a,b to the file named a)
echo a>a,b
Does just the same! What happens here gets a bit clearer with the third test:
echo a>file,b this is a test
This will create a file named file containing a,b this is a test.
Now, three questions arise for me:
What is the explanation for this? If someone asked me, I would've guessed the comma separates commands or filenames, e.g. I would've expected the second test to create two files named a and b.
Is this behaviour documented somewhere?
Is it a cmd specific Windows extension or has it been like this since good old DOS times?

It's expected behaviour as ,;=<space><tab> are delimiters for parameters.
If you put the code into a batch file without echo OFF you will see
test.bat
echo a,b>a
echo a>a,b
echo a>file,b this is a test
Output
C:\temp>test.bat
C:\temp>echo a,b 1>a
C:\temp>echo a,b 1>a
C:\temp>echo a,b this is a test 1>file
After a redirection, only the next token is relevant, the rest is part of the normal line content.
It's unimportant where the redirection occurs in a line.
But there is the rule that when more than one redirection exists for the same stream, the last one will win.
> file.txt echo hello> nul world > con
This will result in hello world at the console.
Btw. There is still an obscure behaviour with redirection and lines extended by carets (multilines).
echo one two three^
four
Result: one two three four
But
echo one two >con three^
four
Result: one two four

The comma is a standard delimiter in batch as well as ; <space> = <tab> and everything after the comma is taken as another parameter to echo and only one parameter is taken for the redirection. You can try to enclose a,b in quotes and this should change the behaviour of the output and produce a,b file. You can also escape the delimiters with ^ - echo a>a^,b
You can try also echo a>a=b - it will be the same.

Related

compatible comments for both bash and batch

We write two similar scripts: one for bash (linux) and one for batch (dos/windows).
Even if the specific code is different we would like to visually compare (diff) both scripts and have the similar parts of code aligned side by side.
We use explicit comments with the same text to achieve this. But the beginning of the comments is different in both scripting (REM or :: in windows) and (# in linux).
Therefore there is a wrong alignment:
linux
windows
# first step
REM first step
foo.sh
foo.bat
# second step
REM second step
bar.sh
bar.bat
Is there a way to use a common character or sequence of characters to make the comments equal?
Is the use of : #; safe for both systems/scripts?
linux
windows
: #; first step
: #; first step
foo.sh
foo.bat
: #; second step
: #; second step
bar.sh
bar.bat
Are there any unwanted side effects?
: in bash is not exactly a comment. It is a void command.
A little bit like pass in some languages.
It helps, for example, to fill empty slots, if needed
if condition
then
:
else
doSomething
fi
So, you may use, somehow, as a sort of comment. That would works both in bash and batch (well, I know nothing of batch. But since you said that :: is a comment there). But beware that it is not exactly a comment. So there are some differences
For example
#!/bin/bash
echo one ||
:: foo
echo two
echo un ||
# bar
echo deux
Displays one, two and un but not deux.
Because echo one || prints one and then execute the following command only if it fails (which it doesn't). Here the following command is :: foo. Which is not executed (you wouldn't know, since it does nothing, but it is not executed). And the echo two is a brand new unrelated command that is executed.
Whereas, on the other hand, echo un || likewise prints un, and doesn't execute the next command, since echo un did not fail. But the next command here is echo deux. Because # bar doesn't count, since it is a comment.
And that is only one of the many examples one could probably find to show that : is not a comment.
But, well, if you use it being aware of that, I suppose you could use it to insert void comments in your bash code that would also be void in batch.
Edit:
I won't edit for each new example that comes to mind. But that one is pretty important
echo un # deux
echo one : two
prints
un
one : two
: is a command. So, as other commands, like ls not all occurrence of it is treated as so (no more than echo ls list the directory constant. ls is just a string here)
So, you can't use it as a replacement for inline comments. Only for full lines comments.

Newlines in shell script variable not being replaced properly

Situation: Using a shell script (bash/ksh), there is a message that should be shown in the console log, and subsequently sent via email.
Problem: There are newline characters in the message.
Example below:
ErrMsg="File names must be unique. Please correct and rerun.
Duplicate names are listed below:
File 1.txt
File 1.txt
File 2.txt
File 2.txt
File 2.txt"
echo "${ErrMsg}"
# OK. After showing the message in the console log, send an email
Question: How can these newline characters be translated into HTML line breaks for the email?
Constraint: We must use HTML email. Downstream processes (such as Microsoft Outlook) are too inconsistent for anything else to be of use. Simple text email is usually a good choice, but off the table for this situation.
To be clear, the newlines do not need to be completely removed, but HTML line breaks must be inserted wherever there is a newline character.
This question is being asked because I have already attempted to use several commands, such as sed, tr, and awk with varying degrees of success.
TL;DR: The following snippet will do the job:
ErrMsg=`echo "$ErrMsg"|awk 1 ORS='<br/>'`
Just make sure there are double quotes around the variable when using echo.
This turned out to be a tricky situation. Some notes of explanation are below.
Using sed
Turns out, sed reads through input line by line, which makes finding and replacing those newlines somewhat outside the norm. There were several clever tricks that appeared to work, but I felt they were far too complicated to apply appropriately to this rather simple situation.
Using tr
According to this answer the tr command should work. Unfortunately, this only translates character by character. The two character strings are not the same length, and I am limited to translating the newline into a space or other single character.
For the following:
ErrMsg="Line 1
Line 2
"
ErrMsg=`echo $ErrMsg| tr '\n' 'BREAK'`
# You might expect:
# "Line 1BREAKLine 2BREAK"
# But instead you get:
# "Line 1BLine 2B"
echo "${ErrMsg}"
Using awk
Using awk according to this answer initially appeared to work, but due to some other circumstances with echo there was a subtle problem. The solution is noted in this forum.
You must have double-quotes around your variable, or echo will strip out all newlines.(Of course, awk will receive the characters with a newline at the end, because that's what echo does after it echos stuff.)
This snippet is good: (line breaks in the middle are preserved and replaced correctly)
ErrMsg=`echo "$ErrMsg"|awk 1 ORS='<br/>'`
This snipped is bad: (newlines converted to spaces by echo, one line break at end)
ErrMsg=`echo $ErrMsg|awk 1 ORS='<br/>'`
You can wrap your message in HTML using <pre>, something like
<pre>
${ErrMsg}
and more.
</pre>

Comparing/finding the difference between two text files using findstr

I have a requirement to compare two text files and to find out the difference between them. Basically I have an input file (input.txt) which will be processed by a batch job and my batch will log the output (successful.txt) where the job has successfully ran.
In simple words, I need to find out the difference between input.txt and successful.txt (input.txt-successful.txt) and I was thinking to use findstr. It seems to be fine, BUT I don't understand one part of it. It always includes the last line of my input.txt in the output. You could see that in the example below. Please note that there is no leading space or line break after the last line of my input.txt.
In below example, you could see the line server1,db1 is present on both the files, but still listed in the output. (It is always the last line of input.txt)
D:\Scripts\dummy>type input.txt
server2,db2
server3,db3
server10,db10
server4,db4
server1,db11
server10,schema11
host1,sch2
host11,sql2
host11,sql3
server1,db1
D:\Scripts\dummy>type successful.txt
server1,db1
server2,db2
server3,db3
server4,db4
server10,db10
host1,sch2
host11,sql2
host11,sql3
D:\Scripts\dummy>findstr /vixg:successful.txt input.txt
server1,db11
server10,schema11
server1,db1
What am I doing wrong?
Cheers,
G
I could reproduce your results by removing the newline after the last line of input.txt, so solution 1 would be to add a newline to the end of input.txt. Since you appear to say that input.txt has no terminal newline, then adding one would cure the problem; findstr is acting as expected because it acts on newline-terminated lines.
Solution 2 would be
type input.txt|findstr /vixg:successful.txt

Echoing to a file results in spaces in Batch

Using a batch file (.bat), I'm making a script that requires dynamic paths so that it can work on multiple computers. My problem is when I echo something to a file, it adds a line and an a return carriage.
Say I have a text file named foo.txt in the directory of the batch file, and its contents are completely empty.
In the batch file, I run:
echo test > foo.txt
The contents of foo.txt will be:
L1: foo
L2:
There would be a space after foo in the first line and a second empty line. Now, this would be completely okay and I would entirely ignore it, but filename paths do not ignore it.
importing text from foo.txt like so:
set /p foo=< foo.txt
...and then:
set /p name=< C:\A.D.V.E.N.T.U.R.E.\test\%foo%\test2.txt
...would be interpreted as:
set /p name=< C:\A.D.V.E.N.T.U.R.E.\test\foo \test2.txt
Including an unwanted space. Is there anyway to make it so you can write text to a file without a space, or a command one could use to delete the carriage return and the space?
You can also use parentheses to make sure unwanted space is not included in the output:
(echo test) >foo.txt
The data should be test, not foo
Clasically, try
>foo.txt echo test
but make sure that there are no trailing spaces after test.
(to APPEND to foo.txt use >> in place of >)
above given answers works.
But, the actual reason I found for it was the space before >
So, instead of
echo test > foo.txt
it must be
echo test> foo.txt
NOTE. Don't put any space between test and >. This, results in a trailing space.

Echo misses ^ characters when long string

I have the following command in a windows batch script
echo =%%k-16,INDIRECT.EXT^("'C:\Users\...\Analysis\[ObsStreamflow.xlsx]Sheet1'^!A%%k"^),INDIRECT.EXT^("'C:\Users\...\Analysis\[sim%%j.xlsx]Sheet1'^!B!val!"^),^=C%%k/1000,^=D%%k-B%%k,^=ABS^(E%%k^),^=(E%%k^)^^2,=^(B%%k-B10^),=Sqrt^(B%%k^),=SQRT^(D%%k^),=^(J%%k - B13^)^^2 >>t%%j.csv
where the omitted file path is 38 characters long (I don't think I'm hitting the line limits, but just in case this is the problem). This is a single line in my .bat file, shown here as multiple lines just to make things more readable.
The output is mostly correct, except that where I have ^^2, it just becomes 2 (so I have =(E1)2 and =(J1-B13)2. If I omit the Indirect.Ext text, and just have
echo =%%k-16,a1,b1,^=C%%k/1000,^=D%%k-B%%k,^=ABS^(E%%k^),^=(E%%k^)^^2,=^(B%%k-B10^),=Sqrt^(B%%k^),=SQRT^(D%%k^),=^(J%%k - B13^)^^2 >>t%%j.csv
it prints correctly, so the relevant comments show as =(E1)^2 and =(J1-B13)^2, which is what I am after.
I've not had any luck finding an answer, everything I have found just points to using ^^ to get echo to return ^. I cannot break this command into multiple lines, I need it to be a single row in csv format.
Any suggestions for a fix much appreciated, I only really need to use this for a week or so, don't need an elegant solution, just one that works. - I'm very new to bat scripts (and indeed programming in general), will keep trying different ideas in the mean time.
It's only the exclamation mark that creates the problems for you.
If at least one ! is in your line (and delayed expansion is enabled), then a second caret escape phase will be started.
In this phase quotes aren't regarded, only carets.
A small test
setlocal EnableDelayedExpansion
echo one^1
echo two^^2
echo two^^2 With exclam!
echo five^^^^^& With exclam!
Output
one1
two^2
two2 With exclam
four^& With exclam
So in your sample, you need five carets.
Four to create one caret and the last one to escape the ), as the escape of the special character is only once required.
Not sure what your specific problem is but you can use a trick in Windows to emulate echo -n (echo without a newline).
The commands:
<nul: >file.csv set /p junk=first field
<nul: >>file.csv set /p junk=,second field
>>file.csv echo ,third field
will result in a single line:
first field,second field,third field
That may make it easier for you to avoid the specific problem and, as a bonus, clean up your script so it's a little more readable (such as one field per script line).
It works because set /p var=prompt is the input command. It first outputs prompt without a newline then waits for the user to enter something, assigning it to the var environment variable.
By getting input from nul:, you basically give it an empty string so it doesn't wait. The prompt is output to file.csv without the newline.
In any case, for something this complex, I'd be bypassing cmd.exe for something a little more powerful such as the UNIX text processing tools under CygWin or MinGW (which require installation but are well worth it), or even VBScript scripts (which should be on Windows by default), where you can more easily control the output.

Resources