I'm trying to write a quick batch file. It will take the result of a command, put some extra text and quotes around it, and put that into a new file.The problem is that the result of the command I'm running includes a new line. Here's the command:
p4 changelists -m 1 -t //depot/...> %FILENAME%
The output of that p4 command has a newline at the end of it. The file I'm putting it into needs to have quotes surrounding the output of that command, but the fact that the command contains a newline in it means that the "closing quote" appears on a new line in the file, which doesn't work for what I'm doing.
I've tried writing the output of that command into a file and reading it back in, and also trying to run FINDSTR on a file containing the output, but I always seem to get back the stupid trailing whitespace. I've even tried inserting backspaces into the file, but that just put a backspace character into the file instead of actually executing a backspace...
Is there anything to be done about this?
I'm no perl wizard, but the following seems to work:
p4 changelists -m 1 -t //depot/...| perl -p -e "s/^/\042/;s/$/\042/"
Check out Strawberry Perl, which provides a Windows version of Perl.
I'm always looking at my Unix tools when solving problems like this, even under Windows. sed and gawk will also get you there, check out msysgit for a nice bundle of Unix tools that will run on Windows.
Related
I have following snippet:
host="https://example.com"
port="80"
url="${host}:${port}"
echo $url
the output is:
:80ps://example.com
How can I escape the colon here. I also tried:
url="${host}\:${port}"
but it did not work.
Expected output is:
https://example.com:80
You've most likely run into what I call the Linefeed-Limbo.
If I copy the code you provided from StackOverflow and run it on my machine (bash version 4.4.19(1)), then it outputs correctly
user#host:~$ cat script.sh
host="https://example.com"
port="80"
url="${host}:${port}"
echo $url
user#host:~$ bash script.sh
https://example.com:80
What is Linefeed-Limbo?
Different operating systems use different ASCII symbols to represent when a new line occurs in a text, such as a script. This Wikipedia article gives a good introduction.
As you can see, Unix and Unix-like systems use the single character \n, also called a "Line Feed". Windows, as well as other systems, use \r\n, so a "carriage return" followed by a "line feed".
What happens now is when you write a script on Windows on an editor such as notepad, what you write is host="example.com"\r\n. When you copy this file into Linux, Linux interprets the \r as if it were part of the script, since only \n is considered a new line. And indeed, when I change my newline style to DOS-style, I get the exact output you get.
How can I fix this?
You have several options to fix this issue.
Converting the script (with dos2unix)
Since all you need to do is replacing every instance of \r\n with \n, you could use any text-editing software you want. However, if you like simple solutions, then dos2unix (and its sister unix2dos) might be what you looking for:
user#host:~$ dos2unix script.sh
dos2unix: converting file script.sh to Unix format...
That's it. Run your file now and you will see it behaves well.
Encoding the source-file correctly
By using a more advanced text editor such as Notepad++, you can define which style of newline you would like to use.
By changing the newline-type to whichever system you intend to run your script on, you will not run into any problems like this anymore.
Bonus round: Why does it output :80ps://example.com?
To understand why your output is like this, you have to look at what your script is doing, and what \r means.
Try thinking of your terminal as an old-fashioned typewriter. Returning the carriage means you start writing on the left again. Making a "new line" means sliding the paper. These two things are seperate, and I think that's why some systems decided to use these two characters as a logical "new line".
But I digress. Let's look at the first line, host="https://example.com"\r.
What this means when printed is "Print https://example.com, then put the carriage back at the start". When you then print :80\r, it doesn't start after ".com", it starts at the beginning of the line, because that's where you (unknowingly) told the cursor to go. it then overwites the first few characters, resulting in ":80ps://example.com" to be written. Keep in mind that after 80, you again placed a carriage return symbol, so any new text you would have written ends up overwriting the beginning again.
It works for me, try to remove carriage returns in variables and then try.
new_host=$(echo "$host" | tr -d '\r')
new_port=$(echo "$port" | tr -d '\r')
new_url="${new_host}:${new_port}"
I would like to run a Bash shell script (.sh) using the Windows Subsystem for Linux as part of a Build Event in Visual Studio, in a C++ project.
But there are lots of errors when I do this, and I can't seem to find the right combination of quotation marks and apostrophes and backslashes to either make Bash run in the first place, or to properly pass the path to the script.
How do I make Visual Studio run my Bash shell script as a build event?
(Feel free to skip to the bottom of this answer if you don't care about how to solve the problem and just want a command you can copy and paste!)
Overview
I run a number of Bash shell scripts as part of Build events in Visual Studio, and I used to use Cygwin to run them. Now that the Windows Subsystem for Linux is available, I spent some time switching my builds over to use WSL, and it wasn't as easy as I'd hoped, but it can work, with a little time and energy.
There are several issues you'll run into if you're going to do this:
The path to bash.exe may not be what you think it is, because under the hood, Visual Studio uses a 32-bit build process, so if you're on a 64-bit machine, you can't simply run the 64-bit bash.exe without getting the dreaded 'C:\Windows\System32\bash.exe' is not recognized error.
The path to your solution or project is a Windows path that uses backslashes (\), and those don't play nice in Unix, which prefers forward slashes (/) as a path delimiter.
The root drive of the solution, typically something like C:\, is meaningless gibberish in Unix; to reach the root drive in WSL, you'll need to use a mounted drive under /mnt.
The casing of the drive letter is different between Windows and WSL: In Windows, it's uppercase C:\, and in WSL, it's lowercase /mnt/c.
And to make it a little harder, we don't want to hard-code any of the paths: It should all Just Work, no matter where the solution is found.
The good news is that they're all solvable issues! Let's tackle them one at a time.
Fixing the Issues
1. The proper path to Bash
Per the answer given here, you'll need to use a magic path to reach Bash when running it from a Visual Studio build. The correct path is not C:\Windows\System32\bash.exe, but is actually
%windir%\sysnative\bash.exe
The magic sysnative folder avoids the invisible filesystem redirection performed by the WOW64 layer, and points to the native bash.exe file.
2. Fixing the backslashes
The next problem you're likely to run into is the backslashes. Ideally, you'd like to run a project script like $(ProjectDir)myscript.sh, but that expands to something like C:\Code\MySolution\MyProject\myscript.sh. At a minimum, you'd like that to be at least C:/Code/MySolution/MyProject/myscript.sh, which isn't exactly right, but which is a lot closer to correct.
So sed to the rescue! sed is a Unix tool that mutates text in files: It searches for text using regular expressions, and, among other things, can replace that text with a modified version. Here, we're going to pipe the path we have into sed, and then use some regex magic to swap the path separators, like this (with lines wrapped here for readability):
%windir%\sysnative\bash.exe -c "echo '$(ProjectDir)myscript.sh'
| sed -e 's/\\\\/\//g;'"
If you include this as your build event, you'll see that it now doesn't run the script, but it at least prints something like C:/Code/MySolution/MyProject/myscript.sh to the output console, which is a step in the right direction.
And yes, that's a lot of backslashes and quotes and apostrophes to get the escaping right, because Nmake.exe and bash and sed are all going to consume some of those special symbols while processing their respective command-lines.
3. Fixing the C:\ root path
We want to mutate the sed script so that it turns the C:\ into /mnt/C. A little more regex substitution magic can do that. (And we have to turn on the -r flag in sed so that we can easily use capture groups.)
%windir%\sysnative\bash.exe -c "echo '$(ProjectDir)myscript.sh'
| sed -re 's/\\\\/\//g; s/([A-Z]):/\/mnt\/\1/i;'"
If you run this, you'll now see the output path as something like /mnt/C/Code/MySolution/MyProject/myscript.sh, which is almost but not quite correct.
4. Fixing the case-change in the root path
WSL mounts your disks in lowercase, and Windows mounts them in uppercase. Consistency! How do we fix this? Yet more sed magic!
The \L command can be used to tell sed to transform succeeding characters to lowercase (and there's an equivalent \U for uppercase). The \E command will switch output back to "normal" mode, where characters are left untouched.
Adding these in finally results in the correct path being output:
%windir%\sysnative\bash.exe -c "echo '$(ProjectDir)myscript.sh'
| sed -re 's/\\\\/\//g; s/([A-Z]):/\/mnt\/\L\1\E/i;'"
5. Running it
This whole time, Bash has just been printing out the path to the script. How do we run it instead, now that it's the correct path?
The answer is to add `backticks`! Backticks cause Bash to execute the command contained within them, and to then use that command's output as the arguments to the next command. In this case, we're not going to output anything: We just want to run the output of sed as a command.
So including the backticks, here's the result:
%windir%\sysnative\bash.exe -c "`echo '$(ProjectDir)myscript.sh'
| sed -re 's/\\\\/\//g; s/([A-Z]):/\/mnt\/\L\1\E/i;'`"
The Complete Solution
Here's what the whole solution looks like, for running a script named myscript.sh as a Build Event, in the current Project directory of the current Solution:
%windir%\sysnative\bash.exe -c "`echo '$(ProjectDir)myscript.sh' | sed -re 's/\\\\/\//g; s/([A-Z]):/\/mnt\/\L\1\E/i;'`"
Here's a screen-shot showing the same thing in Visual Studio 2017, for a real C++ project:
It's not easy, but it's not impossible.
If you have Git for Windows installed, try this. I found it simpler than installing WSL. The basic idea is to create an intermediate batch script to call your bash script, using Git bash's in-built bash or sh command from the batch script.
With Git for Windows, you'll have a Git\bin folder e.g. at:
C:\Program Files\Git\bin
Inside that directory you should see the bash.exe and sh.exe programs. So if you add that directory to your Windows Path environment variable then you'll be able to use sh and bash from the Windows command line. These commands will allow you to run your bash scripts "inline" within a CMD console window. That is, they won't spawn a new bash window; meaning the console output will be visible in your VS build.
From there, just create a .bat file which calls your .sh file using either the sh command or the bash command. Not sure the difference; we just use the sh command. So if your bash script is pre.sh, then your batch file would be just a single line calling the bash script:
sh %~dp0\pre.sh
if errorlevel 1 (
exit /b %errorlevel%
)
The %~dp0 assumes the batch and bash scripts are in the same directory. You then point your VS build event to the .bat file. The check for error level is necessary so that any failures from the bash script are forwarded up to the batch script. See: How do I get the application exit code from a Windows command line?.
To hook this in as a build event in VS2019 then, just follow the standard instructions for hooking in a .bat file: https://learn.microsoft.com/en-us/visualstudio/ide/specifying-custom-build-events-in-visual-studio?view=vs-2019.
Update: Beware Visual Studio's (VS's) Path Variable Behaviour
One thing we found quite frustrating with this solution was the tendency of VS to not load in the path variable correctly. It seems to prefer the user variable over the system variable. But even after we deleted the user variable, sometimes the path didn't seem to be getting picked up by VS, and we kept getting "sh is not recognised..." messages on our build console. Whenever that happened, restarting VS seemed to do the trick. Not very satisfying, but it gets us by.
Update: This is not a Full Unix Solution
Git for Windows does have a lot of Unix commands available, but not all of them. So in general, this won't work. For the general case, WSL is more robust. However, if it's just pretty lightweight Unix you need, this will suffice, and will likely be an easier approach for Windows users who would rather avoid the steeper setup cost of installing the full WSL.
Original idea to use Git bash came from here: https://superuser.com/questions/1218943/windows-command-prompt-capture-output-of-bash-script-in-one-step
Instead of backticks, you can wrap command with $( and )
I am currently trying to create a script that executes a program 100 times, with different parameters, typically pretty simple, but it's adding strange characters into the output filename that is passed into the command call for the program, the script i have written goes as follows
#!/bin/bash
for i in {1..100}
do ./generaterandomizedlist 10 input/input_10_$i.txt
done
I've taken a small screenshot of the output file name here
https://imgur.com/I855Hof
(extra characters are not recognized by chrome so simply pasting the name doesn't work)
It doesn't do this when i manually call the command issued in the script, any ideas?
Your script has some stray CRs in it. Use dos2unix or tr to fix it.
Due to processes out of my control I need run multiple SH files which contains lengthy CURL commands. Problem is that whichever process created these commands seems to have included one line of whitespace at the very end. If I call it as is - it fails. If I physically open the file and hit backspace on the first full empty line and save the file - it works perfectly.
Any way to put some kind of command into the SH file so that it removes any unnecessary stuff?
More info would be helpful, but the following might work:
If you need to put something into each of the files that contain the curl commands as you mention, you could try putting exit as the last line of the curl script (also depends on how you're calling the 'curl files'
exit
If you can run a separate script against the files that have a blank line, perhaps sed the blank lines away?
sed -i s/^\s$// $fileWithLineOfSpaces
edit:
Or (after thinking about it), perhaps simply delete the last line of the file....
sed -i '$d' $file
I once write a simple sed command like this
s/==/EQU/
while I run it in command line:
sed 's/==/EQU' filename
it works well, replace the '==' with 'EQU', but while I write the command to a script file named replace.sed, run it in this way:
sed -f replace.sed filename
there is a error, says that
sed: file replace.sed line 1: unknwon option to 's'
What I want to ask is that is there any problem with my script file replace.sed while it run in windows?
The unknown option is almost invariably a rogue character after the trailing / (which is missing from your command line version, by the way so it should complain about an unterminated command).
Have a look at you replace.sed again. You may have a funny character at the end, which could include the ' if you forgot to delete it, or even a CTRL-M DOS-style line ending, though CygWin seems to handle this okay - you haven't specified which sed you're using (that may help).
Okay, based on your edit, it looks like one of my scattergun of suggestions was right :-) You had CTRL-M at the end of the line because of the CR/LF line endings:
At the end of each line in the *.sed file, there was a 'CR\LF' pair, and that the problem, but you cannot see it by default, I use notepad to delete them manually and fix the problem. But I have not find a way to delete it automatically or do not contain the 'new-line' style while edit a new text file in windows.
You may want to get your hands on a more powerful editor like Notepad++ or gVim (my favourite) but, in fact, you do have a tool that can get rid of those characters :-) It's called sed.
sed 's/\015//g' replace.sed >replace2.sed
should get rid of all the CR characters from your file and give you a replace2.sed that you can use for your real job.