I'm creating a simple, single line batch file to run a git log in my preferred format. When I'm in PowerShell, I type:
git log --pretty=format:"%ai%x09%H%x09%s" > commitlog.csv
And it gives me exactly the output I want. It's the commit author's date and time in ISO format, followed by a tab, followed by the full SHA Hash, followed by a tab, followed by the notes/description. It opens in Excel exactly the way I want.
I tried to make a simple batch file to do the exact same thing so I wouldn't have to remember the format string or search the command history (which occasionally gets cleared, anyway). I noted that I had to use a double percent, %% to replace all the percent signs.
In one version of the batch file I have this line:
git log --pretty=format:"%%ai%%x09%%H%%x09%%s" > commitlog.csv
Which I noted echos the command like this:
git log --pretty=format:"%ai%x09%H%x09%s" 1>commitlog.csv
Which has a "1" interjected into it for some reason, but oddly creates a file with the information, just excluding the tabs, so it all runs together. I read that greater than '>' needs to be preceded by a carat, like so '^>'
So, in the next version of the batch file I changed the line to this:
git log --pretty=format:"%%ai%%x09%%H%%x09%%s" ^> commitlog.csv
Which echos the command like this:
git log --pretty=format:"%ai%x09%H%x09%s" > commitlog.csv
Which I've stared at for a long time, many times, and appears to be EXACTLY what I type in PowerShell, and I would expect to get the same result, but instead I get an error message:
fatal: ambiguous argument '>': unknown revision or path not in the
working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
What am I doing wrong?
The git log --pretty=format: command is terminating each line with just a newline (0x0A). Most tools on Windows (like Notepad) expect lines to be terminated with a carriage return (0x0D) and and newline. However, cmd has special handling for just a newline that makes the console output look pretty.
Thus, the batch file should look something like this:
git log --pretty=format:"%%ai%%x09%%H%%x09%%s%%x0D" > commitlog.csv
Notice the extra %%x0D to add the carriage return manually. As you found out already, the %% is needed to escape the single % inside of a batch script.
1> and just > are basically equivalent here. 1 is just the numeric value for stdout (compare to 2 for stderr), but the default is to redirect stdout. Cmd is just normalizing before printing to command that its going to run.
Minor nit: you may want to redirect to commitlog.tsv, as the values are separated by tabs, not commas. :-)
Git on Windows is often plagued by these sort of newline issues, so be on the look of for them elsewhere.
Related
I am trying to install Operator Lifecycle Manager (OLM) — a tool to help manage the Operators running on your cluster — from the official documentation, but I keep getting the error below. What could possibly be wrong?
This is the result from the command:
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.22.0/install.sh | bash -s v0.22.0
/bin/bash: line 2: $'\r': command not found
/bin/bash: line 3: $'\r': command not found
/bin/bash: line 5: $'\r': command not found
: invalid option6: set: -
set: usage: set [-abefhkmnptuvxBCEHPT] [-o option-name] [--] [-] [arg ...]
/bin/bash: line 7: $'\r': command not found
/bin/bash: line 9: $'\r': command not found
/bin/bash: line 60: syntax error: unexpected end of file
I've tried removing the existing curl and downloaded and installed another version but the issue has still persisted. Most solutions online are for Linux users and they all lead to Windows path settings and files issues.
I haven't found one tackling installing a file using curl.
I'll gladly accept any help.
Using PowerShell on Windows, you must explicitly ensure that the stdout lines emitted by curl.exe are separated with Unix-format LF-only newlines, \n, when PowerShell passes them on to bash, given that bash, like other Unix shells, doesn't recognize Windows-format CRLF newlines, \r\n:
The simplest way to avoid the problem is to call via cmd /c:
cmd /c 'curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.22.0/install.sh | bash -s v0.22.0'
cmd.exe's pipeline (|) (as well as its redirection operator, >), unlike PowerShell's (see below), acts as a raw byte conduit, so it simply streams whatever bytes curl.exe outputs to the receiving bash call, unaltered.
Fixing the problem on the PowerShell side requires more work, and is inherently slower:
(
(
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.22.0/install.sh
) -join "`n"
) + "`n" | bash -s v0.22.0
Note: `n is a PowerShell escape sequence that produces a literal LF character, analogous to \n in certain bash contexts.
Note:
It is important to note that, as of PowerShell 7.2.x, passing raw bytes through the pipeline is not supported: external-program stdout output is invariably decoded into .NET strings on reading, and re-encoded based on the $OutputEncoding preference variable when writing to an(other) external program.
See this answer for more information, and GitHub issue #1908 for potential future support for raw byte streaming between external programs and on redirection to a file.
That is, PowerShell invariably interprets output from external programs, such as curl.exe, as text, and sends it line by line through the pipeline, as .NET string objects (the PowerShell pipeline in general conducts (.NET) objects).
Note that these lines (strings) do not have a trailing newline themselves; that is, the information about what specific newline sequences originally separated the lines is lost at that point (PowerShell itself recognizes CRLF and LF newlines interchangeably).
However, if the receiving command is also an external program, PowerShell adds a trailing platform-native newline to each line, which on Windows is a CRLF newline - this is what caused the problem.
By collecting the lines in an array up front, using (...), they can be sent as a single, LF-separated multi-line string, using the -joinoperator, as shown above.
Note that PowerShell appends a trailing platform-native newline to this single, multi-line string too, but a stray \r\n at the very end of the input is in effect ignored by bash, assuming that the last true input line ends in \n, which is what the extra + "`n" at the end of the expression ensures.
However, there are scenarios where this trailing CRLF newline does cause problems - see this answer for an example and workarounds via the platform-native shell.
To start off, I need to be clear about a few things:
Based on the tags to the question, I see we are in PowerShell rather than a linux/unix or even Windows cmd shell
In spite of this, we are using Unix curl (probably curl.exe), and not the PowerShell alias for Invoke-WebRequest. We know this because of the -sL argument. If Powershell was using the alias, we'd see a completely different error.
Next, I need to talk briefly about line endings. Instead of just a single LF (\n) character as seen in Unix/linux and expected by bash, Windows by default uses the two-character LF/CR pair (\n\r) for line endings.
With all that background out of the way, I can now explain what's causing the problem. It's this single pipe character:
|
This a PowerShell pipe, not a Unix pipe, so the operation puts the output of the curl program in the PowerShell pipeline in order to send it to the bash interpreter. Each line is an individual item on the pipeline, and as such no longer includes any original line breaks. PowerShell pipeline will "correct" this before calling bash using the default line ending for the system, which in this case is the LF/CR pair used by Windows. Now when bash tries to interpret the input, it sees an extra \r character after every line and doesn't know what to do with it.
The trick is most of what we might do in Powershell to strip out those extra characters is still gonna get sent through another pipe after we're done. I guess we could tell curl to write the file to disk without ever using a pipe, and then tell bash to run the saved file, but that's awkward, extra work, and much slower.
But we can do a little better. PowerShell by default treats each line returned by curl as a separate item on the pipeline. We can "trick" it to instead putting one big item on the pipeline using the -join operation. That will give us one big string that can go on the pipeline as a single element. It will still end up with an extra \r character, but by the time bash sees it the script will have done it's work.
Code to make this work is found in the other answer, and they deserve all the credit for the solution. The purpose of my post is to do a little better job explaining what's going on: why we have a problem, and why the solution works, since I had to read through that answer a couple times to really get it.
I am using Windows and MobaXterm.
I created a .bash_profile file in the ~ directory and the following line
alias sbp="source ~/.bash_profile"
is the only code in that file.
However, when I was trying to do sbp, I got an error.
This works on my Mac and it used to work on my old Windows computer (but that one has some water damage so it broke down). Why does this not work now?
Thanks in advance!
From the way that error message is garbled I'm pretty sure that the .bash_profile file you created has DOS/Windows-style line endings, consisting of a carriage return character followed by a newline character. Unix tools expect unix-style line endings consisting of just a newline; if they see DOS/Windows-style endings, they'll treat the carriage return as part of the content of the line. In this case, bash will treat the carriage return as part of the alias definition, and therefore part of the filename to filename to source. Try running alias sbp | cat -vt to print the alias with invisible characters shown; my guess is it'll print alias sbp='source ~/.bash_profile^M' (where the ^M is cat -vt's way of representing the carriage return).
Solution: convert the file to unix format, and either switch to a text editor that knows how to save in unix format, or change your settings in the current editor to do it. For conversion, there are a number of semi-standard tools like dos2unix and fromdos. If you don't have any of those, this answer has some other options.
BTW, the reason the error message is garbled is that the CR gets printed as part of the error message, and the terminal treats that as an instruction to go back to the beginning of the line; it then prints the rest of the message over top of the beginning of the message. It's a little like this:
-bash: /home/dir/path/.bash_profile
: No such file or directory
...but with the second line printed over the first, so it comes out as:
: No such file or directory_profile
I'm fiddling around with the hg log command to store the mercurial messages in a file, such as this (lines broken for readability):
hg log --template "Rev. {rev} from {date|rfc822date}:
\r\n--------------------------------------------
\r\n\t{desc|strip|fill68|tabindent}\r\n\r\n\r\n" > hglog.txt
The problem is with the fill68 filter which breaks lines after 68 chars. In theory, that is, because those line breaks are Unix line breaks and are not displayed in Notepad.
Is there a way to replace \n with \r\n, by piping hg log's output through another command maybe?
I wish to run this command in an automated VS post-build script, so I can't do this manually.
From Windows you can invoke the MORE command as specified here:
hg log ... | more /p > hglog.txt
I have cygwin installed on my Windows 7 box and I have been running into a problem where when I type a command it will occasionally be wrapped back onto the same line, deleting the bash prompt. Here is an example:
The command in question is command "201" (4 lines from the bottom). I included the others for context.
The text of the command I was typing was
git commit -m "Forced LF line endings."
(Note: I am posting this with mostly git commands, but the problem occurs with any command. I have not noticed a pattern yet.)
It jumped to the start of the line and started to overwrite my prompt.
When I push the up arrow (to view the history) the result is even weirder:
(Note the cursor is many characters past the end of the command.)
When I try to backspace the cursor from that position, I can only go back this far:
Then when I go up into the history from that backspaced line, I get this:
The command starts from the end of the text that is displayed. (This is consistent for the entire history) But when I go up in the history to the faulty git commit ... it displays as it did before with the overwritten text but when I go past it, it deletes a line of the prompt and displays the previous entry in the history the same way it was doing it before (a la image 2).
When I was creating my PS1 variable I has odd output like this, but I have since closed my brackets and things and don't think that is causing the issue. However if you would like to see my .bash_profile (that sets the PS1) feel free to see it on GitHub. It is really short.
I have tried searching for the issue and can only find a few cygwin email archives about the line wrapping in xterm, but no solutions.
PS: As I was pushing the latest .bash_profile, in order to link it, I ran into the problem again when I typed git add .bash_profile and hit enter, it ran the command but returned the cursor to the start of the same command instead of printing a new prompt.
Then when I as writing another commit line, it did the same as the first image, but it blacked out the rest of the line (It wrapped the line, but overwrote the entire line and not just the first few chars.)
See http://manpages.ubuntu.com/manpages/lucid/man1/bash.1.html#contenttoc26:
\[
begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
\]
end a sequence of non-printing characters
If you don't enclose non-printing characters (e.g. color sequences) in your prompt with these their length is counted as part of the prompt's length, eventually resulting in the symptoms that you describe.
It doesn't occur frequently to me. When it does, I just type in 'kill -WINCH $$' into the cygwin terminal and it fixes the problem. link to source
I'm trying to write a quick batch file. It will take the result of a command, put some extra text and quotes around it, and put that into a new file.The problem is that the result of the command I'm running includes a new line. Here's the command:
p4 changelists -m 1 -t //depot/...> %FILENAME%
The output of that p4 command has a newline at the end of it. The file I'm putting it into needs to have quotes surrounding the output of that command, but the fact that the command contains a newline in it means that the "closing quote" appears on a new line in the file, which doesn't work for what I'm doing.
I've tried writing the output of that command into a file and reading it back in, and also trying to run FINDSTR on a file containing the output, but I always seem to get back the stupid trailing whitespace. I've even tried inserting backspaces into the file, but that just put a backspace character into the file instead of actually executing a backspace...
Is there anything to be done about this?
I'm no perl wizard, but the following seems to work:
p4 changelists -m 1 -t //depot/...| perl -p -e "s/^/\042/;s/$/\042/"
Check out Strawberry Perl, which provides a Windows version of Perl.
I'm always looking at my Unix tools when solving problems like this, even under Windows. sed and gawk will also get you there, check out msysgit for a nice bundle of Unix tools that will run on Windows.