I have a text file with a list of Mercurial repositories in it, in the form:
IDE
Install
InstallShield
I'm writing a bash script to clone/pull/update all the repositories based on the text file. Right now I'm just echoing before I do any actual cloning. If I do:
while read line; do
echo "hg clone" ${MASTER_HG}/${line};
done < Repos.txt
The output is as expected:
hg clone /media/fs02/IDE
hg clone /media/fs02/Install
hg clone /media/fs02/InstallShield
However, if I do:
while read line; do
echo "hg clone" ${MASTER_HG}/${line} ${REPOROOT}/${line};
done < Repos.txt
The output is:
/var/hg/repos/IDE02/IDE
/var/hg/repos/Installnstall
/var/hg/repos/InstallShieldShield
It seems to be replacing the beginning of the string with the end of the string. Is there some kind of character overflow or something going on? My apologies if this is a dumb question, but I'm a relative noob for bash.
Your file has DOS line endings; the \r at the end of $line causes the cursor to return to the beginning of the line, which only affects your output when $line is not the last thing being printed before the newline. You should remove them with something like dos2unix.
You can use something similar to Perl's chomp command to remove a trailing carriage return, if one is present:
# $'\r' is bash-only, but easy to type. For POSIX shell, you'll need to find
# someway of entering ASCII character 13; perhaps Control-V Control-M
line=${line%$'\r'}
Useful if, for whatever reason, you can't (or don't want to) fix the input before reading it.
From the looks of it, ${REPOROOT} might already include ${line}, try echoing ${REPOROOT} by itself and see what you get.
Related
I'm trying to write simple shell script that clones a github repo then CDs into the folder. I have
#!/bin/bash
# $1 is github repo url
url=$1
reponame=$(basename $url)
git clone $url
echo $reponame
cd $reponame
Now when I run
bash deploy_app.sh https://github.com/username/reponame
It echoes the right $reponame but the next line returns this "cd: $'reponame\r\r\r\r': No such file or directory"
I have no idea where the "\r\r\r\r" part is coming from. It's not in the repo name and it's not part of $reponame when I just echo it. Any idea why this happens??
The first thing I would be looking at is your script itself, by doing something like:
od -xcb deploy_app.sh
and looking for any carriage return characters (CR, ^M, \r, hex 0d, or octal 015) at the end of the cd line (or any line really).
That's because, if you have CR characters at the end, it will cause that problem (the ^M characters below were entered with CTRL-VCTRL-M):
pax:~> cd blah^M^M^M^M
-bash: cd: $'blah\r\r\r\r': No such file or directory
If there are CR characters in your file, you probably want to remove them, and check your line ending settings for git (assuming that file is in a repo) - this is a problem commonly caused by incorrect settings if you're updating your repo from both Windows and Unix.
If that file is not being "damaged" by git, you'll need to discover what else is putting those characters in.
And, as an aside, you wouldn't actually see whether the reponame had those characters with an echo, since echoing them would result in the actual characters, four "go to the start of the line" characters, then a "go to the next line" character. Hence there's no way to tell the difference just by looking, without pushing the output through a command like od.
I am not sure either, but try changing $reponame to "${reponame}" (with the double quotes) and see if it still happens.
I am having issues with my known methods of generating user input prompts:
read -p "Input something: " variabile
This causes issues if attempting to use the arrow keys, it echoes the ANSI code for each arrow key stroke
read -e -p "Input something: " variable
This fixes the arrow keys issue but when reaching the width of the terminal, text input doesn't continue on a newline but on the same line, overwriting (visually) the existing input
echo -n "Input something: "; read -e variable
This apparently fixes both formerly described issues... until I found that typing something then hitting backspace overwrites the prompt and also when the input is longer, from the second newline of input the visual overwriting manifests again.
So is there a good way of producing prompts without the above issues?
UPDATE
After re-checking, I now know what's causing the input overwrite for read -e -p
I am using these variables for highlighting text for the read prompt:
highlight=$(echo -e "\e[1;97m")
clear=$(echo -e "\e[0m")
read -e -p "Input$highlight something$clear: " variable
This is the only way I could make the highlighting work inside read prompt (assigning escape sequences to the variables doesn't work, I need to echo them like I did) but they also seem to cause the input overwrite issue.
As dimo414 mentions, readline thinks the prompt is longer than it is. It counts every character in the terminal escape sequence in computing the length. You can see how long it thinks the escape sequence is as follows
echo ${#highlight}
In the bash PS1 prompt, surrounding such an escape sequence with "\[" and "\]" instructs readline to ignore everything between when calculating current line length, but these are not the right escapes for the bash read built-in.
The escapes for read are $'\001' and $'\002', as mentioned in BashFAQ, but in my experience, you need the -e option on read, as well. The brute force way to do what you want would be:
read -e -p "Input "$'\001'"${highlight}"$'\002'something$'\001'"${clear}"$'\002'": "
You should use tput rather than hard-coded escape sequences, for the sake of terminal independence. Read man 5 termcap.
See my dotfiles for elegant bash functions to do the begin/end quoting above for you.
The shell keeps track of how long it thinks the prompt is, in order to know where the user's input starts and stops. Unfortunately when you print color escape codes in a prompt you throw of Bash's counting, since it expects the escape characters to take up space in the terminal.
To avoid that, you just need to wrap all color sequences in \[ and \], which tells your shell the enclosed characters are non-printing, and should not be counted.
For example, your highlight variable should be:
highlight=$(echo -e "\[\e[1;97m\]")
Personally, I use the color and pcolor functions from my Prompt.gem project, which handles the proper escaping and would make your command much easier to read:
read -e -p "Input $(pcolor DEFAULT BOLD)something$(pcolor): " variable
Using cygwin, I have
currentFold="`dirname $0`"
echo ${currentFold}...
This outputs ...gdrive/c/ instead of /cygdrive/c/...
Why is that ?
Your script is stored in DOS format, with a carriage return followed by linefeed (sometimes written "\r\n") at the end of each line; unix uses just a linefeed ("\n") at the end of lines, and so bash is mistaking the carriage return for part of the command. When it sees
currentFold="`dirname $0`"\r
it dutifully sets currentFold to "/cygdrive/c/\r", and when it sees
echo ${currentFold}...\r
it prints "/cygdrive/c/\r...\r". The final carriage return doesn't really matter, but the one in the middle means that the "..." gets printed on top of the "/cy", and you wind up with "...gdrive/c/".
Solution: convert the script to unix format; I believe you'll have the dos2unix command available for this, but you might have to look around for alternatives. In a pinch, you can use
perl -pi -e 's/\r\n?/\n/g' /path/to/script
(see http://www.commandlinefu.com/commands/view/5088/convert-files-from-dos-line-endings-to-unix-line-endings). Then switch to a text editor that saves in unix format rather than DOS.
I would like to add to Gordon Davisson's anwer.
I am also using Cygwin. In my case this happened because my Git for Windows was configured to Checkout Windows-style, commint Unix style line endings.
This is the default option, and was breaking all my cloned shell scripts.
I reran my git setup and changed to Checkout as-is, commit Unix-style line endings which prevented the problem from happening at all.
This question already has answers here:
Are shell scripts sensitive to encoding and line endings?
(14 answers)
Closed 3 years ago.
I have a problem with a character. I think it's a conversion problem between dos and unix.
I have a variable that is a float value.
When I print it with the echo command i get:
0.495959
But when I try to make an operation on that value with the bc command (I am not sure how to write the bc command).
echo $mean *1000 |bc
I get:
(standard_in) 1 : illegal character: ^M
I already use the dos2unix command on my .sh file.
I think it's because my variable have the ^M character (not printed with the echo command)
How can i eliminate this error?
I don't have Cygwin handy, but in regular Bash, you can use the tr -d command to strip out specified characters, and you can use the $'...' notation to specify weird characters in a command-line argument (it's like a normal single-quoted string, except that it supports C/Java/Perl/etc.-like escape sequences). So, this:
echo "$mean" * 1000 | tr -d $'\r' | bc
will strip out carriage-returns on the way from echo to bc.
You might actually want to run this:
mean=$(echo "$mean" | tr -d $'\r')
which will modify $mean to strip out any carriage-returns inside, and then you won't have to worry about it in later commands that use it.
(Though it's also worth taking a look at the code that sets $mean to begin with. How does $mean end up having a carriage-return in it, anyway? Maybe you can fix that.)
This works:
${mean/^M/}
You can get ^M by typing Ctrl-V followed by Ctrl-M. Or, alternatively:
${mean/$(printf "\r")/}
The benefit of this method compared to #ruakh's is that here you are using bash built-ins only. The first will be faster as the second will run inside a subshell.
If you just want to "unixize" $mean:
mean="${mean/^M/}"
Edit: There's yet another way:
${mean/$'\r'/}
Running Windows stuff in cygwin has one nasty side-effect as you found out - capturing the output of Windows programs in a cygwin bash variable will also capture the CR output by the program.
Judicious use of d2u avoids the issue - for example,
runtime="`mediainfo --Inform='Video;%Duration%' ${movie} | d2u`"
(Without the d2u, ${runtime} would have a CR tacked on the end, which causes the problem you saw when you feed it to 'bc' for example.)
Maybe you should just save your script in UNIX format instead of DOS.
Try this:
echo `echo $mean` *1000 |bc
If echo really isn't printing it, it should work.
^M is a carriage return character that is used in Windows along with newline (\n) character to indicate next line. However, it is not how it is done in UNIX world, and so bash doesn't treat at as a special character and it breaks the syntax. What you need to do is to remove that character using one of many methods. dos2unix tool can come handy, for example.
As others have pointed out, this is a Windows line ending issue. There are many ways to fix the problem, but the question is why did this happen in the first place.
I can see this happening in several places:
This is a WINDOWS environment variable that was set when Cygwin started up. Sometimes these variables get a CRLF on the end of them. You mentioned this was a particular issue with this one variable, but you didn't specify where it was set.
You edited this file using a Windows text editor like Notepad or Winpad.
Never use a text editor to edit a program. Use a program editor. If you like VI, download VIM which is available on Windows and comes on Cygwin (and all other Unix-based platforms). If VIM isn't for you, try the more graphically based Notepad++. Both of these editors handle end of line issues, and can create scripts with Unix line endings in Windows or files with Windows line endings in Cygwin.
If you use VIM, you can do the following to change line endings and to set them:
To see the line ending in the current file, type :set ff? while in command mode.
To set the line ending for Unix, type :set ff=unix while in command mode.
To set the line ending for Windows, type :set ff=dos while in command mode.
If you use Notepad++
You can go into the Edit-->EOL Conversion menu item and see what your current line ending setting (it's the one not highlighted) and change it.
To have Notepad++ use Unix line endings as the default, go into the Settings-->Preferences menu item. In the Dialog box, select the New Document/Default Directory tab. In the Format section which is part of the New Document section, select the line ending you want. WARNING: Do not select Mac as an option. That doesn't even work on Macs. If you have a Mac, select Unix.
I am trying to write a script which reads the hostname of the remote machine and then uses that result in following commands. However, the variable seems to be corrupted or something.
Here is an example of what is happening:
sbaker#eye004:~/workspace/fire_trunk$ REMOTE_HOSTNAME="`ssh $REMOTE 'hostname'`"
sbaker#eye004:~/workspace/fire_trunk$ echo "before $REMOTE_HOSTNAME after"
prints (note the prefixing whitespace):
" after sbaker-PC"
sbaker#eye004:~/workspace/fire_trunk$ echo $REMOTE_HOSTNAME
prints:
"sbaker-PC"
I am wondering why the variable seems unusable and doing weird things (if the after word is longer than the before word, it writes over the top of the characters).
I would expect the first echo to print: "before sbaker-PC after".
Am I just doing something stupid here?
I am using bash on ubuntu 11.
If you put it through od -c you'll see that it's actually returning sbaker-PC\r. The CR at the end is causing it to return the cursor to the first column before echoing the rest of the text, obscuring the "before". As for why it's adding \r, perhaps the file giving the hostname on the other side was saved with DOS line endings (CRLF) instead of *nix line endings (LF).