I once write a simple sed command like this
s/==/EQU/
while I run it in command line:
sed 's/==/EQU' filename
it works well, replace the '==' with 'EQU', but while I write the command to a script file named replace.sed, run it in this way:
sed -f replace.sed filename
there is a error, says that
sed: file replace.sed line 1: unknwon option to 's'
What I want to ask is that is there any problem with my script file replace.sed while it run in windows?
The unknown option is almost invariably a rogue character after the trailing / (which is missing from your command line version, by the way so it should complain about an unterminated command).
Have a look at you replace.sed again. You may have a funny character at the end, which could include the ' if you forgot to delete it, or even a CTRL-M DOS-style line ending, though CygWin seems to handle this okay - you haven't specified which sed you're using (that may help).
Okay, based on your edit, it looks like one of my scattergun of suggestions was right :-) You had CTRL-M at the end of the line because of the CR/LF line endings:
At the end of each line in the *.sed file, there was a 'CR\LF' pair, and that the problem, but you cannot see it by default, I use notepad to delete them manually and fix the problem. But I have not find a way to delete it automatically or do not contain the 'new-line' style while edit a new text file in windows.
You may want to get your hands on a more powerful editor like Notepad++ or gVim (my favourite) but, in fact, you do have a tool that can get rid of those characters :-) It's called sed.
sed 's/\015//g' replace.sed >replace2.sed
should get rid of all the CR characters from your file and give you a replace2.sed that you can use for your real job.
Related
I am trying to use sed to add some text, at every line end, for all .txt files in a directory. This is the exact command I use: find . -name "*.txt" -exec sed -i 's/$/:orig/' {} +
Expected:
https://pbs.twimg.com/media/EUr539_UMAAFqZM.jpg:orig
https://pbs.twimg.com/media/ENTrymcUwAAnd6_.jpg:orig
https://pbs.twimg.com/media/EIzzcrFUYAAgfUo.jpg:orig
That is also what I actually get when I run it in my laptop with Linux Mint 19.2. But when I try it in my Windows PC, running sed through Ubuntu in WSL, what I get is this:
https://pbs.twimg.com/media/EUr539_UMAAFqZM.jpg
:orig
https://pbs.twimg.com/media/ENTrymcUwAAnd6_.jpg
:orig
https://pbs.twimg.com/media/EIzzcrFUYAAgfUo.jpg:orig
If I cat the files in question while still in the Ubuntu terminal, what's displayed is more like this (there's some weird whitespace that makes it look like columns in SO, but generally they all look pretty chaotic):
:orig://pbs.twimg.com/media/EUr539_UMAAFqZM.jpg :orig://pbs.twimg.com/media/ENTrymcUwAAnd6_.jpg https://pbs.twimg.com/media/EIzzcrFUYAAgfUo.jpg:orig
I understand Windows and Linux text is formatted differently and that line ends in particular are problematic, though I am uncertain if that is of any importance here.
Can anyone shed light on this behavior? How can I get the command to behave consistently?
The problem is that your files end in CRLF but the WSL sed uses just LF and the line end. You can get around this with a three step process, if you know it's a CRLF-style file:
get rid of the CR;
do your change;
put the CR back.
That would go something like: sed -i -e 's/\r$//' -e 's/$/:orig/' -e 's/$/\r/'.
However, that this won't work on UNIX-style files since the first substitution will do nothing but the third will put a CR character at the end of each line, even though it wasn't there originally. If you want something that will work on both types of files, this should do it:
sed -E 's/(\r)?$/:orig\1/'
This captures the optional CR at the end of the line and puts it back in the substitution (if it's not in the original line, it won't put it back).
I am using Windows and MobaXterm.
I created a .bash_profile file in the ~ directory and the following line
alias sbp="source ~/.bash_profile"
is the only code in that file.
However, when I was trying to do sbp, I got an error.
This works on my Mac and it used to work on my old Windows computer (but that one has some water damage so it broke down). Why does this not work now?
Thanks in advance!
From the way that error message is garbled I'm pretty sure that the .bash_profile file you created has DOS/Windows-style line endings, consisting of a carriage return character followed by a newline character. Unix tools expect unix-style line endings consisting of just a newline; if they see DOS/Windows-style endings, they'll treat the carriage return as part of the content of the line. In this case, bash will treat the carriage return as part of the alias definition, and therefore part of the filename to filename to source. Try running alias sbp | cat -vt to print the alias with invisible characters shown; my guess is it'll print alias sbp='source ~/.bash_profile^M' (where the ^M is cat -vt's way of representing the carriage return).
Solution: convert the file to unix format, and either switch to a text editor that knows how to save in unix format, or change your settings in the current editor to do it. For conversion, there are a number of semi-standard tools like dos2unix and fromdos. If you don't have any of those, this answer has some other options.
BTW, the reason the error message is garbled is that the CR gets printed as part of the error message, and the terminal treats that as an instruction to go back to the beginning of the line; it then prints the rest of the message over top of the beginning of the message. It's a little like this:
-bash: /home/dir/path/.bash_profile
: No such file or directory
...but with the second line printed over the first, so it comes out as:
: No such file or directory_profile
I have following snippet:
host="https://example.com"
port="80"
url="${host}:${port}"
echo $url
the output is:
:80ps://example.com
How can I escape the colon here. I also tried:
url="${host}\:${port}"
but it did not work.
Expected output is:
https://example.com:80
You've most likely run into what I call the Linefeed-Limbo.
If I copy the code you provided from StackOverflow and run it on my machine (bash version 4.4.19(1)), then it outputs correctly
user#host:~$ cat script.sh
host="https://example.com"
port="80"
url="${host}:${port}"
echo $url
user#host:~$ bash script.sh
https://example.com:80
What is Linefeed-Limbo?
Different operating systems use different ASCII symbols to represent when a new line occurs in a text, such as a script. This Wikipedia article gives a good introduction.
As you can see, Unix and Unix-like systems use the single character \n, also called a "Line Feed". Windows, as well as other systems, use \r\n, so a "carriage return" followed by a "line feed".
What happens now is when you write a script on Windows on an editor such as notepad, what you write is host="example.com"\r\n. When you copy this file into Linux, Linux interprets the \r as if it were part of the script, since only \n is considered a new line. And indeed, when I change my newline style to DOS-style, I get the exact output you get.
How can I fix this?
You have several options to fix this issue.
Converting the script (with dos2unix)
Since all you need to do is replacing every instance of \r\n with \n, you could use any text-editing software you want. However, if you like simple solutions, then dos2unix (and its sister unix2dos) might be what you looking for:
user#host:~$ dos2unix script.sh
dos2unix: converting file script.sh to Unix format...
That's it. Run your file now and you will see it behaves well.
Encoding the source-file correctly
By using a more advanced text editor such as Notepad++, you can define which style of newline you would like to use.
By changing the newline-type to whichever system you intend to run your script on, you will not run into any problems like this anymore.
Bonus round: Why does it output :80ps://example.com?
To understand why your output is like this, you have to look at what your script is doing, and what \r means.
Try thinking of your terminal as an old-fashioned typewriter. Returning the carriage means you start writing on the left again. Making a "new line" means sliding the paper. These two things are seperate, and I think that's why some systems decided to use these two characters as a logical "new line".
But I digress. Let's look at the first line, host="https://example.com"\r.
What this means when printed is "Print https://example.com, then put the carriage back at the start". When you then print :80\r, it doesn't start after ".com", it starts at the beginning of the line, because that's where you (unknowingly) told the cursor to go. it then overwites the first few characters, resulting in ":80ps://example.com" to be written. Keep in mind that after 80, you again placed a carriage return symbol, so any new text you would have written ends up overwriting the beginning again.
It works for me, try to remove carriage returns in variables and then try.
new_host=$(echo "$host" | tr -d '\r')
new_port=$(echo "$port" | tr -d '\r')
new_url="${new_host}:${new_port}"
I'm trying to write a quick batch file. It will take the result of a command, put some extra text and quotes around it, and put that into a new file.The problem is that the result of the command I'm running includes a new line. Here's the command:
p4 changelists -m 1 -t //depot/...> %FILENAME%
The output of that p4 command has a newline at the end of it. The file I'm putting it into needs to have quotes surrounding the output of that command, but the fact that the command contains a newline in it means that the "closing quote" appears on a new line in the file, which doesn't work for what I'm doing.
I've tried writing the output of that command into a file and reading it back in, and also trying to run FINDSTR on a file containing the output, but I always seem to get back the stupid trailing whitespace. I've even tried inserting backspaces into the file, but that just put a backspace character into the file instead of actually executing a backspace...
Is there anything to be done about this?
I'm no perl wizard, but the following seems to work:
p4 changelists -m 1 -t //depot/...| perl -p -e "s/^/\042/;s/$/\042/"
Check out Strawberry Perl, which provides a Windows version of Perl.
I'm always looking at my Unix tools when solving problems like this, even under Windows. sed and gawk will also get you there, check out msysgit for a nice bundle of Unix tools that will run on Windows.
I am using grep to take all the four letter words out of a dictionary text file and place them into a new text file.
This command should work with Unix however on windows it does not.
I need one word per line, on windows it gives me all the words but all piled together without spaces.
This is the grep command I'm using:
grep "^[a-z]\{4\}$" dictionaryfilename > outputfilename
I believe it's something to do with a difference in newline characters between Unix and windows?
Anyway I'm not sure how to make a fix for windows with this could someone please help.
Thanks a lot :)
you probably have a UNIX-formatted textfile (newlines without carriage returns), which looks like one big line in Windows; grep just deals in whatever the system says is 'a line', so it has little to do with the problem.
Try converting the file from LF to CRLF and see if you get better results.