I am using grep to take all the four letter words out of a dictionary text file and place them into a new text file.
This command should work with Unix however on windows it does not.
I need one word per line, on windows it gives me all the words but all piled together without spaces.
This is the grep command I'm using:
grep "^[a-z]\{4\}$" dictionaryfilename > outputfilename
I believe it's something to do with a difference in newline characters between Unix and windows?
Anyway I'm not sure how to make a fix for windows with this could someone please help.
Thanks a lot :)
you probably have a UNIX-formatted textfile (newlines without carriage returns), which looks like one big line in Windows; grep just deals in whatever the system says is 'a line', so it has little to do with the problem.
Try converting the file from LF to CRLF and see if you get better results.
Related
I have following snippet:
host="https://example.com"
port="80"
url="${host}:${port}"
echo $url
the output is:
:80ps://example.com
How can I escape the colon here. I also tried:
url="${host}\:${port}"
but it did not work.
Expected output is:
https://example.com:80
You've most likely run into what I call the Linefeed-Limbo.
If I copy the code you provided from StackOverflow and run it on my machine (bash version 4.4.19(1)), then it outputs correctly
user#host:~$ cat script.sh
host="https://example.com"
port="80"
url="${host}:${port}"
echo $url
user#host:~$ bash script.sh
https://example.com:80
What is Linefeed-Limbo?
Different operating systems use different ASCII symbols to represent when a new line occurs in a text, such as a script. This Wikipedia article gives a good introduction.
As you can see, Unix and Unix-like systems use the single character \n, also called a "Line Feed". Windows, as well as other systems, use \r\n, so a "carriage return" followed by a "line feed".
What happens now is when you write a script on Windows on an editor such as notepad, what you write is host="example.com"\r\n. When you copy this file into Linux, Linux interprets the \r as if it were part of the script, since only \n is considered a new line. And indeed, when I change my newline style to DOS-style, I get the exact output you get.
How can I fix this?
You have several options to fix this issue.
Converting the script (with dos2unix)
Since all you need to do is replacing every instance of \r\n with \n, you could use any text-editing software you want. However, if you like simple solutions, then dos2unix (and its sister unix2dos) might be what you looking for:
user#host:~$ dos2unix script.sh
dos2unix: converting file script.sh to Unix format...
That's it. Run your file now and you will see it behaves well.
Encoding the source-file correctly
By using a more advanced text editor such as Notepad++, you can define which style of newline you would like to use.
By changing the newline-type to whichever system you intend to run your script on, you will not run into any problems like this anymore.
Bonus round: Why does it output :80ps://example.com?
To understand why your output is like this, you have to look at what your script is doing, and what \r means.
Try thinking of your terminal as an old-fashioned typewriter. Returning the carriage means you start writing on the left again. Making a "new line" means sliding the paper. These two things are seperate, and I think that's why some systems decided to use these two characters as a logical "new line".
But I digress. Let's look at the first line, host="https://example.com"\r.
What this means when printed is "Print https://example.com, then put the carriage back at the start". When you then print :80\r, it doesn't start after ".com", it starts at the beginning of the line, because that's where you (unknowingly) told the cursor to go. it then overwites the first few characters, resulting in ":80ps://example.com" to be written. Keep in mind that after 80, you again placed a carriage return symbol, so any new text you would have written ends up overwriting the beginning again.
It works for me, try to remove carriage returns in variables and then try.
new_host=$(echo "$host" | tr -d '\r')
new_port=$(echo "$port" | tr -d '\r')
new_url="${new_host}:${new_port}"
I'm using Ubuntu Bash within Windows 10 and I have a text document with:
{u'osidjfoij23': 3894798, u'oisjdao':234567, u'oaijsdofj': 984759}
using tr, in terminal I change my output to
'osidjfoij23': 3894798,
'oisjdao':234567,
'oaijsdofj': 984759}
when opening the same document via notepad.exe, the newline "\n" added from tr doesn't register and all the data gets presented as a paragraph.
I know this is because bash and notepad have different encodings for their documents, is there a way to make these work together or an alternative I can use for notepad?
You can use unix2dos to convert a file to Windows line endings. Linux programs handle Windows line endings fairly well, so this shouldn't break anything (especially if that's JSON as it appears to be).
I have an output file (namely a log from screen) containing several control characters. Inside the screen, I have programs running that use control characters to refresh certain lines (examples would be top or anything printing progress bars).
I would like to output a tail of this file using PHP. If I simply read in that file and echo its contents (either using PHP functions or through calling tail, the output is messy and much more than these last lines as it also includes things that have been overwritten. If I instead run tail in the command line, it returns just what I want because the terminal evaluates the control characters.
So my question is: Is there a way to evaluate the control characters, getting the output that a terminal would show me, in a way that I could then use elsewhere (e.g., write to a file)?
#5gon12eder's answer got rid of some control characters (thanks for that!) but it did not handle the carriage return part that was even more important to me.
I figured out that I could just delete anything from the beginning of a line to the last carriage return inside that line and simply keep everything after that, so here is my sed command accomplishing that:
sed 's/^.*\r\([^\r]\+\)\r\?$/\1\r/g'
The output can then be further cleaned using #5gon12eder's answer:
cat screenlog.0 | sed 's/^.*\r\([^\r]\+\)\r\?$/\1\r/g' | sed 's,\x1B\[[0-9?;]*[a-zA-Z],,g'
Combined, this looks exactly like I wanted.
I'm not sure what you mean by “evaluating” the control characters but you could remove them easily.
Here is an example using sed but if you are already using PHP, its internal regex processing functionality seems more appropriate. The command
$ sed 's,\x1B\[[0-9?;]*[a-zA-Z],,g' file.dat
will dump the contents of file.dat to standard output with all ANSI escape sequences removed. (And I'm pretty sure that nothing else is removed except if your file contains invalid escape sequences in which case the operation is ill-defined anyway.)
Here is a little demo:
$ echo -e "This is\033[31m a \033[umessy \033[46mstring.\033[0m" > file.dat
$ cat file.dat
# The output of the above command is not shown to protect small children
# that might be browsing this site.
$ reset # your terminal
$ sed 's,\x1B\[[0-9?;]*[a-zA-Z],,g' file.dat
This is a messy string.
The less program has some more advanced logic built in to selectively replace some escape sequences. Read the man page for the relevant options.
I have a .txt (Mac OS X Snow Leopard) file that has a lot of text. At the end of a paragraph, there is a hard return that moves the next paragraph onto another line. This is causing some issues with what I am wanting to do to get the content into my db, so I am wondering if there is anyway I can remove the hard returns? Is there some sort of script I can run? I am really hoping I don't have to go through and manually take the hard returns out.
To recap, here is what it looks like now:
This is some text. Text is what this is.
And then this is the next paragraph that is on a different line.
And this is what I would like to get to:
This is some text. Text is what this is. And then this is the next paragraph that is on a different line.
For all several thousand lines in my .txt file.
Thanks!
EDIT:
The text I am dealing with in my txt file is actually HTML:
<span class="text">1 </span> THis is where my text is<br/>
And when I run the cat command in terminal like mentioned below, only the first is there. Everything else is missing...
In a terminal:
cat myfile.txt | tr -d '\r' > file2.txt
There's probably a more efficient way to do this, since the "tr -d '\r'" is the active ingredient, but that's the idea.
I normally just use an editor with good Regular Expression support. TextWrangler is great.
An end of line in TextWrangler is \r, so to remove it, just search for \r and replace it with a space. TBH, I always wondered how it handles CRLF-encoded files, but somehow it works.
I believe you can do this with Applescript. Unfortunately I'm not familiar with it however the following should help you to acomplish this (it's for a different problem but it will lead you in the direction you need to go): http://macscripter.net/viewtopic.php?id=18762
Alternatively if you didn't want to do this with Applescript and have Excel installed (or access to a machine with it) then the following should help: http://www.mrexcel.com/forum/showthread.php?t=474054
In Linux terminal cat file.txt | tr -d "\r\n" | > new file.txt will do. Modify \r\n part to remove desired charters.
I once write a simple sed command like this
s/==/EQU/
while I run it in command line:
sed 's/==/EQU' filename
it works well, replace the '==' with 'EQU', but while I write the command to a script file named replace.sed, run it in this way:
sed -f replace.sed filename
there is a error, says that
sed: file replace.sed line 1: unknwon option to 's'
What I want to ask is that is there any problem with my script file replace.sed while it run in windows?
The unknown option is almost invariably a rogue character after the trailing / (which is missing from your command line version, by the way so it should complain about an unterminated command).
Have a look at you replace.sed again. You may have a funny character at the end, which could include the ' if you forgot to delete it, or even a CTRL-M DOS-style line ending, though CygWin seems to handle this okay - you haven't specified which sed you're using (that may help).
Okay, based on your edit, it looks like one of my scattergun of suggestions was right :-) You had CTRL-M at the end of the line because of the CR/LF line endings:
At the end of each line in the *.sed file, there was a 'CR\LF' pair, and that the problem, but you cannot see it by default, I use notepad to delete them manually and fix the problem. But I have not find a way to delete it automatically or do not contain the 'new-line' style while edit a new text file in windows.
You may want to get your hands on a more powerful editor like Notepad++ or gVim (my favourite) but, in fact, you do have a tool that can get rid of those characters :-) It's called sed.
sed 's/\015//g' replace.sed >replace2.sed
should get rid of all the CR characters from your file and give you a replace2.sed that you can use for your real job.