I am having trouble with an old sed script that I dug out of the archives that worked fine years ago, but now is not.
The old script was complex, having multiple substitutions between the curly braces, but even really dumbed down to a single substitution, I can not get it to work now.
I am attempting to use it with home-brew gnu-sed on Mac OS X 10.11 El Capitan, but had the same trouble on debian as well, so it seems like a gnu sed specific issue. It works with regular OS X /usr/bin/sed, but not with gnu sed which I would prefer to use for the sake of portability.
Does gnu sed need different flags to allow the use of the curly braces or something else along those lines? in all cases below the sed command links to /usr/local/bin/gsed on the OS X machine. I suppose I could just use OS X sed, but that wouldn't help me on a linux box should I need to use the script there.
Thanks.
The error I get is
sed: file script_file line 3: extra characters after command
Here is my simple two line data file.
[[aaaaaaaaaaaaaaaa]]
aaaaaaaaaaaaaaaaaa
Here is the correct output assuming the script ran successfully.
[[bbbbbbbbbbbbbbbb]]
aaaaaaaaaaaaaaaaaa
Here is the sed script that does not work when stored in a file.
/\[\[.*\]\]/{
s/a/b/g
}
Here is the command I am using to run it.
sed -f script_file data_file
If I eliminate the curly braces in the file it works, but then I can't apply multiple substitutions to the same pattern space. Here is a version of the script file that works without a problem.
/\[\[.*\]\]/s/a/b/g
if I put everything on the command line including the curly braces, it also works.
sed '/\[\[.*\]\]/{s/a/b/g}' data_file
This one also works with stdin data as well.
What do I need to do to get the multi-line curly brace block script file version to work properly with gnu sed?
Related
I am currently running sed with variable line input to remove a set number of lines from a file.
sed -i '1,'$SomeVariable'd' somefile.extention
Now, this works as intended if I enter it in cygwin. However, if I run it via a shell script, it does not delete the lines as intended. It still picks up their contents, but does not delete them afterwards.
While looking for a solution, I came across Why does sed behave differently in a shell script?
This does not answer my question as I have verified that bash is running sed and not gsed.
I am trying to batch fix some .scc files using vim and need to pass a substitution sting to it.
I'm trying to use vim -E to pass this line ":s/\r/\r/g"
I've tried embedding individual characters as a variable within the quotes but the \r messing it up.
Any advice would be appreciated.
Guessing at what you really want to do as it doesn't sound like a one-off change. The following will remove all \rs.
NOTE it does an inplace edit - no need to redirect etc.
perl -i.bak -p -e 's/\r+//g' filename(s)...
If you want to replace it with a single \r do
perl -i.bak -p -e 's/\r+/\r/g' filename(s)...
Perl will keep a backup but so should you.
It's not using vim but then perhaps you shouldn't want to.
In Vim, you must use \n in the search part and \r in the replacement part. But replacing \n with \r sounds a lot like doing nothing to me.
Anyway, launching Vim just for a substitution is probably a bit too much. You could use sed:
$ sed -i.orig 's/foo/bar/' foo.txt
Read $ man sed for more info.
got it. ':s/\\r/\\r/g'
I just needed the right level of quotation. The issue as I touched on earlier is that we get files that include PC double line breaks presumable made on a mac and the PC's can't read them properly.
So using sed to s/\r/\r\n/g wasn't really working for some reason, and maybe I could explore it more. But I think passing these to vim will help. I must use vim to send :set format=dos to the file.
Thanks all.
This question already has answers here:
Are shell scripts sensitive to encoding and line endings?
(14 answers)
Closed 3 years ago.
I have a problem with a character. I think it's a conversion problem between dos and unix.
I have a variable that is a float value.
When I print it with the echo command i get:
0.495959
But when I try to make an operation on that value with the bc command (I am not sure how to write the bc command).
echo $mean *1000 |bc
I get:
(standard_in) 1 : illegal character: ^M
I already use the dos2unix command on my .sh file.
I think it's because my variable have the ^M character (not printed with the echo command)
How can i eliminate this error?
I don't have Cygwin handy, but in regular Bash, you can use the tr -d command to strip out specified characters, and you can use the $'...' notation to specify weird characters in a command-line argument (it's like a normal single-quoted string, except that it supports C/Java/Perl/etc.-like escape sequences). So, this:
echo "$mean" * 1000 | tr -d $'\r' | bc
will strip out carriage-returns on the way from echo to bc.
You might actually want to run this:
mean=$(echo "$mean" | tr -d $'\r')
which will modify $mean to strip out any carriage-returns inside, and then you won't have to worry about it in later commands that use it.
(Though it's also worth taking a look at the code that sets $mean to begin with. How does $mean end up having a carriage-return in it, anyway? Maybe you can fix that.)
This works:
${mean/^M/}
You can get ^M by typing Ctrl-V followed by Ctrl-M. Or, alternatively:
${mean/$(printf "\r")/}
The benefit of this method compared to #ruakh's is that here you are using bash built-ins only. The first will be faster as the second will run inside a subshell.
If you just want to "unixize" $mean:
mean="${mean/^M/}"
Edit: There's yet another way:
${mean/$'\r'/}
Running Windows stuff in cygwin has one nasty side-effect as you found out - capturing the output of Windows programs in a cygwin bash variable will also capture the CR output by the program.
Judicious use of d2u avoids the issue - for example,
runtime="`mediainfo --Inform='Video;%Duration%' ${movie} | d2u`"
(Without the d2u, ${runtime} would have a CR tacked on the end, which causes the problem you saw when you feed it to 'bc' for example.)
Maybe you should just save your script in UNIX format instead of DOS.
Try this:
echo `echo $mean` *1000 |bc
If echo really isn't printing it, it should work.
^M is a carriage return character that is used in Windows along with newline (\n) character to indicate next line. However, it is not how it is done in UNIX world, and so bash doesn't treat at as a special character and it breaks the syntax. What you need to do is to remove that character using one of many methods. dos2unix tool can come handy, for example.
As others have pointed out, this is a Windows line ending issue. There are many ways to fix the problem, but the question is why did this happen in the first place.
I can see this happening in several places:
This is a WINDOWS environment variable that was set when Cygwin started up. Sometimes these variables get a CRLF on the end of them. You mentioned this was a particular issue with this one variable, but you didn't specify where it was set.
You edited this file using a Windows text editor like Notepad or Winpad.
Never use a text editor to edit a program. Use a program editor. If you like VI, download VIM which is available on Windows and comes on Cygwin (and all other Unix-based platforms). If VIM isn't for you, try the more graphically based Notepad++. Both of these editors handle end of line issues, and can create scripts with Unix line endings in Windows or files with Windows line endings in Cygwin.
If you use VIM, you can do the following to change line endings and to set them:
To see the line ending in the current file, type :set ff? while in command mode.
To set the line ending for Unix, type :set ff=unix while in command mode.
To set the line ending for Windows, type :set ff=dos while in command mode.
If you use Notepad++
You can go into the Edit-->EOL Conversion menu item and see what your current line ending setting (it's the one not highlighted) and change it.
To have Notepad++ use Unix line endings as the default, go into the Settings-->Preferences menu item. In the Dialog box, select the New Document/Default Directory tab. In the Format section which is part of the New Document section, select the line ending you want. WARNING: Do not select Mac as an option. That doesn't even work on Macs. If you have a Mac, select Unix.
I have a number of files (more than a hundred) that I want to process using Vim. A sample of the files’ contents is as follows:
xyz.csv /home/user/mydocs/abc.txt
/home/user/waves/wav.wav , user_wav.wav
I want this to be replaced by:
xyz.csv /var/lib/mydir/abc.txt
/var/sounds/wav.wav , wav.wav
In each of the files, the changes I need to make are the same. My questions are:
Can I use Vim search and replace functionality by calling it from within a Bash script?
If so, how do I go about it?
P.S. I have searched StackOverflow for similar questions and found some answers using ex scripts, etc. I want to know how I can call an ex script from within a bash script.
While vim is quite powerful, this is not something I would normally use vim for. It can be done using a combination of common command line utilities instead.
I've assumed that the blank line in your example above is actually blank and does not contain spaces or any other whitespace characters. You can use the following to do what you want.
sed -e "s,/home/user/mydocs,/var/lib/mydir," -e "s,/home/user/waves,/var/sounds," -e "/^$/d" file1
You can use that command together with find and a for loop to do this for a bunch of files:
for file in `find . -maxdepth 1 -type f`
do
sed -e "s,/home/user/mydocs,/var/lib/mydir," -e "s,/home/user/waves,/var/sounds," -e "/^$/d" $file
done
In the for loop, the find command above limits the output to all files in the current directory (including dot files), assigning each line from the output of find to the file variable and then running the sed command posted earlier to transform the file the way you want it to be transformed.
This is how you'd invoke an ed script from bash:
ed filename <<END
/^$/d
%s|/home/user/mydocs|/var/lib/mydir|
%s|/home/user/waves|/var/sounds|
%s|, user_|, |
w
q
END
To answer with vim, you can do
vim -e 'bufdo!%s:\(xyz.csv \)/home/user/mydocs/\(abc.txt\n\)\n.*:\1/var/lib/mydir/\2/var/sounds/wav.wav , wav.wav:' -e 'xa' FILES
Note, I had assumed, that the second line is statically replaced, as it had looked like in the question.
If you don't like writing long lines in your script, you can create a file like:
s/FOO/BAR/
" several replacement and other commands
w " write the file
bd " if you want to
Then do:
vim -e "buffdo!source /your_scriptfile" -e "x" FILES
HTH
If all the editing consists in a series of substitutions, the most
idiomatic way of accomplishing it using Vim would be the following.
Open all the target files at once:
vim *.txt
Run the substitution commands on the loaded files:
:argdo %s#/home/user/mydocs#/var/lib/mydir#
:argdo %s#/home/user/waves#/var/sounds#
:argdo %s#, \zsuser_##
...
If changes are correctly made, save the files:
:wall
If the editing you want to automate could not be expressed only
in substitutions, record a macro and run it via the :normal
command:
:argdo norm!#z
(Here z is the name of the macro to be run.)
Lastly, if the editing should be performed from time to time and
needs to be stored in a script, try using the approach described
in the answer to a similar question.
Answer
While most vim users would be aware of the % motion command for executing inclusive commands on the whole document in the current buffer. Most modern versions of vim (ie 6.x+) support actions on regex searches for exclusive actions like so:
:/regex/substitute/match/replace/ # as vim command line
+"/reges/s/match/replace" # as bash cli parameter
This breaks down into vim doing the following,
search for regex and put the cursor at start of found point
call internal substitute function (see :help s or :help substitute) [ could be other commands ]
match string with regex for substitution
replace with new string value
Effectively it operates the same as the :global command.
Notes
Command after regex search can be any command, including '!"shell command"' filter commands.
Reference Help
:help global
:help substitute
:help filter
Okay, I am sure this is simple but it is driving me nuts. I recently went to work on a program where I have had to step back in time a bit and use Redhat 9. When I'm typing on the command line from a standard xterm running KornShell (ksh), and I reach the end of the line the screen slides to the right (cutting off the left side of my command) instead of wrapping the text around to a new line. This makes things difficult for me because I can't easily copy and paste from the previous command straight from the command line. I have to look at the history and paste the command from there. In case you are wondering, I do a lot of command-line awk scripts that cause the line to get quite long.
Is there a way to force the command line to wrap instead of shifting visibility to the right side of the command I am typing?
I have poured through man page options with no luck.
I'm running:
XFree86 4.2.99.903(174)
KSH 5.2.14.
Thanks.
Did you do man ksh?
You want to do a set -o multiline.
Excerpt from man ksh:
multiline:
The built-in editors will use multiple lines on the screen for
lines that are longer than the width of the screen. This may not
work for all terminals.
eval $(resize) should do it.
If possible, try to break the command down to multiple lines by adding \
ie:
$ mycommand -a foo \
-f bar \
-c dif
The simple answer is:
$ set -o multiline
ksh earlier than 5.12, like the ksh shipped with NetBSD 6.1, doesn't have this option. You will have to turn off current Interactive Input Line Editing mode, which is usually emacs:
$ set +o emacs
This turns off a lot of featuers altogether, like tab-completion or the use of 'Up-arrow' key to roll back the previous command.
If you decide to get used to emacs mode somehow, remember ^a goes to the begining of the line ("Home" key won't workk) and ^e goes to the end.
I don't know of a way of forcing the shell to wrap, but I would ask why you'd be writing lines that long. With awk scripts, I simply wrap the script in single quotes, and then break the lines where I want. It only gets tricky if you need single quotes in the script -- and diabolical if you need both single and double quotes. Actually, the rule is simple enough: use single quotes to wrap the whole script, and when you want a single quote in the script, write '\''. The first quote terminates the previous single-quoted string; the backslash-single quote yields a single quote; and the last single quote starts a new single quoted string. It really gets hairy if you need to escape those characters for an eval or something similar.
The other question is - why not launch into an editor. Since I'm a die-hard vim nutcase (ok - I've been using vi for over 20 years, so it is easier for me than the alternatives), I have Korn shell set to vi mode (set -o vi), and can do escape-v to launch the editor on whatever I've typed.
This is kind of a pragmatic answer, but when that's an issue for me I usually do something like:
strings ~/.history | grep COMMAND
or
strings ~/.history | tail
(The history file has a little bit of binary data in it, hence 'strings')