I am trying to batch fix some .scc files using vim and need to pass a substitution sting to it.
I'm trying to use vim -E to pass this line ":s/\r/\r/g"
I've tried embedding individual characters as a variable within the quotes but the \r messing it up.
Any advice would be appreciated.
Guessing at what you really want to do as it doesn't sound like a one-off change. The following will remove all \rs.
NOTE it does an inplace edit - no need to redirect etc.
perl -i.bak -p -e 's/\r+//g' filename(s)...
If you want to replace it with a single \r do
perl -i.bak -p -e 's/\r+/\r/g' filename(s)...
Perl will keep a backup but so should you.
It's not using vim but then perhaps you shouldn't want to.
In Vim, you must use \n in the search part and \r in the replacement part. But replacing \n with \r sounds a lot like doing nothing to me.
Anyway, launching Vim just for a substitution is probably a bit too much. You could use sed:
$ sed -i.orig 's/foo/bar/' foo.txt
Read $ man sed for more info.
got it. ':s/\\r/\\r/g'
I just needed the right level of quotation. The issue as I touched on earlier is that we get files that include PC double line breaks presumable made on a mac and the PC's can't read them properly.
So using sed to s/\r/\r\n/g wasn't really working for some reason, and maybe I could explore it more. But I think passing these to vim will help. I must use vim to send :set format=dos to the file.
Thanks all.
Related
We have an application that keeps some info in an encrypted file. To edit the file we have to put the text editor name in an environment variable in bash, for example, EDITOR=vi. Then we run the application and it opens the decrypted file in vi. I am trying to come up with a bash script that updates the encrypted file. The only solution that I can think of is passing sed command instead of vi to the EDITOR variable. It works perfectly for something like EDITOR='sed -i s#aaaa#bbbb#'.
Problem starts when I need space and regular expression. For example: EDITOR='sed -i -r "s#^(\s*masterkey: )(.*)#\1xxxxx#"' which return error. I tried running the EDITOR in bash with $EDITOR test.txt and I can see the problem. It doesn't like double quotes and space between them so I added a backslash before the double quotes and \s instead of space. Now it says unterminated address regex. For several hours I googled and couldn't find any solution. I tried replacing single quotes with double quotes and vice versa and everything that I could find on the internet but no luck.
How can I escape and which characters should I escape here?
Update:
Maybe if I explain the whole situation somebody could suggest an alternative solution. There is an application written by Ruby and it is inside a container. Ruby application has a secret_key_base for production and we supposed to change the key with EDITOR=vi rails credentials:edit --environment=production. I don't know Ruby and google did not return any ruby solution for automation so I could only think about sending sed instead of vi to Ruby.
How can I escape and which characters should I escape here?
That is not possible. Word splitting on the result of expansion cannot be escaped from inside the result of that expansion, it will always run. Note that filename expansion is also running over the result of the expansion.
Create an executable file with the script content and set EDITOR to it.
You could export a bash shell function, after some tries I got to:
myeditor() {
sed -i -E 's#^(\s*masterkey: )(.*)#\1xxxxx#' "$#"
}
export -f myeditor
EDITOR='bash -c "$#" _ myeditor'
I am having trouble with an old sed script that I dug out of the archives that worked fine years ago, but now is not.
The old script was complex, having multiple substitutions between the curly braces, but even really dumbed down to a single substitution, I can not get it to work now.
I am attempting to use it with home-brew gnu-sed on Mac OS X 10.11 El Capitan, but had the same trouble on debian as well, so it seems like a gnu sed specific issue. It works with regular OS X /usr/bin/sed, but not with gnu sed which I would prefer to use for the sake of portability.
Does gnu sed need different flags to allow the use of the curly braces or something else along those lines? in all cases below the sed command links to /usr/local/bin/gsed on the OS X machine. I suppose I could just use OS X sed, but that wouldn't help me on a linux box should I need to use the script there.
Thanks.
The error I get is
sed: file script_file line 3: extra characters after command
Here is my simple two line data file.
[[aaaaaaaaaaaaaaaa]]
aaaaaaaaaaaaaaaaaa
Here is the correct output assuming the script ran successfully.
[[bbbbbbbbbbbbbbbb]]
aaaaaaaaaaaaaaaaaa
Here is the sed script that does not work when stored in a file.
/\[\[.*\]\]/{
s/a/b/g
}
Here is the command I am using to run it.
sed -f script_file data_file
If I eliminate the curly braces in the file it works, but then I can't apply multiple substitutions to the same pattern space. Here is a version of the script file that works without a problem.
/\[\[.*\]\]/s/a/b/g
if I put everything on the command line including the curly braces, it also works.
sed '/\[\[.*\]\]/{s/a/b/g}' data_file
This one also works with stdin data as well.
What do I need to do to get the multi-line curly brace block script file version to work properly with gnu sed?
This question already has answers here:
Are shell scripts sensitive to encoding and line endings?
(14 answers)
Closed 3 years ago.
I have a problem with a character. I think it's a conversion problem between dos and unix.
I have a variable that is a float value.
When I print it with the echo command i get:
0.495959
But when I try to make an operation on that value with the bc command (I am not sure how to write the bc command).
echo $mean *1000 |bc
I get:
(standard_in) 1 : illegal character: ^M
I already use the dos2unix command on my .sh file.
I think it's because my variable have the ^M character (not printed with the echo command)
How can i eliminate this error?
I don't have Cygwin handy, but in regular Bash, you can use the tr -d command to strip out specified characters, and you can use the $'...' notation to specify weird characters in a command-line argument (it's like a normal single-quoted string, except that it supports C/Java/Perl/etc.-like escape sequences). So, this:
echo "$mean" * 1000 | tr -d $'\r' | bc
will strip out carriage-returns on the way from echo to bc.
You might actually want to run this:
mean=$(echo "$mean" | tr -d $'\r')
which will modify $mean to strip out any carriage-returns inside, and then you won't have to worry about it in later commands that use it.
(Though it's also worth taking a look at the code that sets $mean to begin with. How does $mean end up having a carriage-return in it, anyway? Maybe you can fix that.)
This works:
${mean/^M/}
You can get ^M by typing Ctrl-V followed by Ctrl-M. Or, alternatively:
${mean/$(printf "\r")/}
The benefit of this method compared to #ruakh's is that here you are using bash built-ins only. The first will be faster as the second will run inside a subshell.
If you just want to "unixize" $mean:
mean="${mean/^M/}"
Edit: There's yet another way:
${mean/$'\r'/}
Running Windows stuff in cygwin has one nasty side-effect as you found out - capturing the output of Windows programs in a cygwin bash variable will also capture the CR output by the program.
Judicious use of d2u avoids the issue - for example,
runtime="`mediainfo --Inform='Video;%Duration%' ${movie} | d2u`"
(Without the d2u, ${runtime} would have a CR tacked on the end, which causes the problem you saw when you feed it to 'bc' for example.)
Maybe you should just save your script in UNIX format instead of DOS.
Try this:
echo `echo $mean` *1000 |bc
If echo really isn't printing it, it should work.
^M is a carriage return character that is used in Windows along with newline (\n) character to indicate next line. However, it is not how it is done in UNIX world, and so bash doesn't treat at as a special character and it breaks the syntax. What you need to do is to remove that character using one of many methods. dos2unix tool can come handy, for example.
As others have pointed out, this is a Windows line ending issue. There are many ways to fix the problem, but the question is why did this happen in the first place.
I can see this happening in several places:
This is a WINDOWS environment variable that was set when Cygwin started up. Sometimes these variables get a CRLF on the end of them. You mentioned this was a particular issue with this one variable, but you didn't specify where it was set.
You edited this file using a Windows text editor like Notepad or Winpad.
Never use a text editor to edit a program. Use a program editor. If you like VI, download VIM which is available on Windows and comes on Cygwin (and all other Unix-based platforms). If VIM isn't for you, try the more graphically based Notepad++. Both of these editors handle end of line issues, and can create scripts with Unix line endings in Windows or files with Windows line endings in Cygwin.
If you use VIM, you can do the following to change line endings and to set them:
To see the line ending in the current file, type :set ff? while in command mode.
To set the line ending for Unix, type :set ff=unix while in command mode.
To set the line ending for Windows, type :set ff=dos while in command mode.
If you use Notepad++
You can go into the Edit-->EOL Conversion menu item and see what your current line ending setting (it's the one not highlighted) and change it.
To have Notepad++ use Unix line endings as the default, go into the Settings-->Preferences menu item. In the Dialog box, select the New Document/Default Directory tab. In the Format section which is part of the New Document section, select the line ending you want. WARNING: Do not select Mac as an option. That doesn't even work on Macs. If you have a Mac, select Unix.
I'm using perl on windows and am trying to do a one liner using perl to substitute a placeholder in a file using a windows variable that contains a dollar sign. Does anyone know what the correct usage is to make it work with the dollar sign. I've tried various ways and can't seem to get it to work.
For example, I have a properties file that has a token in it (!MYPASSWORD!) that I'm trying to replace like:
somevalue="!MYPASSWORD!"
I have a batch file that looks up a variable say called NEWPASSWORD that contains the password $abc12345$ and I want to use perl substitution to replace the value like the following. Note I may not always know where the $ signs are so I cant escape them. For example another password may be abc$124$563:
echo %NEWPASSWORD% <-- this would contain $abc12345$
perl -p -i.bak -e "s/!MYPASSWORD!/%NEWPASSWORD%/g" a.properties
When its done I want a.properties to be :
somevalue="$abc12345$"
Thanks in advance
Use ' as regexp delimeter symbol. It will disable all variable substitution:
perl -p -i.bak -e "s'!MYPASSWORD!'%NEWPASSWORD%'g" a.properties
I presume you are getting password from user input. why not just do that in Perl without having to go through batch since you are already using Perl? Its easier. you can then use modules like Term::Inkey to mask password and stuff.
simply use escape before dollar, like that :
\$
Okay, I am sure this is simple but it is driving me nuts. I recently went to work on a program where I have had to step back in time a bit and use Redhat 9. When I'm typing on the command line from a standard xterm running KornShell (ksh), and I reach the end of the line the screen slides to the right (cutting off the left side of my command) instead of wrapping the text around to a new line. This makes things difficult for me because I can't easily copy and paste from the previous command straight from the command line. I have to look at the history and paste the command from there. In case you are wondering, I do a lot of command-line awk scripts that cause the line to get quite long.
Is there a way to force the command line to wrap instead of shifting visibility to the right side of the command I am typing?
I have poured through man page options with no luck.
I'm running:
XFree86 4.2.99.903(174)
KSH 5.2.14.
Thanks.
Did you do man ksh?
You want to do a set -o multiline.
Excerpt from man ksh:
multiline:
The built-in editors will use multiple lines on the screen for
lines that are longer than the width of the screen. This may not
work for all terminals.
eval $(resize) should do it.
If possible, try to break the command down to multiple lines by adding \
ie:
$ mycommand -a foo \
-f bar \
-c dif
The simple answer is:
$ set -o multiline
ksh earlier than 5.12, like the ksh shipped with NetBSD 6.1, doesn't have this option. You will have to turn off current Interactive Input Line Editing mode, which is usually emacs:
$ set +o emacs
This turns off a lot of featuers altogether, like tab-completion or the use of 'Up-arrow' key to roll back the previous command.
If you decide to get used to emacs mode somehow, remember ^a goes to the begining of the line ("Home" key won't workk) and ^e goes to the end.
I don't know of a way of forcing the shell to wrap, but I would ask why you'd be writing lines that long. With awk scripts, I simply wrap the script in single quotes, and then break the lines where I want. It only gets tricky if you need single quotes in the script -- and diabolical if you need both single and double quotes. Actually, the rule is simple enough: use single quotes to wrap the whole script, and when you want a single quote in the script, write '\''. The first quote terminates the previous single-quoted string; the backslash-single quote yields a single quote; and the last single quote starts a new single quoted string. It really gets hairy if you need to escape those characters for an eval or something similar.
The other question is - why not launch into an editor. Since I'm a die-hard vim nutcase (ok - I've been using vi for over 20 years, so it is easier for me than the alternatives), I have Korn shell set to vi mode (set -o vi), and can do escape-v to launch the editor on whatever I've typed.
This is kind of a pragmatic answer, but when that's an issue for me I usually do something like:
strings ~/.history | grep COMMAND
or
strings ~/.history | tail
(The history file has a little bit of binary data in it, hence 'strings')