I am currently running sed with variable line input to remove a set number of lines from a file.
sed -i '1,'$SomeVariable'd' somefile.extention
Now, this works as intended if I enter it in cygwin. However, if I run it via a shell script, it does not delete the lines as intended. It still picks up their contents, but does not delete them afterwards.
While looking for a solution, I came across Why does sed behave differently in a shell script?
This does not answer my question as I have verified that bash is running sed and not gsed.
Related
I am having trouble with an old sed script that I dug out of the archives that worked fine years ago, but now is not.
The old script was complex, having multiple substitutions between the curly braces, but even really dumbed down to a single substitution, I can not get it to work now.
I am attempting to use it with home-brew gnu-sed on Mac OS X 10.11 El Capitan, but had the same trouble on debian as well, so it seems like a gnu sed specific issue. It works with regular OS X /usr/bin/sed, but not with gnu sed which I would prefer to use for the sake of portability.
Does gnu sed need different flags to allow the use of the curly braces or something else along those lines? in all cases below the sed command links to /usr/local/bin/gsed on the OS X machine. I suppose I could just use OS X sed, but that wouldn't help me on a linux box should I need to use the script there.
Thanks.
The error I get is
sed: file script_file line 3: extra characters after command
Here is my simple two line data file.
[[aaaaaaaaaaaaaaaa]]
aaaaaaaaaaaaaaaaaa
Here is the correct output assuming the script ran successfully.
[[bbbbbbbbbbbbbbbb]]
aaaaaaaaaaaaaaaaaa
Here is the sed script that does not work when stored in a file.
/\[\[.*\]\]/{
s/a/b/g
}
Here is the command I am using to run it.
sed -f script_file data_file
If I eliminate the curly braces in the file it works, but then I can't apply multiple substitutions to the same pattern space. Here is a version of the script file that works without a problem.
/\[\[.*\]\]/s/a/b/g
if I put everything on the command line including the curly braces, it also works.
sed '/\[\[.*\]\]/{s/a/b/g}' data_file
This one also works with stdin data as well.
What do I need to do to get the multi-line curly brace block script file version to work properly with gnu sed?
I meant to run crontab -l | grep sh, but accidently ran crontab -l | sh. How likely is it that this actually ran any commands? I saw a lot of shell errors about command not found (because lines began with numbers), but only saw the tail end of the output. What did it likely do? How likely was it to have run a command?
I think that any redirections in the crontab actually created or truncated files, but I'm wondering if any of the commands might have run. The crontab contained comments (#), regular crontab formatted jobs, and blank lines.
It depends on what was in your crontab.
If you set any environment variables in it, you should probably check and fix them.
Apart from that, you should be okay. Your shell (should) have stopped attempting to execute each line at the first *, unless the expansion itself produced a valid command.
I've been putting together a bash script that takes an ini file (with a format that I've been developing alongside the script) and reads through the file, performing the actions specified.
One of the functions in the ini format allows for a shell command to be passed in and run using eval. I'm running into a problem when the commands contain a variable name.
eval (or the shell in general) doesn't seem to be substituting the values correctly and most of the time it seems to replace all the variable names with blanks, breaking the command. Subshells to create a string output seem to have the same problem.
The strange part is that this worked on my development machine (Running linux mint 13), but when I moved the script to the target machine running CentOS 5.8, these issues showed up.
Some examples of code I read in from the ini file:
shellcmd $toolspath/program > /path/file
shellcmd parsedata=$( cat /path/file )
These go through a script function that strips off the leading shellcmd and then evals the string using
eval ${scmd}
Any ideas on what might be causing the weird behavior and anything I can try to resolve the problem? My ultimate goal here is to have the ability to read in a line from a file and have my script execute it and be able to correctly handle script variables from the read in command.
Using Bash 3.2.25 (CentOS 5) I tried this, and it works fine:
toolspath='/bin'
while read prefix scmd
do
if [[ $prefix == 'shellcmd' ]]
then
echo "Evaluating: <$scmd>"
eval ${scmd}
else
echo "$prefix ignored"
fi
done < ini
with:
shellcmd $toolspath/ls > /home/user1/file
shellcmd parsedata=$( cat /home/user1/file )
shellcmd echo $parsedata
I obviously had to set paths. Most likely you had to change the paths when you switched machines. Do your paths have embedded spaces?
How did you transfer the files? Did you perchance go via Windows? On a whim I did a unix2dos on the ini file and I got similar symptoms to that you describe. That's my best guess.
I found a suitable alternative for the one command that what causing this issue so I'll mark this question as solved.
From all my investigation it appears that I've discovered an obscure bug in the bash shell. The particular command I was trying to eval returned a terminal code in it's output, and because the shell was in a read loop with input redirected from a file it resulted in some strange behavior. My solution was to move the call to this particular command outside of the read loop. It still doesn't solve the root problem, which I believe to be a bug in the bash shell. Hope this will help someone else who has run into this same (obscure) issue.
I have a number of files (more than a hundred) that I want to process using Vim. A sample of the files’ contents is as follows:
xyz.csv /home/user/mydocs/abc.txt
/home/user/waves/wav.wav , user_wav.wav
I want this to be replaced by:
xyz.csv /var/lib/mydir/abc.txt
/var/sounds/wav.wav , wav.wav
In each of the files, the changes I need to make are the same. My questions are:
Can I use Vim search and replace functionality by calling it from within a Bash script?
If so, how do I go about it?
P.S. I have searched StackOverflow for similar questions and found some answers using ex scripts, etc. I want to know how I can call an ex script from within a bash script.
While vim is quite powerful, this is not something I would normally use vim for. It can be done using a combination of common command line utilities instead.
I've assumed that the blank line in your example above is actually blank and does not contain spaces or any other whitespace characters. You can use the following to do what you want.
sed -e "s,/home/user/mydocs,/var/lib/mydir," -e "s,/home/user/waves,/var/sounds," -e "/^$/d" file1
You can use that command together with find and a for loop to do this for a bunch of files:
for file in `find . -maxdepth 1 -type f`
do
sed -e "s,/home/user/mydocs,/var/lib/mydir," -e "s,/home/user/waves,/var/sounds," -e "/^$/d" $file
done
In the for loop, the find command above limits the output to all files in the current directory (including dot files), assigning each line from the output of find to the file variable and then running the sed command posted earlier to transform the file the way you want it to be transformed.
This is how you'd invoke an ed script from bash:
ed filename <<END
/^$/d
%s|/home/user/mydocs|/var/lib/mydir|
%s|/home/user/waves|/var/sounds|
%s|, user_|, |
w
q
END
To answer with vim, you can do
vim -e 'bufdo!%s:\(xyz.csv \)/home/user/mydocs/\(abc.txt\n\)\n.*:\1/var/lib/mydir/\2/var/sounds/wav.wav , wav.wav:' -e 'xa' FILES
Note, I had assumed, that the second line is statically replaced, as it had looked like in the question.
If you don't like writing long lines in your script, you can create a file like:
s/FOO/BAR/
" several replacement and other commands
w " write the file
bd " if you want to
Then do:
vim -e "buffdo!source /your_scriptfile" -e "x" FILES
HTH
If all the editing consists in a series of substitutions, the most
idiomatic way of accomplishing it using Vim would be the following.
Open all the target files at once:
vim *.txt
Run the substitution commands on the loaded files:
:argdo %s#/home/user/mydocs#/var/lib/mydir#
:argdo %s#/home/user/waves#/var/sounds#
:argdo %s#, \zsuser_##
...
If changes are correctly made, save the files:
:wall
If the editing you want to automate could not be expressed only
in substitutions, record a macro and run it via the :normal
command:
:argdo norm!#z
(Here z is the name of the macro to be run.)
Lastly, if the editing should be performed from time to time and
needs to be stored in a script, try using the approach described
in the answer to a similar question.
Answer
While most vim users would be aware of the % motion command for executing inclusive commands on the whole document in the current buffer. Most modern versions of vim (ie 6.x+) support actions on regex searches for exclusive actions like so:
:/regex/substitute/match/replace/ # as vim command line
+"/reges/s/match/replace" # as bash cli parameter
This breaks down into vim doing the following,
search for regex and put the cursor at start of found point
call internal substitute function (see :help s or :help substitute) [ could be other commands ]
match string with regex for substitution
replace with new string value
Effectively it operates the same as the :global command.
Notes
Command after regex search can be any command, including '!"shell command"' filter commands.
Reference Help
:help global
:help substitute
:help filter
I'm having trouble with a shell script in Cygwin. The specific command that's causing the problem is this:
sed -e "s/6.0.[0123456789]\{1,\}/6.0.${REV}/g" "path/to/file/config.xml" > "path/to/file/config.xml.tmp"
The problem is that the file is being created with a strange character at the end, so instead of being named config.xml.tmp, it's named "config.xml.tmp". From the "ls" command and from the windows command prompt, it looks like "config.xml.tmp?"
If I run the sed command just from the shell, the file named correctly, and the script works fine in Linux.
Any idea what could be wrong? Thanks!
My guess is that your script file doesn't have UNIX line endings. The \r character in the windows line ending is what's getting added to the end of your filename. You can check with od or hexdump to see if that's the problem.