I have a number of files (more than a hundred) that I want to process using Vim. A sample of the files’ contents is as follows:
xyz.csv /home/user/mydocs/abc.txt
/home/user/waves/wav.wav , user_wav.wav
I want this to be replaced by:
xyz.csv /var/lib/mydir/abc.txt
/var/sounds/wav.wav , wav.wav
In each of the files, the changes I need to make are the same. My questions are:
Can I use Vim search and replace functionality by calling it from within a Bash script?
If so, how do I go about it?
P.S. I have searched StackOverflow for similar questions and found some answers using ex scripts, etc. I want to know how I can call an ex script from within a bash script.
While vim is quite powerful, this is not something I would normally use vim for. It can be done using a combination of common command line utilities instead.
I've assumed that the blank line in your example above is actually blank and does not contain spaces or any other whitespace characters. You can use the following to do what you want.
sed -e "s,/home/user/mydocs,/var/lib/mydir," -e "s,/home/user/waves,/var/sounds," -e "/^$/d" file1
You can use that command together with find and a for loop to do this for a bunch of files:
for file in `find . -maxdepth 1 -type f`
do
sed -e "s,/home/user/mydocs,/var/lib/mydir," -e "s,/home/user/waves,/var/sounds," -e "/^$/d" $file
done
In the for loop, the find command above limits the output to all files in the current directory (including dot files), assigning each line from the output of find to the file variable and then running the sed command posted earlier to transform the file the way you want it to be transformed.
This is how you'd invoke an ed script from bash:
ed filename <<END
/^$/d
%s|/home/user/mydocs|/var/lib/mydir|
%s|/home/user/waves|/var/sounds|
%s|, user_|, |
w
q
END
To answer with vim, you can do
vim -e 'bufdo!%s:\(xyz.csv \)/home/user/mydocs/\(abc.txt\n\)\n.*:\1/var/lib/mydir/\2/var/sounds/wav.wav , wav.wav:' -e 'xa' FILES
Note, I had assumed, that the second line is statically replaced, as it had looked like in the question.
If you don't like writing long lines in your script, you can create a file like:
s/FOO/BAR/
" several replacement and other commands
w " write the file
bd " if you want to
Then do:
vim -e "buffdo!source /your_scriptfile" -e "x" FILES
HTH
If all the editing consists in a series of substitutions, the most
idiomatic way of accomplishing it using Vim would be the following.
Open all the target files at once:
vim *.txt
Run the substitution commands on the loaded files:
:argdo %s#/home/user/mydocs#/var/lib/mydir#
:argdo %s#/home/user/waves#/var/sounds#
:argdo %s#, \zsuser_##
...
If changes are correctly made, save the files:
:wall
If the editing you want to automate could not be expressed only
in substitutions, record a macro and run it via the :normal
command:
:argdo norm!#z
(Here z is the name of the macro to be run.)
Lastly, if the editing should be performed from time to time and
needs to be stored in a script, try using the approach described
in the answer to a similar question.
Answer
While most vim users would be aware of the % motion command for executing inclusive commands on the whole document in the current buffer. Most modern versions of vim (ie 6.x+) support actions on regex searches for exclusive actions like so:
:/regex/substitute/match/replace/ # as vim command line
+"/reges/s/match/replace" # as bash cli parameter
This breaks down into vim doing the following,
search for regex and put the cursor at start of found point
call internal substitute function (see :help s or :help substitute) [ could be other commands ]
match string with regex for substitution
replace with new string value
Effectively it operates the same as the :global command.
Notes
Command after regex search can be any command, including '!"shell command"' filter commands.
Reference Help
:help global
:help substitute
:help filter
Related
I want to use the content of a file.txt as part of a bash command.
Suppose that the bash command with its options that I want to execute is:
my_command -a first value --b_long_version_option="second value" -c third_value
but the first 2 options (-a and --b_long_version_option ) are very verbose so instead of inserting directly on the command line (or bash script) I wrote them in a file.txt like this:
-a first value \
--b_long_version_option="second value"
Now I expect to call the command "my_command" with the following syntax (where "path_to/file.txt" is the path to file.txt, expressed in relative or absolute form):
my_command "$(cat path_to/file.txt)" -c third_value
This however is not the right syntax, as my command is breaking and complaining.
How should I write the new version of the command and/or the file.txt so that it is equivalent to its native bash usage?
Thanks in advance!
The quotes are preserving the newlines. Take them off.
You also don't need the cat unless you're running an old bash parser.
my_command $(<path_to/file.txt) -c third_value
You'll need to take the backslashes at the ends of lines out.
Be careful doing things like this, though. It's probably better to just put the whole command in the file, rather than just pieces of it. If you really just want arguments, maybe define them a little more carefully in an array, source the file and then apply them, like this:
in file:
myArgs=( "-a" "first value"
"--b_long_version_option=second value"
)
Note the quoting. Then run with
. file
my_command "${myArgs[#]" -c third_value
e.g.,
$: printf "[%s] " "${myArgs[#]}" -c=foo
[-a] [first value] [--b_long_version_option=second value] [-c=foo]
I haven't seen any example of what you're trying. But, there are simpler ways to achieve your goal.
Bash Alias
ll for example is a bash alias for ls -al. It usually is defined in .bash_profile or .bashrc as follows :
alias ll='ls -al'
So, what you can do is to set another alias for your shorthand command.
alias mycmd='mycommand -a first value --b_long_version_option="second value"'
then you can use it as follows :
mycmd -c third_value
Config file
You can also define a mycommand.json file or mycommand.ini file for default arguments. Then, you will need to check for config file in your software, and assign arguments from it.
Using config file is more advanced solution. You can define multiple config files. You can set default config file in /etc/mycommand/config.ini for example. When running on different directories, you should check ${cwd}/mycommand.ini to check local config file exists or not. You can even add a --config-file argument to your command.
Using alias is more convenient for small tasks, or thing that won't change much. If your command's behavior should be different in some other project, the using a config file would be a better solution.
NOTE: The goal of this question is to find a suitable character sequence for an effective placeholder substitution in a bash script, not finding if a command is evaluated or not.
I have a skeleton named my_script.skelof a bash script in which I have to put a placeholder. I want to find a placeholder sequence that can be safely substituted (I mean that there aren't any clashes with other bash commands or other pathological substitutions, see A non-safe example for an example).
I've figured out on my own that enveloping placeholder_name within #~ and ~# seems a good solution, but I'm not sure that this solution is safe in every possible case.
A non-safe example
One can decide to use /placeholder_name/. So my_dumb_script.skel is:
#!/bin/bash
AN_INNOCENT_PATH="/a/simple/placeholder_name/path"
echo /placeholder_name/
The goal is to replace only /placeholder_name/ in the echo command. If now I use sed on the placeholder:
sed 's%/placeholder_name/%foobar%g' my_dumb_script.skel > output.bash
The output could be unexpected:
#!/bin/bash
AN_INNOCENT_PATH="/a/simplefoobarpath"
echo foobar
In this case we've obtained an unwanted substitution inside AN_INNOCENT_PATH, since it's easy to pattern-match on placeholders that contains the / character. I know that it seems very dumb, but you cannot know how people will use your code in the future (and someone could create a folder named placeholder_name).
A safe example that uses #~
In this case my_better_script.skel is the following one:
#!/bin/bash
AN_INNOCENT_PATH="/a/simple/placeholder_name/path"
echo #~placeholder_name~#
And now we can use sed:
sed 's%#~placeholder_name~#%foobar%g' my_better_script.skel > output.bash
The output now is better:
#!/bin/bash
AN_INNOCENT_PATH="/a/simple/placeholder_name/path"
echo foobar
Now everything works as intended.
You can use type to check if a command will evaluate.
$ placeholder1="testing"
$ placeholder2="test"
$ type $placeholder1
-bash: type: testing: not found
$ type $placeholder2
test is a shell builtin
I have a file that I pass to a bash command that will create an output in a loop like so:
for file in /file/list/*
do
command
done
I wish to save the output that would have gone to standard out of each loop to a text file in my working directory. Currently I am trying this:
for file in /file/list/*
do
command | tee "$file_command output.txt"
done
What I expect to see are new files created in my current directory titled file1.txt_commandoutput.txt, file2.txt_commandoutput.txt, etc. The output of the command should be saved as a different file for each file. However I get only one file created and it's called ".txt" and can't be opened by any standard software on Mac. I am new to bash scripting, so help would be much appreciated!
Thanks.
Your problem comes from the variable name you're using:
"$file_command_output.txt" looks for a variable named file_command_output (the dot cannot be in the variable name, but the alphanumerical characters and the underscore all can).
What you're looking for is "${file}_command_output.txt" to make the variable name more explicit.
You have two issues in your script.
First, the wrong parameter/variable is expanded (file_command instead of file) because it's followed by a character that can be interpreted as part of the name (the underscore, _). To fix it, enclose the parameter name in braces, like this: ${file}_command (see Shell Parameter Expansion in bash manual).
Second, even with fixed variable name expansion, the file won't be created in your working directory, because the file holds an absolute pathname (/file/list/name). To fix it, you'll have to strip the directory from the pathname. You can do that with either basename command, or even better with a modified shell parameter expansion that will strip the longest matching prefix, like this: ${file##*/} (again, see Shell Parameter Expansion, section on ${parameter##word}).
All put together, your script now looks like:
#!/bin/bash
for file in /file/list/*
do
command | tee "${file##*/}_command output.txt"
done
Also, to just save the command output to a file, without printing it in terminal, you can use a simple redirection, instead of tee, like this: command > "${file##*/}_com...".
If you are not aware of xargs, try this:
$ ls
file
$ cat > file
one
two
three
$ while read this; do touch $this; done < ./file
$ ls
file one three two
This question already has answers here:
Are shell scripts sensitive to encoding and line endings?
(14 answers)
Closed 3 years ago.
I have a problem with a character. I think it's a conversion problem between dos and unix.
I have a variable that is a float value.
When I print it with the echo command i get:
0.495959
But when I try to make an operation on that value with the bc command (I am not sure how to write the bc command).
echo $mean *1000 |bc
I get:
(standard_in) 1 : illegal character: ^M
I already use the dos2unix command on my .sh file.
I think it's because my variable have the ^M character (not printed with the echo command)
How can i eliminate this error?
I don't have Cygwin handy, but in regular Bash, you can use the tr -d command to strip out specified characters, and you can use the $'...' notation to specify weird characters in a command-line argument (it's like a normal single-quoted string, except that it supports C/Java/Perl/etc.-like escape sequences). So, this:
echo "$mean" * 1000 | tr -d $'\r' | bc
will strip out carriage-returns on the way from echo to bc.
You might actually want to run this:
mean=$(echo "$mean" | tr -d $'\r')
which will modify $mean to strip out any carriage-returns inside, and then you won't have to worry about it in later commands that use it.
(Though it's also worth taking a look at the code that sets $mean to begin with. How does $mean end up having a carriage-return in it, anyway? Maybe you can fix that.)
This works:
${mean/^M/}
You can get ^M by typing Ctrl-V followed by Ctrl-M. Or, alternatively:
${mean/$(printf "\r")/}
The benefit of this method compared to #ruakh's is that here you are using bash built-ins only. The first will be faster as the second will run inside a subshell.
If you just want to "unixize" $mean:
mean="${mean/^M/}"
Edit: There's yet another way:
${mean/$'\r'/}
Running Windows stuff in cygwin has one nasty side-effect as you found out - capturing the output of Windows programs in a cygwin bash variable will also capture the CR output by the program.
Judicious use of d2u avoids the issue - for example,
runtime="`mediainfo --Inform='Video;%Duration%' ${movie} | d2u`"
(Without the d2u, ${runtime} would have a CR tacked on the end, which causes the problem you saw when you feed it to 'bc' for example.)
Maybe you should just save your script in UNIX format instead of DOS.
Try this:
echo `echo $mean` *1000 |bc
If echo really isn't printing it, it should work.
^M is a carriage return character that is used in Windows along with newline (\n) character to indicate next line. However, it is not how it is done in UNIX world, and so bash doesn't treat at as a special character and it breaks the syntax. What you need to do is to remove that character using one of many methods. dos2unix tool can come handy, for example.
As others have pointed out, this is a Windows line ending issue. There are many ways to fix the problem, but the question is why did this happen in the first place.
I can see this happening in several places:
This is a WINDOWS environment variable that was set when Cygwin started up. Sometimes these variables get a CRLF on the end of them. You mentioned this was a particular issue with this one variable, but you didn't specify where it was set.
You edited this file using a Windows text editor like Notepad or Winpad.
Never use a text editor to edit a program. Use a program editor. If you like VI, download VIM which is available on Windows and comes on Cygwin (and all other Unix-based platforms). If VIM isn't for you, try the more graphically based Notepad++. Both of these editors handle end of line issues, and can create scripts with Unix line endings in Windows or files with Windows line endings in Cygwin.
If you use VIM, you can do the following to change line endings and to set them:
To see the line ending in the current file, type :set ff? while in command mode.
To set the line ending for Unix, type :set ff=unix while in command mode.
To set the line ending for Windows, type :set ff=dos while in command mode.
If you use Notepad++
You can go into the Edit-->EOL Conversion menu item and see what your current line ending setting (it's the one not highlighted) and change it.
To have Notepad++ use Unix line endings as the default, go into the Settings-->Preferences menu item. In the Dialog box, select the New Document/Default Directory tab. In the Format section which is part of the New Document section, select the line ending you want. WARNING: Do not select Mac as an option. That doesn't even work on Macs. If you have a Mac, select Unix.
Okay, I am sure this is simple but it is driving me nuts. I recently went to work on a program where I have had to step back in time a bit and use Redhat 9. When I'm typing on the command line from a standard xterm running KornShell (ksh), and I reach the end of the line the screen slides to the right (cutting off the left side of my command) instead of wrapping the text around to a new line. This makes things difficult for me because I can't easily copy and paste from the previous command straight from the command line. I have to look at the history and paste the command from there. In case you are wondering, I do a lot of command-line awk scripts that cause the line to get quite long.
Is there a way to force the command line to wrap instead of shifting visibility to the right side of the command I am typing?
I have poured through man page options with no luck.
I'm running:
XFree86 4.2.99.903(174)
KSH 5.2.14.
Thanks.
Did you do man ksh?
You want to do a set -o multiline.
Excerpt from man ksh:
multiline:
The built-in editors will use multiple lines on the screen for
lines that are longer than the width of the screen. This may not
work for all terminals.
eval $(resize) should do it.
If possible, try to break the command down to multiple lines by adding \
ie:
$ mycommand -a foo \
-f bar \
-c dif
The simple answer is:
$ set -o multiline
ksh earlier than 5.12, like the ksh shipped with NetBSD 6.1, doesn't have this option. You will have to turn off current Interactive Input Line Editing mode, which is usually emacs:
$ set +o emacs
This turns off a lot of featuers altogether, like tab-completion or the use of 'Up-arrow' key to roll back the previous command.
If you decide to get used to emacs mode somehow, remember ^a goes to the begining of the line ("Home" key won't workk) and ^e goes to the end.
I don't know of a way of forcing the shell to wrap, but I would ask why you'd be writing lines that long. With awk scripts, I simply wrap the script in single quotes, and then break the lines where I want. It only gets tricky if you need single quotes in the script -- and diabolical if you need both single and double quotes. Actually, the rule is simple enough: use single quotes to wrap the whole script, and when you want a single quote in the script, write '\''. The first quote terminates the previous single-quoted string; the backslash-single quote yields a single quote; and the last single quote starts a new single quoted string. It really gets hairy if you need to escape those characters for an eval or something similar.
The other question is - why not launch into an editor. Since I'm a die-hard vim nutcase (ok - I've been using vi for over 20 years, so it is easier for me than the alternatives), I have Korn shell set to vi mode (set -o vi), and can do escape-v to launch the editor on whatever I've typed.
This is kind of a pragmatic answer, but when that's an issue for me I usually do something like:
strings ~/.history | grep COMMAND
or
strings ~/.history | tail
(The history file has a little bit of binary data in it, hence 'strings')