I have multiple files to edit. I have to make certain changes, like copying some lines based on patterns and then I have to insert some text at different places. I am planning to use vim to automate my task. I have written this script
gg
i
some text to add
some more text to add
<esc>
:%s/text/TEXT/g
:wq
But it just opens up the vim editor and inserts even the commands in the file and then I have to manually remove the following text
<esc>
:%s/text/TEXT/g
:wq
and save the file.
I am invoking the following command:
vi -s vimscript mytextfile
I have used vim scripting earlier to do other things than inserting text like searching and replacing or copy-pasting patterns etc.
Alternatives
Unless you really need special Vim capabilities, you're probably better off using non-interactive tools like sed, awk, or Perl / Python / Ruby / your favorite scripting language here.
That said, you can use Vim non-interactively:
Silent Batch Mode
For very simple text processing (i.e. using Vim like an enhanced 'sed' or 'awk', maybe just benefitting from the enhanced regular expressions in a :substitute command), use Ex-mode.
REM Windows
call vim -N -u NONE -n -es -S "commands.ex" "filespec"
Note: silent batch mode (:help -s-ex) messes up the Windows console, so you may have to do a cls to clean up after the Vim run.
# Unix
vim -T dumb --noplugin -n -es -S "commands.ex" "filespec"
Attention: Vim will hang waiting for input if the "commands.ex" file doesn't exist; better check beforehand for its existence! Alternatively, Vim can read the commands from stdin. You can also fill a new buffer with text read from stdin, and read commands from stderr if you use the - argument.
Full Automation
For more advanced processing involving multiple windows, and real automation of Vim (where you might interact with the user or leave Vim running to let the user take over), use:
vim -N -u NONE -n -c "set nomore" -S "commands.vim" "filespec"
Here's a summary of the used arguments:
-T dumb Avoids errors in case the terminal detection goes wrong.
-N -u NONE Do not load vimrc and plugins, alternatively:
--noplugin Do not load plugins.
-n No swapfile.
-es Ex mode + silent batch mode -s-ex
Attention: Must be given in that order!
-S ... Source script.
-c 'set nomore' Suppress the more-prompt when the screen is filled
with messages or output to avoid blocking.
From :help -s:
-s {scriptin} The script file "scriptin" is read. The characters in the
file are interpreted as if you had typed them. The same can
be done with the command ":source! {scriptin}". If the end
of the file is reached before the editor exits, further
characters are read from the keyboard. Only works when not
started in Ex mode, see |-s-ex|. See also |complex-repeat|.
{not in Vi}
Think of it as a macro. This is how your vimscript file should look:
Osome text to add^Msome more text to add^[:%s/text/TEXT^M:wq^M
The ^M special character is a literal <CR> and is obtained with <C-v><CR>.
The ^[ special character is a literal <Esc> and is obtained with <C-v><Esc>.
Why using vim to automate that kind of task whereas there are many commands or shell languages which are excellent and more efficient to do it ? You could use other tools instead of vim to automate your task.
You could try to use sed for exemple :
[ ~]$ cat file
Here is a file
[ ~]$ echo "some text to add" >> file
[ ~]$ echo "some more text to add" >> file
[ ~]$ cat file
Here is a file
some text to add
some more text to add
[ ~]$ sed -i "s/text/TEXT/g" file
[ ~]$ cat file
Here is a file
some TEXT to add
some more TEXT to add
EDIT : There are different ways to insert some text at the top of a file.
Using a temporary file :
echo "some text to add" > tmp.txt
cat file.text >> tmp.txt
mv tmp.text file.txt
Using sed :
sed -i "1isome text to add" file.text # or
sed -i "1s/^/some text to add\n/" file.txt
Using subshell :
echo -e "some text to add\n$(cat file.txt)" > file.txt
I find it more natural to use a standard Vim script, not a normal mode script. You can invoke it using:
vim -c "source vimscript" mytextfile
Your vim script will need a little update to work using this approach. It will look something like:
1
i
some text to add
some more text to add
.
%s/text/TEXT/g
wq
Related
I have two simple scripts:
./cpthat
BlueM/cliclick types on the keyboard: Shift+Cmd+A, then Cmd+C, to the active iTerm terminal:
#!/bin/zsh
cliclick kd:shift,cmd t:a ku:shift t:c ku:cmd
pbpaste>$THATF
Shift+Cmd+A selects the output from the previous command, and
Cmd+C copies "that" to the clipboard.
pbpaste then writes that to the file $THATF defined system-wide.
./that
#!/bin/zsh
cat $THATF
This prints out the output of the last command as stored by cpthat.
(I know I can run $ command > $THATF directly but for other reasons I need to act retroactively on the command output. Also, not thread safe.)
The challenge:
I'm trying to get to where I can start a zsh or bash command with a pipe:
$ |grep -i sometext
Where, in effect, this happens:
$ that|grep -i sometext
Would this be possible somehow?
Overriding the pipe operator?
zsh config magic?
I'm using zsh heavily but am open for any solution.
You don't need to start with a |. Thegrep utility naturally reads STDIN.
Here's a contrived example:
# /bin/sh
# count_matches
grep $1 | wc -l
$ cat file | count_matches thing
You can see the | you're looking for is on the command line itself, not within the script
Similarly this works:
$ count_matches thing < file
In the first example, the STDIN is connected (via the pipe) to the output of the first command (trivially cat). In the second, it's from the actual file via redirection.
So, just get rid of the | and you should be good to go.
Animation showing that | alone at the beginning of command can be replaced automatically by the output of previous command:
Edit ~/.zshrc to override zsh's zle accept-line widget:
readonly THATF="path/to/your/temporary/file"
my-accept-line () {
if [[ "${BUFFER:0:1}" == '|' ]]; then
/usr/local/bin/cliclick kd:shift,cmd t:a ku:shift w:100 t:c ku:cmd
pbpaste>"${THATF}"
BUFFER='cat ${THATF} '${BUFFER}
fi
zle .accept-line
}
zle -N accept-line my-accept-line
Explanation
When you hit enter after entering a command, zsh runs the accept-line widget.
We override that widget, but before exiting we remember to call the original widget with zle .accept-line. With the dot prefix the factory widget is ran.
In iTerm2, shift+cmd+a selects all the output from the previous command, and cmd+c copies that to the system pasteboard.
We paste the contents of the pasteboard and redirects that to the temporary file declared earlier, pointed to by ${THATF}.
We prepend $BUFFER, the zsh special variable available within zle widget code, with the output of the previous command.
Dependencies, caveats:
This particular solution depends on:
cliclick dispatching macOS keyboard events. Perhaps a native solutions exist e.g. ANSI/escape sequence.
iTerm to handle the keybind for copying last commands output.
zsh for the zle widget.
Xode snippet above is proof-of-concept only and is wildly insecure.
My question is simple, how can I redirect all outputs of a bash script to file and terminal and remove color characters from within the script itself?
I can't find an answer which fits all my needs.
So far I tried tee to output to file and terminal, combined with 2>&1 to get stederr and stdout, sed to remove color characters and all of this with exec to do it from within my script but it doesn't work, I only get colored logs into the terminal and nothing in file.
#!/usr/bin/env bash
exec 2>&1 | sed -r 's/\x1b\[[0-9;]*m//g' | tee script.logs
python somepython.py
python someotherpython.py
Here the python scripts produce outputs which are colored.
I want to log those to terminal (untouched) and in file (without the color). In reality there is a lot more going on in my bash script than those just two python scripts, that's why I want to globally redirect the output of my script bash and not just pipe after each python script.
Thus I used exec because I though it allowed to redirect all output produce by a script.
Thanks in advance for any advice and help,
PS: I dont want colored logs in file but I don't care in terminal if this is what needs to be done for logs to not be colored in file.
You may put all your calls in a curly-braced group and redirect the whole lot, e.g.:
#!/usr/bin/env bash
{
python somepython.py
python someotherpython.py
} 2>&1 | sed -r 's/\x1b\[[0-9;]*m//g' | tee script.logs
This way, all stdout and stderr outputs will be passed along the filter.
Color the terminal, not the file
If you want to write the colors to the terminal and write the uncolored text to the file, you may apply the sed filter to the file written by tee, e.g. your script would look something like:
#!/usr/bin/env bash
{
python somepython.py
python someotherpython.py
} 2>&1 | tee >(sed -r 's/\x1b\[[0-9;]*m//g' > script.logs)
This uses process substitution which is a very powerful tool in bash, albeit a bit difficult at first.
Remove buffering
Assuming you would like to read the contents as soon as possible, you may want to deactivate python’s block buffering. This can be done using the -u option:
#!/usr/bin/env bash
{
python -u somepython.py
python -u someotherpython.py
} 2>&1 | tee >(sed -r 's/\x1b\[[0-9;]*m//g' > script.logs)
Side Note: Beware of CSI
Not all special characters are based on the Control Sequence Initiator ESC [.
If a text is colored in green with tput setaf 1, it will use the control sequence ESC [31m, but if the color is reset with tput sgr0, the control sequence may be ESC (B ESC [m (please note the ESC (B sequence). So if you filter only the ESC [ sequences, you may still have control sequence waste in your log file.
Things get even worse if the program uses other types of control characters such as cursor commands.
For those reasons, the best way to avoid problems is to simply avoid writing control sequences from your python scripts. Most builtin programs are protected against that by checking if the output is a terminal before choosing to display colors or not. When the output is not a terminal it assumes that colors may cause issues.
With that said I don’t know if you have control over the python scripts (or other calls you might have), but if you do you might want to test if the output is a terminal. In Bash you check this way:
if [ -t 1 ] # does stdout end up on a terminal?
then
# Display fancy colors
else
# Minimalist display
fi
In Python it would be:
if sys.stdout.isatty(): # does stdout end up on a terminal?
# Display fancy colors
else:
# Minimalist display
Each line in a given file 'a.txt' contains the directory/path to another unique file. Suppose we want to parse 'a.txt' line-by-line, extract the path in string format, and then use a tool such as vim to process the file at this path, and so on.
After going through this thread - Read a file line by line assigning the value to a variable, I wrote the following script, say 'open-file.sh' on bash (I'm new to it)
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
vim -c ":q" -cq $line # Just open the file and close it using :q
done < "$1"
We would then run the above script as -
./open-file.sh a.txt
The problem is that although the path to a new file is correctly specified by $line, when vim opens the file, vim continues to receive the text contained in 'a.txt' as a command. How can I write a script where I can correctly obtain the path from 'a.txt', open it using vim, and then continue parsing the remaining lines in 'a.txt' ?
Replace:
vim -c ":q" -cq $line
With:
vim -c ":q" -cq "$line" </dev/tty
The redirection </dev/tty tells vim to take its standard input from the terminal. Without that, the standard input for vim is "$1".
Also, it is good practice to put $line in double-quotes to protect it from word splitting, etc.
Lastly, while vim is excellent for interactive work, if your end-goal is fully automated processing of each file, you might want to consider tools such as sed or awk.
Although I'm not sure of your ultimate goal, this shell command will execute vim once per line in a.txt:
xargs -o -n1 vim -c ':q' < a.txt
As explained in the comments to Read a file line by line assigning the value to a variable, the issue you're encountering is due to the fact that vim is an interactive program and thus continues to read input from $line.
The problem was already mentioned in a comment under the answer you based your script on.
vim is consuming stdin which is given to the loop by done < $1. We can observe the same behavior in the following example:
$ while read i; do cat; done < <(seq 3)
2
3
<(seq 3) simulates a file with the three lines 1, 2, and 3. Instead of three silent iterations we get only one iteration and the output 2 and 3.
stdin is not only passed to read in the head of the loop, but also to cat in the body of the loop. Therefore read reads one line, the loop is entered, cat reads all remaining lines, stdin is empty, read has nothing to read anymore, the loop exits.
You could circumvent the problem by redirecting something to vim, however there is an even better way. You don't need the loop at all:
< "$1" xargs -d\\n -n1 vim -c :q -cq
xargs will execute vim once for every line in the file given by $1.
I have the following bash script (play.bash):
#!/bin/bash
pico2wave -l=en-US -w=/tmp/test.wav "$1"
aplay /tmp/test.wav
rm /tmp/test.wav
I would like to use it for reading selected text in the vim editor.
The following does not work
:'<,'>w !bash play.bash
The command executes, but I'm not hearing it.
I'd like to only hear the text, without leaving the vim window.
Is this possible?
When using the ! command in vim, your selection will be piped to your script on standard input, not given as a command line argument.
A simple solution would be to modify your script so it reads from standard input:
xargs pico2wave -l=en-US -w=/tmp/test.wav
xargs reads from standard input, then calls pico2wave with the data as command line arguments, removing newlines.
What is the easiest way to append text to a file in Linux?
I had a look at this question, but the accepted answer uses an additional program (sed) I'm sure there should be an easier way with echo or similar.
How about:
echo "hello" >> <filename>
Using the >> operator will append data at the end of the file, while using the > will overwrite the contents of the file if already existing.
You could also use printf in the same way:
printf "hello" >> <filename>
Note that it can be dangerous to use the above. For instance if you already have a file and you need to append data to the end of the file and you forget to add the last > all data in the file will be destroyed. You can change this behavior by setting the noclobber variable in your .bashrc:
set -o noclobber
Now when you try to do echo "hello" > file.txt you will get a warning saying cannot overwrite existing file.
To force writing to the file you must now use the special syntax:
echo "hello" >| <filename>
You should also know that by default echo adds a trailing new-line character which can be suppressed by using the -n flag:
echo -n "hello" >> <filename>
References
echo(1) - Linux man page
noclobber variable
I/O Redirection
cat >> filename
This is text, perhaps pasted in from some other source.
Or else entered at the keyboard, doesn't matter.
^D
Essentially, you can dump any text you want into the file. CTRL-D sends an end-of-file signal, which terminates input and returns you to the shell.
Other possible way is:
echo "text" | tee -a filename >/dev/null
The -a will append at the end of the file.
If needing sudo, use:
echo "text" | sudo tee -a filename >/dev/null
Follow up to accepted answer.
You need something other than CTRL-D to designate the end if using this in a script. Try this instead:
cat << EOF >> filename
This is text entered via the keyboard or via a script.
EOF
This will append text to the stated file (not including "EOF").
It utilizes a here document (or heredoc).
However if you need sudo to append to the stated file, you will run into trouble utilizing a heredoc due to I/O redirection if you're typing directly on the command line.
This variation will work when you are typing directly on the command line:
sudo sh -c 'cat << EOF >> filename
This is text entered via the keyboard.
EOF'
Or you can use tee instead to avoid the command line sudo issue seen when using the heredoc with cat:
tee -a filename << EOF
This is text entered via the keyboard or via a script.
EOF