I'm trying to use vim with -s option to run a script that replaces some lines in a file like this (text.txt):
test1
ab
ac
ae
test2
sd
Script file is like this (script):
:silent %s/test1\zs\_.\+\zetest2/\=substitute(submatch(0), '\n\(\w\)', '\n#\1', 'g')/g
:wq
It comments out lines between test1 and test2. Which is what I want. What I don't want though is output before and after prompt. I run it and get:
user#hostname: ~/vimtest$ vim -s script text.txt
^[[?1;2cuser#hostname: ~/vimtest$ 1;2c
So this ^[[?1;2c is bad news already but 1;2c is in the input as if I already typed it. If I hit enter it gives me a bash error. So I have to remove these symbols each time the script is used. Any ideas?
It seems like vim (or some vim startup script) is trying to figure out what type of terminal you are using. The ^[[?1;2c, with the last few characters left in the input buffer, is almost certainly part of your terminal emulator's response to a DA (Device Attributes) query. You can see this yourself by typing in bash:
printf '\033[c'
or, to see the complete return, pause a bit:
printf '\033[c'; sleep 0.1; echo
The response \033[?1;2c means "I'm a VT100 with Advanced Video Option.", which is what xterm and many other console programs respond. (The Linux console itself responds \033[?6c, which means "I'm a VT102.")
The reason that only 1;2c is left in the console input buffer, by the way, is that the initial escape code \033[? was ignored when it was read. The readline library will ignore it without echoing it, whereas normal console input will echo it and then ignore it; that's why the two shell commands above differ.
I can't reproduce this problem with my vim installation, so I don't really even know where to start looking. But you might try to see if disabling all startup files helps:
vim -u NONE -s script text.txt
If that helps, start disabling installed extensions one by one until you find the one which is causing the problem.
:%s/test1\zs\_.\+\ze\ntest2/\=substitute(submatch(0), '\n', '\n#', 'g')/g
:wq
this is tested here, it changed the input file in required way.
Some changes done based on your command:
add \n after \ze
in substitute() function we can just handle the \n, we don't need to capture the word after the \n
I noticed that you tagged the question with bash, so I thought a shell-solution should be accepted too.
awk '/test1/{p=1;print;next}/test2/{p=0;print;next}{$0=(p?"#":"")$0}7' file
this awk oneliner should do that for you. vim is very powerful editor, I love vim. But if you want to do some automatic transformation, I prefer a script or a proper text processing tool. On a linux box you can always find one. It is easier to test and debug.
Test with your input:
kent$ cat f
test1
ab
ac
ae
test2
sd
kent$ awk '/test1/{p=1;print;next}/test2/{p=0;print;next}{$0=(p?"#":"")$0}7' f
test1
#ab
#ac
#ae
test2
sd
If you want to save the text back to your file, you can :
awk '...' file > tmp.file && mv tmp.file file
Related
TL;DR?
How can I change every instance of one character (e.g. 'E') in a terminal window to another string of characters (e.g. '~E'), moving all other characters along in the window in the process? So:
abcdEfghij
becomes:
abcd~Efghij
This should work in the gnome-terminal and work with whatever output is on that terminal, from whatever program. Ideally it will be a script or other program I can run within that terminal emulator.
The context
I am using a Braille display (the Canute 360) with a Braille screen-reader (brltty), which at present does not support capital letters. They show up in Braille the same as lower-case letters. Until this is developed for BRLTTY I sometimes need to be able to force showing which letters are capitalised in order for me to, for example, write code.
The proposed solution
N.B. The below proposed solution is not intended to be elegant; it is a quick and dirty way of letting me and other Braille-using developers continue to program with this display for our work until the proper solution is forthcoming in the screen-reader proper.
I am trying to essentially 'wrap' the output of the terminal emulator (in this case gnome-terminal to force a certain character in front of every capital letter so on the Braille display it can be identified. I am going to assume that character is tilde (~). I am not concerned about this breaking vertical alignment, or forcing the line off the edge of the display. I am assuming a 40 character wide terminal (width of the Canute).
So this normal output:
$ echo ${string}
A Quick Brown Fox
Jumps over the lazy
Dog. Etc.
$
Becomes this:
$ echo ${string}
~A ~Quick ~Brown ~Fox
~Jumps over the lazy
~Dog. ~Etc.
$
That would then be viewable on the Canute (in US Computer Braille) as this:
$ echo ${string}
~a ~quick ~brown ~fox
~jumps over the lazy
~dog. ~etc.
$
It is fine for this to be a command that has to be called first, like screen. So:
$ caps-hack
$ nano
[doing my thing, then quit nano]
$
[ctrl-x to quit caps-hack]
$
Or alternatively it could launch a new terminal window, or it could be tied to specific TUI applications (like nano). I will primarily be using it for working inside vi, nano, micro and other editors, so if it cannot capture all terminal output the solution is still valuable.
Example use case: Micro/nano text editor
When I need to see capitals whilst editing text using micro or nano I would first launch caps-hack, then I could use the TUI editor, exit it, be back on the terminal, then cancel caps-hack if I wanted to revert to usual behaviour.
This is what nano normally looks like:
GNU nano 4.8 New Buffer Modified
This is a nonsense file for
Stackoverflow.
^G Get Hel^O Write O^W Where I^K Cut Tex
^X Exit ^R Read Fi^\ Replace^U Paste T
I'm looking for a solution that would then make it look like this:
~G~N~U nano 4.8 ~New ~Buffer ~Modi
~This is a nonsense file for
~Stackoverflow.
^~G Get ~Hel^~O ~Write ~O^~W ~Where ~I^~
^~X ~Exit ^~R ~Read ~Fi^\ ~Replace^~U
(Note that I have cut it off at 40 characters.)
The effect would be the same (inserting tildes, cutting off at 40 characters) whether I was in the terminal itself, in mc, or watching a ping stream.
This active use case means that so far as I can see I cannot simply pipe the output of programs to a bash script (like the one below), as that wouldn't work with a TUI. So I cannot do: $ nano myfile.txt | caps-hack
What I have tried so far
I have not worked out how to essentially capture the output, modify it and write it back to the terminal window. However I have the following shell snippet which I believe could be used for it, if I know where to put it.
# Repeat for all visible lines in the terminal
moddedline1=$( echo ${originalLine1} | sed -E -e 's/\([A-Z]\)/~\1/g' )
moddedline1="${moddedline1:0:40}"
tput cup 1 0 && printf "${moddedline1}"
I was just trying to figure out EOF and here documents. I read something online that seemed like a cool idea I could expand upon. It is supposed to create a text file, and add text to it, all using a here document. The syntax is as follows:
#!/bin/sh
filename=test.txt
vim $filename <<EndOfCommands
i
This file was created automatically from
a shell script.
^[
ZZ
EndOfCommands
Now unfortunately I get an error from this:
./EOF.sh
Vim: Warning: Input is not from a terminal
Vim: Error reading input, exiting...
Vim: Finished.
Can anyone help fix this error? Also im assuming ^[ stands for the escape button being hit?
vi and vim are meant to be used interactively in a terminal, not scripted.
Using ed instead, which can happily accept input from standard input sources of all types:
#!/bin/sh
filename=test.txt
ed -s "$filename" <<EndOfCommands
i
This file was created automatically from
a shell script.
.
w
EndOfCommands
(In insert mode, a line with a single period indicates end-of-input and goes back to command mode, kind of like escape in vi(m).)
Using ex is another option; on systems with vim installed it's often a non-visual version of it.
With vim:
filename="test.txt"
vim -c ":wq! $filename" - << EOF
This file was created automatically from
a shell script.
EOF
See: man vim
There seems to be quite a lot of information on how to edit and execute a command using your editor using "edit-and-execute-command (C-x C-e)", but what I would like to achieve is take the current shell command, apply certain filtering (using a script) and then return it to prompt for further approval/manual changes before execution. Is this possible with bash?
Latest update based on my experience
The part 0"+y$dd in the following mapping is really something that you should carefully think about and tailor it to your taste/workflow/experience.
For instance, very frequently I've found myself ending up with multiple lines in the buffer, where I only want to execute the one the cursor is on; in this case I can use 0"+y$dd:%d<CR> instead of 0"+y$dd.
And this is just one of the possible scenarios.
Final answer for those who like vim
Set vim as your EDITOR/VISUAL, so that when editing a command line, you will use vim to edit it.
Put au BufEnter /tmp/bash-fc.* nn <Leader>d 0"+y$dd:wq<CR> in your ~/.vimrc file to map Leaderd (which you will rarely use when editing a command) to the action "delete the current line into the + register without the trailing EOL".
you can use either the + or the * register in the mapping above; the ways to paste into the terminal will likely differ; you need the +clipboard option for these registers to be available.
When finished editing a command in the vim editor, hit EscapeLeaderd.
Paste the clipboard into the terminal (this is terminal-dependent).
Original answer
I often need to do the same, and I do it as follows. (I normally use the set -o vi in bash, so points 1 and 2 in the following are different if you use set -o emacs, the default; based on your question it looks like points 1 and 2 are unified in Ctrl+x followed by Ctrl+e, which is harder to type, imho.)
hit Escape to be in normal mode,
hit v to enter the editor to edit the command,
edit the command as I like,
(This is where you ask the question.)
hit Escape0"+y$dd:wq,
Note: 0"+y$, not simply "+yy, as the latter would copy the newline too, and this would result in executing the command upon pasting it in the command line,
paste the clipboard on the command line
how to do this depends on the terminal you are using, I guess; I hit Ctrl+Alt+v in URxvt.
proceed to approval/manual edit.
Clearly this is just a workaround, consisting in copying the edited command into the clipboard before deleting the whole command, so that nothing gets executed upon exiting the editor; however it's the best I can get for myself.
Update
As my EDITOR (and VISUAL) is equal to vim, when I edit the command, I edit it in vim.
In this respect, I have noticed that the buffer is named /tmp/bash-fc.random, where random is a 6-characters alphanumeric random string.
This gives space to a lot of possiblities, if you use vim as your editor, as you can define some mapping in your .vimrc to execute the whole sequence Escape0"+y$dd:wq. For instance, one command that you'd rarely use when editing a command line is Leaderd; therefore you can put the following mapping in your .vimrc file
au BufEnter /tmp/bash-fc.* nn <Leader>d 0"+y$dd:wq<CR>
so that step 4 in the above recipe becomes
hit EscapeLeaderd
It's not possible to do that in Bash/readline but it's possible in zsh
using edit-command-line command:
darkstar% autoload edit-command-line; zle -N edit-command-line
darkstar% bindkey "^X^E" edit-command-line
Now press Control-x Control-e to open your editor, edit line, leave the editor - you will see the updated command line but it will not be executed automatically.
Now that I think about it, maybe a variation of what #kenorb suggested in a comment is the best workaround (as it seems no solution exists), if we want to stick to bash.
What you can do is prepend a # (the comment character in bash) to the command, rather than echo. Then when you exit the editor, the command will be ineffective, and you will only have to press arrow up (or k, if you use set -o vi), remove the # and confirming.
Note that this strategy adds just a few keystrokes, so it can be fairly efficient, depending on your typing level.
These pieces might get you closer:
a) replace the the normal binding for newline newline (ctrl-M)
bind -x '"\C-M":date"
b) grab the current line from the history using !#
replace date with whatever script you want.
c) edit manually
d) if necessary, somehow tie in !!:p which prints the new command to the command line but does not execute it, thus letting you manually edit it.
e) using ctrl-J submit edited command rather than a newline
or they might not ....
There is an option in bash to modify command from history without executing it. I'm not sure it it's possible to use script for this, doesn't seem to be likely. Although, you can make modifications using history modifiers.
Enable option histverify to prevent execution of modified command
Use chain of modifiers to change last command
Use "!!" to put your result to command line for final edit
Here is how it looks:
$ shopt -s histverify
$ ls *.sh
script1.sh script2.sh script3.sh script-return.sh
$ !!:s/*/script1/:p
ls script1.sh
$ !!:s/1/2/:p
ls script2.sh
$ !!
$ ls script2.sh
script2.sh
I'd like to point you to the Composure framework for Bash (I'm not affiliated with it): https://github.com/erichs/composure
It provides draft and revise functions that sound like they could help with what you're trying to do. Here's a (long) quote from the project's readme file:
Composure helps by letting you quickly draft simple shell functions,
breaking down your long pipe filters and complex commands into
readable and reusable chunks.
Draft first, ask questions later
Once you've crafted your gem of a command, don't throw it away! Use
draft () and give it a good name. This stores your last command as a
function you can reuse later. Think of it like a rough draft.
$ cat servers.txt
bashful: up
doc: down
up-arrow
$ cat servers.txt | grep down
doc: down
$ draft finddown
$ finddown | mail -s "down server(s)" admin#here.com
Revise, revise, revise!
Now that you've got a minimal shell function, you may want to make it
better through refactoring and revision. Use the revise () command
to revise your shell function in your favorite editor.
generalize functions with input parameters
add or remove functionality
add supporting metadata for documentation
$ revise finddown
finddown ()
{
about finds servers marked 'down' in text file
group admin
cat $1 | grep down
}
$ finddown servers.txt
doc: down
It does not seem possible with a keyboard shortcut, at least:
$ bind -P | grep -e command -e edit
complete-command can be found on "\e!".
edit-and-execute-command can be found on "\C-x\C-e".
emacs-editing-mode is not bound to any keys
possible-command-completions can be found on "\C-x!".
vi-editing-mode is not bound to any keys
This can be done in native bash using readline specifically READLINE_LINE and READLINE_POINT variables. I use this functionality all the time though not through vim, you would need to get the value of $selected from your vim command and if not empty it takes your original line + your input and replaces your original line with the combination without executing. output as a variable
_main() {
selected="$(__coms_select__ "$#")"
origonal_text=$READLINE_LINE READLINE_LINE="${READLINE_LINE:0:$READLINE_POINT}$selected${READLINE_LINE:$READLINE_POINT}"
READLINE_POINT=$(( READLINE_POINT + ${#selected} ))
}
bind -m emacs-standard -x '"\C-e": _main '
bind -m vi-command -x '"\C-e": _main '
bind -m vi-insert -x '"\C-e": _main '
Edit
Just remembered these two utilities that will let you do this as well.
Vipe allows you to run your editor in the middle of a unix pipeline and edit the data that is being piped between programs.
vp, up, vipe, Neomux (upgrade of nvim terminal) you can do some pretty neat throwing buffers between the terminal and split window.
and Athame (full vim on the command line)
https://github.com/ardagnir/athame
careful with that one though plugins work on the cli and it can get funky if you got tons of plugins
I have following snippet:
host="https://example.com"
port="80"
url="${host}:${port}"
echo $url
the output is:
:80ps://example.com
How can I escape the colon here. I also tried:
url="${host}\:${port}"
but it did not work.
Expected output is:
https://example.com:80
You've most likely run into what I call the Linefeed-Limbo.
If I copy the code you provided from StackOverflow and run it on my machine (bash version 4.4.19(1)), then it outputs correctly
user#host:~$ cat script.sh
host="https://example.com"
port="80"
url="${host}:${port}"
echo $url
user#host:~$ bash script.sh
https://example.com:80
What is Linefeed-Limbo?
Different operating systems use different ASCII symbols to represent when a new line occurs in a text, such as a script. This Wikipedia article gives a good introduction.
As you can see, Unix and Unix-like systems use the single character \n, also called a "Line Feed". Windows, as well as other systems, use \r\n, so a "carriage return" followed by a "line feed".
What happens now is when you write a script on Windows on an editor such as notepad, what you write is host="example.com"\r\n. When you copy this file into Linux, Linux interprets the \r as if it were part of the script, since only \n is considered a new line. And indeed, when I change my newline style to DOS-style, I get the exact output you get.
How can I fix this?
You have several options to fix this issue.
Converting the script (with dos2unix)
Since all you need to do is replacing every instance of \r\n with \n, you could use any text-editing software you want. However, if you like simple solutions, then dos2unix (and its sister unix2dos) might be what you looking for:
user#host:~$ dos2unix script.sh
dos2unix: converting file script.sh to Unix format...
That's it. Run your file now and you will see it behaves well.
Encoding the source-file correctly
By using a more advanced text editor such as Notepad++, you can define which style of newline you would like to use.
By changing the newline-type to whichever system you intend to run your script on, you will not run into any problems like this anymore.
Bonus round: Why does it output :80ps://example.com?
To understand why your output is like this, you have to look at what your script is doing, and what \r means.
Try thinking of your terminal as an old-fashioned typewriter. Returning the carriage means you start writing on the left again. Making a "new line" means sliding the paper. These two things are seperate, and I think that's why some systems decided to use these two characters as a logical "new line".
But I digress. Let's look at the first line, host="https://example.com"\r.
What this means when printed is "Print https://example.com, then put the carriage back at the start". When you then print :80\r, it doesn't start after ".com", it starts at the beginning of the line, because that's where you (unknowingly) told the cursor to go. it then overwites the first few characters, resulting in ":80ps://example.com" to be written. Keep in mind that after 80, you again placed a carriage return symbol, so any new text you would have written ends up overwriting the beginning again.
It works for me, try to remove carriage returns in variables and then try.
new_host=$(echo "$host" | tr -d '\r')
new_port=$(echo "$port" | tr -d '\r')
new_url="${new_host}:${new_port}"
I'd like to have a blank line after my bash prompt and before the output on my Mac. It should look like this would:
echo; ls
Can I add a newline to my bash prompt and then go back up one line to wait for user input? Is there something obvious I'm missing?
I know this is old but for someone like me who came across this while googling for it. This is how you do this...
It's actually pretty simple!
Check out this link --> Cursor Movement
Basically to move up N number of lines:
echo -e "\033[<N>A HELLO WORLD\n"
Just change the "< N >" to however many lines you want to go back...
For instance, to move up 5 lines it would be "/033[5A"
To my knowledge this is not possible unless you delve into more low-level stuff like full-screen emulators like curses.
This is a bit of a stab in the dark, but you may be able to use VT102 terminal codes to control the cursor without having to use Curses. The relevant VT102 commands that you'd be interested in all consist of sending ESC, then [, then the specific command parameters.
For instance, to move the cursor up one line, one needs to output:
ESC [ 1 A
0x1B 0x5B 0x31 0x41
Be warned that the VT102 documentation generally uses octal, so keep an ascii table handy if you're using hex.
All of this advice is given without having tested it -- I don't know if VT102 commands can be embedded into your bash prompt, but I thought it might be worth a shot.
Edit: Yeah -- looks like a lot of people use VT102 formatting codes in their bash prompts. To translate my above example into something Bash would recognize, putting:
\e[1A
into your prompt should move the cursor up one line.
This is very possible. If your bash has C-v set as the readline quoted-insert command, you can simply add the following to your ~/.inputrc:
RETURN: "\C-e\C-v\n\C-v\n\n"
This wil make bash (readline, actually) insert two verbatim newlines before a regular interpreted newline. By default, only one is inserted, which is what causes output to start on the line after the prompt.
You can test if C-v is set to quoted-insert by typing it in bash (that's Ctrl+V) followed by e.g. an up arrow. This should print ^[[A or something similar. If it doesn't, you can bind it in ~/.inputrc too:
C-v: quoted-insert
RETURN: "\C-e\C-v\n\C-v\n\n"
~/.inputrc can be created if it doesn't exist. The changes will not take effect in running bashes unless you issue a readline re-read-init-file command (by default on C-x C-r). Be careful though. If you do something wrong, enter will no longer issue commands, and fixing your mistake could prove to be difficult. If you should do something wrong, C-o will by default also accept the line.
Adding a newline followed by moving the cursor back to the regular prompt (like you described) is possible, but will not have the effect you intend. The newline you inserted would simply be overwritten by the application output, since you moved the cursor back in front of it.
This works:
trap echo DEBUG
It doesn't add an extra newline if you hit return at an empty prompt.
The command above will cause a newline to be output for every member of a pipeline or multi-command line such as:
$ echo foo; echo bar
\n
foo
\n
bar
To prevent that so that only one extra newline is output before all command output:
PROMPT_COMMAND='_nl=true'; trap -- '$_nl && [[ $BASH_COMMAND != $PROMPT_COMMAND ]] && echo; _nl=false' DEBUG
The DEBUG trap is performed before each command so before the first command it checks to see if the flag is true and, if so, outputs a newline. Then it sets the flag to false so each command afterwards on the line doesn't trigger an extra newline.
The contents of $PROMPT_COMMAND are executed before the prompt is output so the flag is set to true - ready for the next cycle.
Because pressing enter on an empty command line still triggers the execution of the contents of $PROMPT_COMMAND the test in the trap also checks for those contents as the current command and doesn't perform the echo if they match.
I believe (but haven't tried) if you put '\n\b' in the prompt string it would do that.
In general, if you want to find out the codes to do anything a terminal can do, read the terminfo man page.
In this case, the cursor up one line code can be determined by:
tput cuu1
If you redirect the tput output to a file, you can see what control characters are used.
Bash also supports the PROMPT_COMMAND variable, allowing you to run arbitrary commands before each prompt is issued.