I wrote simple jsonview script to view json files:
#!/bin/bash
tmp_file=/tmp/jsonview.json
cat "${#}" | python -m json.tool > $tmp_file
[[ -f $tmp_file ]] && vim $tmp_file
I am not using less because I need syntax highlighting.
That useless use of cat cat ${#} | ... is so that script can be used as a filter:
jsonview t.json
and:
cat t.json | jsonview
If jsonview used as in second, pipe case - despite the fact that vim is invoked not on pipe but on concrete file, I am getting that warning in subject. I can view json file, but after exit, it messes up terminal. Why is this warning? Why vim thinks that it reads from a pipe?
Vim doesn't like it when standard input is redirected unless you invoke it as vim -. In that case it knows stdin is redirected and handles it. As a side benefit it also lets you get rid of the temp file.
#!/bin/bash
cat "$#" | python -m json.tool | vim +'set syntax=javascript' -R -
Always quote "$#" to ensure file names with whitespace won't mess your script up.
-R gets rid of the prompt to save the buffer when exiting Vim.
you can also do vim $tmp_file </dev/tty to tell vim input is from the terminal and not from standard input.
Related
This question already has answers here:
How do I use sudo to redirect output to a location I don't have permission to write to? [closed]
(15 answers)
sudo cat << EOF > File doesn't work, sudo su does [duplicate]
(5 answers)
Closed 1 year ago.
I am trying to automate the addition of a repository source in my arch's pacman.conf file but using the echo command in my shell script. However, it fails like this:-
sudo echo "[archlinuxfr]" >> /etc/pacman.conf
sudo echo "Server = http://repo.archlinux.fr/\$arch" >> /etc/pacman.conf
sudo echo " " >> /etc/pacman.conf
-bash: /etc/pacman.conf: Permission denied
If I make changes to /etc/pacman.conf manually using vim, by doing
sudo vim /etc/pacman.conf
and quiting vim with :wq, everything works fine and my pacman.conf has been manually updated without "Permission denied" complaints.
Why is this so? And how do I get sudo echo to work? (btw, I tried using sudo cat too but that failed with Permission denied as well)
As #geekosaur explained, the shell does the redirection before running the command. When you type this:
sudo foo >/some/file
Your current shell process makes a copy of itself that first tries to open /some/file for writing, then if that succeeds it makes that file descriptor its standard output, and only if that succeeds does it execute sudo. This is failing at the first step.
If you're allowed (sudoer configs often preclude running shells), you can do something like this:
sudo bash -c 'foo >/some/file'
But I find a good solution in general is to use | sudo tee instead of > and | sudo tee -a instead of >>. That's especially useful if the redirection is the only reason I need sudo in the first place; after all, needlessly running processes as root is precisely what sudo was created to avoid. And running echo as root is just silly.
echo '[archlinuxfr]' | sudo tee -a /etc/pacman.conf >/dev/null
echo 'Server = http://repo.archlinux.fr/$arch' | sudo tee -a /etc/pacman.conf >/dev/null
echo ' ' | sudo tee -a /etc/pacman.conf >/dev/null
I added > /dev/null on the end because tee sends its output to both the named file and its own standard output, and I don't need to see it on my terminal. (The tee command acts like a "T" connector in a physical pipeline, which is where it gets its name.) And I switched to single quotes ('...') instead of doubles ("...") so that everything is literal and I didn't have to put a backslash in front of the $ in $arch. (Without the quotes or backslash, $arch would get replaced by the value of the shell parameter arch, which probably doesn't exist, in which case the $arch is replaced by nothing and just vanishes.)
So that takes care of writing to files as root using sudo. Now for a lengthy digression on ways to output newline-containing text in a shell script. :)
To BLUF it, as they say, my preferred solution would be to just feed a here-document into the above sudo tee command; then there is no need for cat or echo or printf or any other commands at all. The single quotation marks have moved to the sentinel introduction <<'EOF', but they have the same effect there: the body is treated as literal text, so $arch is left alone:
sudo tee -a /etc/pacman.conf >/dev/null <<'EOF'
[archlinuxfr]
Server = http://repo.archlinux.fr/$arch
EOF
But while that's how I'd do it, there are alternatives. Here are a few:
You can stick with one echo per line, but group all of them together in a subshell, so you only have to append to the file once:
(echo '[archlinuxfr]'
echo 'Server = http://repo.archlinux.fr/$arch'
echo ' ') | sudo tee -a /etc/pacman.conf >/dev/null
If you add -e to the echo (and you're using a shell that supports that non-POSIX extension), you can embed newlines directly into the string using \n:
# NON-POSIX - NOT RECOMMENDED
echo -e '[archlinuxfr]\nServer = http://repo.archlinux.fr/$arch\n ' |
sudo tee -a /etc/pacman.conf >/dev/null
But as it says above, that's not POSIX-specified behavior; your shell might just echo a literal -e followed by a string with a bunch of literal \ns instead. The POSIX way of doing that is to use printf instead of echo; it automatically treats its argument like echo -e does, but doesn't automatically append a newline at the end, so you have to stick an extra \n there, too:
printf '[archlinuxfr]\nServer = http://repo.archlinux.fr/$arch\n \n' |
sudo tee -a /etc/pacman.conf >/dev/null
With either of those solutions, what the command gets as an argument string contains the two-character sequence \n, and it's up to the command program itself (the code inside printf or echo) to translate that into a newline. In many modern shells, you have the option of using ANSI quotes $'...', which will translate sequences like \n into literal newlines before the command program ever sees the string. That means such strings work with any command whatsoever, including plain old -e-less echo:
echo $'[archlinuxfr]\nServer = http://repo.archlinux.fr/$arch\n ' |
sudo tee -a /etc/pacman.conf >/dev/null
But, while more portable than echo -e, ANSI quotes are still a non-POSIX extension.
And again, while those are all options, I prefer the straight tee <<EOF solution above.
The problem is that the redirection is being processed by your original shell, not by sudo. Shells are not capable of reading minds and do not know that that particular >> is meant for the sudo and not for it.
You need to:
quote the redirection ( so it is passed on to sudo)
and use sudo -s (so that sudo uses a shell to process the quoted redirection.)
http://www.innovationsts.com/blog/?p=2758
As the instructions are not that clear above I am using the instructions from that blog post. With examples so it is easier to see what you need to do.
$ sudo cat /root/example.txt | gzip > /root/example.gz
-bash: /root/example.gz: Permission denied
Notice that it’s the second command (the gzip command) in the pipeline that causes the error. That’s where our technique of using bash with the -c option comes in.
$ sudo bash -c 'cat /root/example.txt | gzip > /root/example.gz'
$ sudo ls /root/example.gz
/root/example.gz
We can see form the ls command’s output that the compressed file creation succeeded.
The second method is similar to the first in that we’re passing a command string to bash, but we’re doing it in a pipeline via sudo.
$ sudo rm /root/example.gz
$ echo "cat /root/example.txt | gzip > /root/example.gz" | sudo bash
$ sudo ls /root/example.gz
/root/example.gz
sudo bash -c 'echo "[archlinuxfr]" >> /etc/pacman.conf'
STEP 1 create a function in a bash file (write_pacman.sh)
#!/bin/bash
function write_pacman {
tee -a /etc/pacman.conf > /dev/null << 'EOF'
[archlinuxfr]
Server = http://repo.archlinux.fr/\$arch
EOF
}
'EOF' will not interpret $arch variable.
STE2 source bash file
$ source write_pacman.sh
STEP 3 execute function
$ write_pacman
append files (sudo cat):
cat <origin-file> | sudo tee -a <target-file>
append echo to file (sudo echo):
echo <origin> | sudo tee -a <target-file>
(EXTRA) disregard the ouput:
echo >origin> | sudo tee -a <target-file> >/dev/null
Here's the full version of my question. I'm including all this detail in case my hunch is wrong, but you may want to skip to the tl;dr below.
I'm trying to write a function that runs an arbitrary command and also captures whether any output was printed to the terminal. I don't want to interfere with the output being printed. In case it's a relevant complication (probably not), I also want to branch on the exit code of the command.
Here's what I have:
function run_and_inspect {
# this subshell ensures the stdout of ${#} is printed and captured
if output=$(
set -o pipefail
"${#}" | tee /dev/tty
); then
did_cmd_work="yes"
else
did_cmd_work="no"
fi
if [ -z "$output" ]; then
was_there_output="no"
else
was_there_output="yes"
fi
echo "output?" $was_there_output
}
Generally this works fine:
$ run_and_inspect true
output? no
$ run_and_inspect "echo hello"
hello
output? yes
But I've found one problem command:
git pull | grep -v 'Already up to date.'
If there is nothing to pull, this pipeline usually produces no output. But, if ssh-add needs to prompt for a passphrase, there is output. It just doesn't get noticed by run_and_inspect:
function git_pull_quiet {
git pull | grep -v 'Already up to date.'
}
$ run_and_inspect git_pull_quiet
Enter passphrase for key '/home/foo/.ssh/id_ed25519':
output? no
There was output to my terminal. I assume the problem is it didn't come from stdout of the git pull pipeline, which is all run_and_inspect knows about. Where did it come from? How do I fix this? I've tried redirecting stderr too (i.e. git pull 2>&1), but with no luck. Is there some way to monitor /dev/tty directly?
tl;dr (I think!)
I think this question boils down to: why isn't the passphrase prompt in log.txt?
$ git pull 2>&1 | tee log.txt
Enter passphrase for key '/home/foo/.ssh/id_ed25519':
Already up to date.
$ cat log.txt
Already up to date.
why isn't the passphrase prompt in log.txt?
The prompt is printed from openssh load_identify_file with readpass.c read_passphrase(). The function does open(_PATH_TTY with _PATH_TTY "/dev/tty" and then write()s to it.
The output is displayed directly to the terminal /dev/tty, not with standard streams.
Just like you do with your tee /dev/tty, which means that your function will also not work. Prefer to preserve the stdout-ness of the child program and preserve buffering:
if { tmp=$("$#" > >(tee >(cat >&3))); } 3>&1; then
Is there some way to monitor /dev/tty directly?
Write your own terminal emulator and spawn your process in it, and then parse all input inside that terminal emulator. Programs like screen or tmux may be of use.
The workaround could be to download proot and open a file descriptor to some file, then create a chroot with just /dev/tty file symlinked to /proc/self/fd/<that file descriptor> and run the process with proot inside that chroot. The idea is that process will see that chroot with /dev/tty file replaced and will write to your file descriptor instead to the terminal.
I have a file like this (text.txt):
ls -al
ps -au
export COP=5
clear
Each line corresponds at a command. In my script, I need to read each line and launch each command.
ps: I tried all these options and with all of them I have the same problem with the command "export". In the file there is "export COP=5", but after running the script, if I do echo $COP in the same terminal, no value is displayed
while IFS= read line; do eval $line; done < text.txt
Be careful about it, it's generally not advised to use eval as it's quite powerful and as easy to be abused.
However, if there is no risk of influence from unprivileged users on text.txt it should be ok.
cat test.txt | xargs -l1 bash -c '"$#"' echo
In order to avoid confusion I would simply rename the file from text.txt to text and add a shebang (e.g. #!/bin/bash) as the first line of the file. Make sure it is executable by calling chmod +x text. Afterwards you can execute it as expected.
$ cat text
#!/bin/bash
ls -al
ps -au
clear
$ chmod +x text
$ ./text
This question already has answers here:
How do I use sudo to redirect output to a location I don't have permission to write to? [closed]
(15 answers)
sudo cat << EOF > File doesn't work, sudo su does [duplicate]
(5 answers)
Closed 1 year ago.
I am trying to automate the addition of a repository source in my arch's pacman.conf file but using the echo command in my shell script. However, it fails like this:-
sudo echo "[archlinuxfr]" >> /etc/pacman.conf
sudo echo "Server = http://repo.archlinux.fr/\$arch" >> /etc/pacman.conf
sudo echo " " >> /etc/pacman.conf
-bash: /etc/pacman.conf: Permission denied
If I make changes to /etc/pacman.conf manually using vim, by doing
sudo vim /etc/pacman.conf
and quiting vim with :wq, everything works fine and my pacman.conf has been manually updated without "Permission denied" complaints.
Why is this so? And how do I get sudo echo to work? (btw, I tried using sudo cat too but that failed with Permission denied as well)
As #geekosaur explained, the shell does the redirection before running the command. When you type this:
sudo foo >/some/file
Your current shell process makes a copy of itself that first tries to open /some/file for writing, then if that succeeds it makes that file descriptor its standard output, and only if that succeeds does it execute sudo. This is failing at the first step.
If you're allowed (sudoer configs often preclude running shells), you can do something like this:
sudo bash -c 'foo >/some/file'
But I find a good solution in general is to use | sudo tee instead of > and | sudo tee -a instead of >>. That's especially useful if the redirection is the only reason I need sudo in the first place; after all, needlessly running processes as root is precisely what sudo was created to avoid. And running echo as root is just silly.
echo '[archlinuxfr]' | sudo tee -a /etc/pacman.conf >/dev/null
echo 'Server = http://repo.archlinux.fr/$arch' | sudo tee -a /etc/pacman.conf >/dev/null
echo ' ' | sudo tee -a /etc/pacman.conf >/dev/null
I added > /dev/null on the end because tee sends its output to both the named file and its own standard output, and I don't need to see it on my terminal. (The tee command acts like a "T" connector in a physical pipeline, which is where it gets its name.) And I switched to single quotes ('...') instead of doubles ("...") so that everything is literal and I didn't have to put a backslash in front of the $ in $arch. (Without the quotes or backslash, $arch would get replaced by the value of the shell parameter arch, which probably doesn't exist, in which case the $arch is replaced by nothing and just vanishes.)
So that takes care of writing to files as root using sudo. Now for a lengthy digression on ways to output newline-containing text in a shell script. :)
To BLUF it, as they say, my preferred solution would be to just feed a here-document into the above sudo tee command; then there is no need for cat or echo or printf or any other commands at all. The single quotation marks have moved to the sentinel introduction <<'EOF', but they have the same effect there: the body is treated as literal text, so $arch is left alone:
sudo tee -a /etc/pacman.conf >/dev/null <<'EOF'
[archlinuxfr]
Server = http://repo.archlinux.fr/$arch
EOF
But while that's how I'd do it, there are alternatives. Here are a few:
You can stick with one echo per line, but group all of them together in a subshell, so you only have to append to the file once:
(echo '[archlinuxfr]'
echo 'Server = http://repo.archlinux.fr/$arch'
echo ' ') | sudo tee -a /etc/pacman.conf >/dev/null
If you add -e to the echo (and you're using a shell that supports that non-POSIX extension), you can embed newlines directly into the string using \n:
# NON-POSIX - NOT RECOMMENDED
echo -e '[archlinuxfr]\nServer = http://repo.archlinux.fr/$arch\n ' |
sudo tee -a /etc/pacman.conf >/dev/null
But as it says above, that's not POSIX-specified behavior; your shell might just echo a literal -e followed by a string with a bunch of literal \ns instead. The POSIX way of doing that is to use printf instead of echo; it automatically treats its argument like echo -e does, but doesn't automatically append a newline at the end, so you have to stick an extra \n there, too:
printf '[archlinuxfr]\nServer = http://repo.archlinux.fr/$arch\n \n' |
sudo tee -a /etc/pacman.conf >/dev/null
With either of those solutions, what the command gets as an argument string contains the two-character sequence \n, and it's up to the command program itself (the code inside printf or echo) to translate that into a newline. In many modern shells, you have the option of using ANSI quotes $'...', which will translate sequences like \n into literal newlines before the command program ever sees the string. That means such strings work with any command whatsoever, including plain old -e-less echo:
echo $'[archlinuxfr]\nServer = http://repo.archlinux.fr/$arch\n ' |
sudo tee -a /etc/pacman.conf >/dev/null
But, while more portable than echo -e, ANSI quotes are still a non-POSIX extension.
And again, while those are all options, I prefer the straight tee <<EOF solution above.
The problem is that the redirection is being processed by your original shell, not by sudo. Shells are not capable of reading minds and do not know that that particular >> is meant for the sudo and not for it.
You need to:
quote the redirection ( so it is passed on to sudo)
and use sudo -s (so that sudo uses a shell to process the quoted redirection.)
http://www.innovationsts.com/blog/?p=2758
As the instructions are not that clear above I am using the instructions from that blog post. With examples so it is easier to see what you need to do.
$ sudo cat /root/example.txt | gzip > /root/example.gz
-bash: /root/example.gz: Permission denied
Notice that it’s the second command (the gzip command) in the pipeline that causes the error. That’s where our technique of using bash with the -c option comes in.
$ sudo bash -c 'cat /root/example.txt | gzip > /root/example.gz'
$ sudo ls /root/example.gz
/root/example.gz
We can see form the ls command’s output that the compressed file creation succeeded.
The second method is similar to the first in that we’re passing a command string to bash, but we’re doing it in a pipeline via sudo.
$ sudo rm /root/example.gz
$ echo "cat /root/example.txt | gzip > /root/example.gz" | sudo bash
$ sudo ls /root/example.gz
/root/example.gz
sudo bash -c 'echo "[archlinuxfr]" >> /etc/pacman.conf'
STEP 1 create a function in a bash file (write_pacman.sh)
#!/bin/bash
function write_pacman {
tee -a /etc/pacman.conf > /dev/null << 'EOF'
[archlinuxfr]
Server = http://repo.archlinux.fr/\$arch
EOF
}
'EOF' will not interpret $arch variable.
STE2 source bash file
$ source write_pacman.sh
STEP 3 execute function
$ write_pacman
append files (sudo cat):
cat <origin-file> | sudo tee -a <target-file>
append echo to file (sudo echo):
echo <origin> | sudo tee -a <target-file>
(EXTRA) disregard the ouput:
echo >origin> | sudo tee -a <target-file> >/dev/null
I'm trying to record the output of a command with post processing to clean things up
(like removing ansi escape codes to a file while outputing the command to screen)
(command is minicom which functions as a terminal among other things).
currently I have the following but it doesn't work(seems to block).
rm "${fifo}"
mkfifo "${fifo}"
cat "${fifo}"|filter_1 >"${log_file}" &
command |tee "${fifo}"
p.s.
command | tee "${log_file}"
works fine
Besides unbuffer, you can try
{ command ; printf "\n" ; } | tee "${log_file}"
I hope this helps.