I am writing a KornShell (ksh) script that is logging to a file. I am redirecting the output of one of my commands (scp) to the same file, but I would like to add a tab at the start of those lines in the log file if possible.
Is this possible to do?
EDIT: Also I should mention that the text I am redirecting is coming from stderr. My line currently looks like this:
scp -q ${wks}:${file_location} ${save_directory} >> ${script_log} 2>&1
Note: the below doesn't work for ksh (see this question for possible solutions).
You probably can do something like
my_command | sed 's/^/\t/' >> my.log
The idea is to process the output of the command with a stream editor like sed in some manner. In this case, a tab will be added at the beginning of every line. Consider:
$ echo -e 'Test\nfoobar' | sed 's/^/\t/'
Test
foobar
I haven't tested this in ksh, but a quick web search suggests that it should work.
Also note that some commands can write to both stdout and stderr, don't forget to handle it.
Edit: in response to the comment and the edit in the question, the adjusted command can look like
scp -q ${wks}:${file_location} ${save_directory} 2>&1 | \
sed 's/^/\t/' >> ${script_log}
or, if you want to get rid of stdout completely,
scp -q ${wks}:${file_location} ${save_directory} 2>&1 >/dev/null | \
sed 's/^/\t/' >> ${script_log}
The technique is described in this answer.
Related
I have the following problem: in a script, that must not be executed as root, I have to write a line to a newly created, empty file. This file is in /etc, so I need elevated privilages to write to it.
Creating the file is simple:
sudo touch /etc/myfile
Now, just using echo to write to the file like so doesn't work...
sudo echo "something" > /etc/myfile
... because only the first part of the command (echo) is executed with sudo, but not the redirection into the file.
Previously I used something like this...
sudo sed -i -e "\$aInsert this" /etc/differntfile
...to add to the end of the file, which worked, because the file wasn't empty. But since sed works line based, it doesn't do anything, the file stays completely empty.
Any suggestions? Is there a way to do this with echo somehow? Any special sed expression? Any other tools I could use?
You can use tee:
echo "something" | sudo tee /etc/myfile # tee -a to append
Or redirect to /dev/null if you don't want to see the output:
echo "something" | sudo tee /etc/myfile > /dev/null
Another option is to use sh -c to perform the full command under sudo:
sudo sh -c 'echo "something" > /etc/myfile'
Regarding doing this with sed: I don't think it is possible. Since sed is a stream editor, if there is no stream, there is nothing it can do with it.
If you're willing to upgrade from sed (which I also think cannot do this) to awk (GNU awk 4.1.0 or higher to be precise), it can be done like this:
sudo gawk -i inplace '{ print } ENDFILE { print "something" }' /etc/myfile
The alternatives given by the accepted answer are probably easier; I just needed something like this for a use case where I could just use a simple shell command (so no redirection), and the file had to be passed as a single standalone argument (so no sh -c which has the target file embedded inside its single argument). Unlike sed, awk also handles an incomplete last line (which is missing the trailing newline) without adding such.
This uses the inplace awk source library; see man 3am inplace for details.
Is there a way to redirect stdin to a file but at the same time reflect what's being read from the file on the console?
Update: I'm trying to redirect the contents of a file to the standard input of a program, but at the same time reflect the standard input and output of that program on the console. I've tried something like:
echo "$(cat inputfile)" | tee /dev/tty | ./program
which doesn't seem to be the right thing to do.
What you are doing seems fine to me. You can avoid the crazy stuff, though:
tee /dev/tty <inputfile | ./program
echo $(cat) will coincidentally squish whitespace. I assume you used this by mistake, but if that's what you genuinely want to accomplish, try
tr -s '\n\t' ' ' <inputfile | tee /dev/tty | ./program
How would I go about taking the output from a post call made using wget and filtering out everything but a string I want using sed. In other words, let's say I have some wget call that returns (amongst part of some string) :
'userPreferences':'some stuff' }
How would I get the string "some stuff" such that the command would look something like:
sed whatever-command-here | wget my-post-parameters some-URL
Also is that the proper way to chain the two as one line?
You want the output of wget to go to sed, so the order would be wget foo | sed bar
wget -q -O - someurl | sed ...
The -q flag will silence most of wget's output and -O - will write to standard output, so you can then pipe everything to sed.
The pipe works the other way around. They chain the left command's output to the right command's input:
wget ... | sed -n "/'userPreferences':/{s/[^:]*://;s/}$//p}" # keeps quotes
The filtering might be easier to express with GNU grep though:
wget ... | grep -oP "(?<='userPreferences':').*(?=' })" # strips the quotes, too
If you are on a system that supports named pipes (FIFOs) or the /dev/fd method of naming open files, you could avoid a pipe and use < <(...)
sed whatever-command-here < <(wget my-post-parameters some-URL)
I am trying to run the following command
./someprogram | tee /dev/tty | sed 's/^.\{2\}//' > output_file
But the file is always blank when I go to check it. If I remove > output_file from the end of the command, I am able to see the output from sed without any issues.
Is there any way that I can redirect the output from sed in this command to a file?
Remove output-buffering from sed command using the -u flag and make sure what you want to log isn't on stderr
-u, --unbuffered
load minimal amounts of data from the input files and flush the output buffers more often
Final command :
./someprogram | tee /dev/tty | sed -u 's/^.\{2\}//' > output_file
This happens with streams (usually a program sending output to stdout during its whole lifetime).
sed / grep and other commands do some buffering in those cases and you have to explicitly disable it to be able to have an output while the program is still running.
You got a Stderr & stdout problem. Checkout In the shell, what does " 2>&1 " mean? on this topic. Should fix you right up.
I know that
./executable &>outputfile
will redirect the standard output and standard error to a file. This is what I want, but I would also like the output to continue to be printed in the terminal. What is the best way to do this?
Ok, here is my exact command: I have tried
./damp2Plan 10 | tee log.txt
and
./damp2Plan 10 2>&1 | tee log.txt
where 10 is just an argument passed to main. Neither work correctly. The result is that the very first printf statement in the code does go to terminal and log.txt just fine, but none of the rest do. I'm on Ubuntu 12.04 (Precise Pangolin).
Use tee:
./executable 2>&1 | tee outputfile
tee outputs in chunks and there may be some delay before you see any output. If you want closer to real-time output, you could redirect to a file as you are now, and monitor it with tail -f in a different shell:
./executable 2>&1 > outputfile
tail -f outputfile