I'm trying to redirect the output of the tail -f -n 1 to the Postgres COPY command, the requirement is to execute the COPY command for every output of the tail command.
Came out with the following:
tail -f -n 1 <source_file> | xargs -n 1 psql -c 'copy <table_name> from stdin'
but this is not working as the output of the tail command is used as a parameter for the psql command rather than as stdin.
Also the more generic:
tail -f -n 1 <source_file> | psql -tc "copy <table_name> from stdin"
is not working as the COPY command perform a commit at the end of the stream and not for every single row.
Realized that the issue is that COPY is not intended to accept streams and data is available only when COPY command is over (end of data from program), in this specific case program was never terminated as it was intended to work as a consumer.
Related
So i'm trying to take the line-by-line output of a program, then send it to a file if it matches "CPU", also I want every line to go to the screen.
This command works but only after quitting the script with ^C:
cpuminer-multi/cpuminer -u user -p pass -a sha256d -o stratum+tcp://stratum.pool.com:3333 -t cputhreads | tee >(grep "CPU" >> cpu.txt);
but then if I copy and paste it into a bash script "start.sh"
#!/bin/bash
cpuminer-multi/cpuminer -u user -p pass -a sha256d -o stratum+tcp://stratum.pool.com:3333 -t cputhreads | tee >(grep "CPU" >> cpu.txt);
and run it from bash as "./start.sh", it populates cpu.txt with nothing, even after quitting with ^C
So my questions are
A: Why does it only populate the cpu.txt file after ^C?
B: Why does it work as a plain bash command, but not in a script?
I am trying to output to two different files using tee. My first file will basically be tail -f /myfile and my second output will be a subset of the first file. I have looked online that they were saying we can use `|
tee >(proc1) >(proc2)
I have tried the above but both my files are blank.
Here is what i have so far:
myscript.sh
ssh root#server 'tail -f /my/dir/text.log' | tee >(/mydir/my.log) >(grep 'string' /mydir/my.log > /mydir/mysecond.log)
myexpect.sh
#!/usr/bin/expect -f
set pass password
spawn /my/dir/myexpect.sh
expect {
"key fingerprint" {send "yes/r"; exp_contiue}
"assword: " {send "$pass\r"}
}
interact
In your script, there are some problems in the usage of tee,
tee >(/mydir/my.log): can be substitute with tee /mydir/my.log, since tee would write to stdout and files, i.e. /mydir/my.log
grep 'string' /mydir/my.log > /mydir/mysecond.log: as I mentioned, tee would also write to stdout, so no need to grep the string from file, you can grep from stdout directly. Use pipeline to do it.
So the whole command shall be modified as followed,
ssh root#server 'tail -f /my/dir/text.log | tee /mydir/my.log | grep --line-buffered "string" > /mydir/mysecond.log'
Edit:
For your further question
The command would hang because of tail -f was still waiting for output the growing file. If you don't want the command hanged, try to remove -f for tail.
Depends on the option -f existed for tail, you shall use two different way to allow the grep write file.
For tail case: grep can successfully write file
For tail -f case: --line-buffered for grep would use line buffering on output
Looking over the Dokku source code, I noticed two uses of pipe and redirect that I am not familiar with.
One is: cat | command
Example: id=$(cat | docker run -i -a stdin progrium/buildstep /bin/bash -c "mkdir -p /app && tar -xC /app")
The other is cat > file
Example: id=$(cat "$HOME/$APP/ENV" | docker run -i -a stdin $IMAGE /bin/bash -c "mkdir -p /app/.profile.d && cat > /app/.profile.d/app-env.sh")
What is the use of pipe and redirect in the two cases?
Normally, both usages are completely useless.
cat without arguments reads from stdin, and writes to stdout.
cat | command is equivalent with command.
&& cat >file is equivalent with >file, assuming the previous command processed the stdin input.
Looking at it more closely, the sole purpose of that cat command in the second example is to read from stdin. Without it, you would redirect the output of mkdir to the file. So the command first makes sure the directory exists, then writes to the file whatever you feed to it through the stdin.
I am trying to redirect output of a command to a file. The command I am using (zypper) downloads packages from the internet. The command I am using is
zypper -x -n in geany >> log.txt
The command gradually prints output to the console. The problem I am facing is that the above command writes the command output all at once after the command finishes executing. How do I redirect the bash output as I get it onto the terminal, rather than writing all the command output at the end.
Not with bash itself, but via the tee command:
zipper -x -n in geany | tee log.txt
&>>FILE COMMAND
will append the output of COMMAND to FILE
In your case
&>>log.txt zypper -x -n in geany
If you want to pipe a command through a filter, you must assure that the command outputs to standard output (file descriptor 1) -- if it outputs to standard error (file descriptor 2), you have to redirect the 2 to 1 before the pipe. Take into account that only stdout passed through a pipe.
So you have to do so:
2>&1 COMMAND | FILTER
If you want to grep the output and in the same keep it into a log file, you have to duplicate it with tee, and use a filter like ... | tee log-file | grep options
Should be fairly simple to answer:
Let's say I wanted to execute a command determined by the output of a previous one in Bash:
curl http://website.com 2> /dev/null | grep -i "test" --count | <MY-COMMAND>
What I need: <MY-COMMAND> should only execute if grep had some matches (at least 1).
How can I achieve that?
Also, please feel free to add matching tags, I couldn't come up with any
ifne utility ("run a program if the standard input is not empty") from Jeoy Hess's moreutils package will serve you.
A description of it:
a command that would run the following
command if and only if the standard
input is not empty. I often want this
in crontabs, as in:
find . -name core | ifne mail -s "Core files found" root
Do you need the output of grep to be piped to your command? The answer is simpler if you do not. In that case since grep's return code is success only if it finds a match, you can use && or if:
curl http://website.com 2> /dev/null | grep -q -i "test" && <MY-COMMAND>
if curl http://website.com 2> /dev/null | grep -q -i "test"; then
<MY-COMMAND>
fi
The && operator is a shorthand way of performing an if-else check. It is a short-circuiting operator, which means that the right hand side will only be executed if the left hand side fails.
If you need to pipe the output to your command then you'll need to save the output to a temporary file, test for a match, and then execute your command:
if curl http://website.com 2> /dev/null | grep -i "test" > /tmp/grep.txt; then
<MY-COMMAND> < /tmp/grep.txt
fi
curl http://website.com 2> /dev/null | grep -i "test" && <MY-COMMAND>
From the grep man page: "the exit status is 0 if selected lines are found and 1 otherwise"
The command after && is executed only if the previous command returned exit status 0.
curl http://www.google.com 2>/dev/null | grep window -i -c && echo "this is a success"