I've got a log file that is rotated automatically when it reaches a certain size. The system keeps 5 rotated logs at a time, the older ones are deleted, and the lifetime of a log file is about 20 minutes.
The task is to monitor the log file (system.log) for a specified error code and when it occurs – to copy the file into another directory, before it is deleted.
I tried this:
tail -F system.log | grep -l "error code" | xargs cp /another/directory
but it returns "cp: taget 'input)' is not a directory"
Apparently this is because grep command does not return the file name as soon as the error code is found in it as I expected.
So I need some help here please.
The normal order of arguments to cp is
cp source destination
xargs puts its arguments at the end of the command, so you're executing the command
cp /another/directory input
which has to arguments backwards.
To solve this, use the -t option to cp to specify the destination explicitly.
xargs cp -t /another/directory
I tried this tail -F system.log | grep -l "error code" | xargs -i cp {} /another/directory and it returned 'cp: cannot stat '(standard input)': no such file or directory' It seems that something is wrong with the part tail -F system.log | grep -l "error code" as it returns (standard input) instead of the name of the file
Oops. I can't believe I didn't see that before...
$: echo foo | grep -l foo
(standard input)
tail is sending the file to grep, so grep's file IS stdin, so that's what it's listing.
edit
Are you using logrotate?
Check out the manual and look carefully at the prerotate/postrotate/firstaction/lastaction options.
For an example, Option 7 here -
have your script scan the just-rotated log, and if it has the trigger string in it, copy it somewhere.
Related
I'm working on a script to cipher a bunch of folders through the use of tar, gzip and gpg, plus pv with du and awk to keep track of progress. Here is the line that causes problems
tar cf - "$f" | pv -s $(($(du -sk "$f" | awk '{print $1}') * 1024)) | gzip | gpg -e -o "$output/$(basename "$f").tar.gz.gpg"
This works well most of the time. However, if the output file already exists, gpg prompts the user, asking if we want to override the file or not. And in this case, when the script exits, the console kind of breaks: what I type does not appear anymore, pressing Enter does not create a new line, and so on.
The problem does not appear if the outfile does not exist yet, nor if the -s option of pv is missing or computed without du and awk (ex: $((500 * 500)). This won't break the console, but obviously the progress bar would be completely off)
The problem is reproducable even by using this command line outside of a script and replacing $f and $output with desired options.
Perhaps one or a combination of these changes will help.
Change the gpg command to write to stdout, redirected to the file you want: gpg -e -o - > "$output/$(basename "$f").tar.gz.gpg".
Calculate the file size with stat: stat -c "%s" "$f".
The whole line might then look like this:
tar cf - "$f" | pv -s $(stat -c "%s" "$f") | gzip | gpg -e -o - > "$output/$(basename "$f").tar.gz.gpg"
How do you use a command line argument as a file path and check for file existence in Bash?
I have the simple Bash script test.sh:
#!/bin/bash
set -e
echo "arg1=$1"
if [ ! -f "$1" ]
then
echo "File $1 does not exist."
exit 1
fi
echo "File exists!"
and in the same directory, I have a data folder containing stuff.txt.
If I run ./test.sh data/stuff.txt I see the expected output:
arg1=data/stuff.txt
"File exists!"
However, if I call this script from a second script test2.sh, in the same directory, like:
#!/bin/bash
fn="data/stuff.txt"
./test.sh $fn
I get the mangled output:
arg1=data/stuff.txt
does not exist
Why does the call work when I run it manually from a terminal, but not when I run it through another Bash script, even though both are receiving the same file path? What am I doing wrong?
Edit: The filename does not have spaces. Both scripts are executable. I'm running this on Ubuntu 18.04.
The filename was getting an extra whitespace character added to it as a result of how I was retrieving it in my second script. I didn't note this in my question, but I was retrieving the filename from folder list over SSH, like:
fn=$(ssh -t "cd /project/; ls -t data | head -n1" | head -n1)
Essentially, I wanted to get the filename of the most recent file in a directory on a remote server. Apparently, head includes the trailing newline character. I fixed it by changing it to:
fn=$(ssh -t "cd /project/; ls -t data | head -n1" | head -n1 | tr -d '\n' | tr -d '\r')
Thanks to #bigdataolddriver for hinting at the problem likely being an extra character.
When using
stat -f "%Sm" -t "%Y%m%d%H%M" test/0025-05-026-107339_14.PDF
the modifcation date as shown in the finder is returned (here: 201611110137)
However, when piping in order to get the newest file in a directory; like
ls -t $dir/0025-05*.PDF | head -1 | stat -f "%Sm" -t "%Y%m%d%H%M"
the system time is returned (here: 201701061146), even though
ls -t $dir/0025-05*.PDF | head -1
returns the same file (test/0025-05-026-107339_14.PDF).
What do I get wrong?
Your stat statement seems a bit odd, but apart from that, the pipe returns stat: missing operand for me. My theory is that the pipe doesn't actually provide the file itself to stat, only a string containing the filename, which is generated on-the-fly, which is why the system time gets returned. I would try substituting the pipe with command substitution, like this:
stat -f "%Sm" -t "%Y%m%d%H%M" $(ls -t $dir/0025-05*.PDF | head -1)
At least it did the trick for me with a slighly differently formulated stat command.
I'm trying to write a small shell script to find the most recently-added file in a directory and then move that file elsewhere. If I use:
ls -t ~/directory | head -1
and then store this in the variable VARIABLE_NAME, why can't I then then move this to ~/otherdirectory via:
mv ~/directory/$VARIABLE_NAME ~/otherdirectory
I've searched around here and Googled, but there doesn't seem to be any information on using variables in file paths? Is there a better way to do this?
Edit: Here's the portion of the script:
ls -t ~/downloads | head -1
read diags
mv ~/downloads/$diags ~/desktop/testfolder
You can do the following in your script:
diags=$(ls -t ~/downloads | head -1)
mv ~/downloads/"$diags" ~/desktop/testfolder
In this case, diags is assigned the value of ls -t ~/downloads | head -1, which can be called on by mv.
The following commands
ls -t ~/downloads | head -1
read diags
are probably not what you intend: the read command does not receive its input from the command before. Instead, it waits for input from stdin, which is why you believe the script to 'hang'. Maybe you wanted to do the following (at least this was my first erroneous attempt at providing a better solution):
ls -t ~/downloads | head -1 | read diags
However, this will (as mentioned by alvits) also not work, because each element of the pipe runs as a separate command: The variable diags therefore is not part of the parent shell, but of a subprocess.
The proper solution therefore is:
diags=$(ls -t ~/downloads | head -1)
There are, however, further possible problems, which would make the subsequent mv command fail:
The directory might be empty.
The file name might contain spaces, newlines etc.
When securing a Drupal or WordPress installation on a shared host that does not expose SSH access (a lousy situation, fwiw) lftp seems like the right approach to batch setting permissions for directories and files. The find command boasts that you can redirect its output, so one should be able to run a find, grep exclude to only match lines ending in "/" meaning a directory, and then set the permissions on such matches to 755 and perform the inverse on file matches and set to 644 and then fine tune specific files, such as settings.php and so forth.
lftp prompt> find . | grep "/$" | xargs chmod -v 755
Isn't working and I'm sure I have failed to chain these commands in the correct sequence and format.
How to get this to work?
Update: by "isn't working" I mean that the above command produces no output to the console, nor to the lftp error log. It isn't running these commands locally, fwiw. I'll reduce the command as a demonstration:
find . | grep "/$"
Will take the output of "find" and return matches, here, directories, by nature of the string match:
./daily/
./ffmpeg-installer/
./hourly/
./includes/
./includes/database/
./includes/database/mysql/
./and_so_forth_on_down
Which is cool, since I wish to perform a chmod (and internal command for lftp, with support varying by ftp server) So I expand the command like this:
find . | grep "/$" | xargs echo
Which outputs — nothing. No error output, either. The pipe from grep to xargs isn't happening.
My goal is to form the equivalent of:
chmod 755 ./daily/
chmod 755 ./ffmpeg-installer/
In lftp, the chmod command is performing an ftp-server-permissions change, not a local perms change.
For an explanation of why this does not work as expected, read on - for a solution to the given problem, scroll down.
The answer can be found in the manpage for lftp, which states that
"[s]ome commands allow redirecting their output (cat, ls, ...) to file or via pipe to external command."
So, when you are using a pipe like this on a command that does support redirection in lftp, you are piping its output to your local tools, which will eventually result in chmod trying to change the permissions for a file/directory on our local machine, and most likely fail in case you don't coincidally have the same directory layout in your current directory locally - which is probably the problem you encountered.
The grep + xargs pipe does work, I just tested the following:
lftp> find -d 2 | grep "/$"
./
./applications/
./lost+found/
./netinfo/
./packages/
./security/
./systems/
lftp> find -d 2 | grep "/$" | xargs echo
./ ./applications/ ./lost+found/ ./netinfo/ ./packages/ ./security/ ./systems/
My wild guess is that it did not appear to work for you because you did not specify a max-depth to find and the network connection + buffering in the pipe got in the way. When I try the same on a directory containing many files/subfolders it takes really long to finish and print. Did the command actually finish for you without output?
But still, what you are trying to do is not possible. As I stated, the right-hand-side of the pipe works with external commands (even if an inbuilt of the same name exists) as explained by the manual, so
lftp> chmod 644 foobar
and
lftp> echo "foobar" | xargs chmod 644
are not equivalent.
Yes, chmod is an inbuilt but used in a pipe in the client it will not execute the inbuilt - the manpage clearly states this and you can easily test this yourself. Try the following commands and check their output:
lftp> echo foo | uname -a
lftp> echo foo | ls -al
lftp> echo foo | chmod --help
lftp> chmod --help
Solution
As far as a solution to your problem is concerned, you can try something along the lines of:
#!/bin/bash
server="ftp.foo.bar"
root_folder="/my/path"
{
{
lftp "${server}" <<EOF
cd "${root_folder}"
find | grep "/$"
quit
EOF
} | awk '{ printf "chmod 755 \"%s\"\n", $0 }'
{
lftp "${server}" <<EOF
cd "${root_folder}"
find | grep -v "/$"
quit
EOF
} | awk '{ printf "chmod 644 \"%s\"\n", $0 }'
} | lftp "${server}"
This logs in to your server, cds to the folder where you want to recursively start changing the permissions, uses find + grep to find all directories, logs out, pipes this file list into awk to build chmod commands around it, repeats the whole process for files and then pipes the whole list of commands into a new lftp invocation to actually run the generated chmod commands.
You will also have to add your credentials to the lftp invocations and you might want to comment out the final | lftp "${server}" to check if it produces the desired output before you actually run the whole thing. Please report back if this works for you!