Organized logging in Bash - bash

I have this script that deletes files 7 days or older and then logs them to a folder. It logs and deletes everything correctly but when I open up the log file for viewing, its very sloppy.
log=$HOME/Deleted/$(date)
find $HOME/OldLogFiles/ -type f -mtime +7 -delete -print > "$log"
The log file is difficult to read
Example File Output: (when opened in notepad)
/home/u0146121/OldLogFiles/file1.txt/home/u0146121/OldLogFiles/file2.txt/home/u0146121/OldLogFiles/file3.txt
Is there anyway to log the file nicer and cleaner? Maybe with the Filename, date deleted, and how old it was?

try
log=$HOME/Deleted/$(date);
# change it to echo -e and insert new line if needed
find $HOME/OldLogFiles/ -type f -mtime +7 -exec echo {} \; > "$log"

Use logger for logging and leave the actual log handling to syslog and logrotate.

In you deletion script you can add 2 variables at the top, lets call them "timeDate" and "logDestination". They would look like this:
timeDate=$(date "+%d/%m/%Y %T")
logDestination=/home/$USER/.deleteLog; touch $logDestination
Now the $(date "+%d/%m/%Y %T") part simply gets the current date and time. It goes off to get the system date as; date (+%d) then the month and year (%m) (%Y) b.t.w the capital Y returns a full year as in YYYY. It then stores that date and time in the variable for later use.
The logDestination variable is holding a directory or file path for us, pointing at a file called .deleteLog . Now the touch part that follows isn't entirely necessary but it will make the file exist if it does not exist or has been deleted or renamed by accident.
In bash scripting and in a host of programming languages one makes methods or functions, which are just simply sections of code that usually do just one job. This function below is designed to write a message to a logfile:
## Logging function
function _logger () {
echo -e "$timeDate $user - $#\r" >> $log
}
A simply explanation of the function is it is able to receive a form of information. If you look at the above function, note the "$#" symbol this is a placeholder if you like, it will put any and all strings (text) you point at the function. Here is a more cut down version of a function that can receive several strings or input:
function _example () {
echo -e "$#"
}
To call this function and give it a message we can literally type (anywhere under the function in your script):
example "hello"
This string "hello" is given to the function and the echo line in the function outputs the "hello" to your terminal or screen. The -e in the echo line helps the echo to distinguish active parts, Tho best use "man echo" to give you a understanding of its behavior modification.
So back to your script.
Lets say you have a line that deletes the contents of a directory (I advise care and caution and discourage such line but meh).
after it has done a delete you can call;
_logger "deletion of file (or $FILE) successful."
and the _logger function will nicely place the date time, message and start a new line for you (\r). The $user in the function is placed by your current view. as in it will say your username.
Functions can be called numerous times, saving you code duplication and makes the scripts look neater.

You're asking a few things here...
Why does my log file look funny in notepad?
Probably because of the tool you're using to view it, but it's hard to say without more information. Notepad is a Windows application, and Windows uses a strategy than unix for ending lines in a text file. If you want to know for sure, use a tool like hexdump or od to see what's really going on inside your file. And as was suggested in comments, head filename.log on the unix host may show you what you expect, in which case you will have confirmed the line ending theory.
If Notepad is your tool of choice, then we can show you how to convert the files so that they're more compatible with Windows. Otherwise, let us know what you really need.
How do I include this extra data in my log?
You're asking for a few items:
filename - easy enough, you've already got that.
date deleted - easy enough, you just need a timestamp for each of the entries.
age of file - harder. Will the timestamp of the file be sufficient, or do you want, for example, the date extracted from the first line of the file itself? If the latter, you'll need to include a sample in your question. I'll go with the former for now.
First off, bash version 4 and up lets you use time formats as part of a printf, so you could do the following:
printf -v log '%s/Deleted/%(%F)T' "$HOME"
But honestly, I think this was a hack, and what you really want is a single text file that gets rotated using some other tool.
log=/var/log/whatever.log
The "some other tool" will depend on your operating system. In FreeBSD, look in /etc/newsyslog.conf. In Linux, refer to the documentation for your distribution, looking for things like "logrotate".
The output from your find command can be the files which were successfully deleted, but you don't have to have just filenames. Consider the following:
find "$HOME/OldLogFiles/" -type f -mtime +7 -ls -delete
The output of this is the entire ls line, before the file is deleted. It includes the file's date and inode number. Preface this with a timestamp, and you may be good to go:
find "$HOME/OldLogFiles/" -type f -mtime +7 -ls -delete | sed "s/^/[$(printf '%(%F %T)T')] /"
Or if you prefer to do this without sed:
while read line; do
printf '[%(%F %T)T] %s\n' -1 "$line"
done < <(find "$HOME/OldLogFiles/" -type f -mtime +7 -ls -delete)
In this version, the -1 is interpreted by printf's %T as "now".
Redirect to $log as you like.
Finally, and this is probably the "best" option, is to avoid managing logfiles in bash entirely, and use your system's logger command instead. The POSIX version is pretty bare-bones, but you can control things syslog style in BSD, Linux, etc.

Related

output from while read loop using iterative filename/timestamp

I am attempting to create a script that will iteratively run a command against a variable (which is a fully qualified filename) and output the results of that command to an individually named/timestamped file (to %S accuracy). Im not great with this stuff at all
here is what I do:
find /vmfs/volumes/unlistedpathname/unlistedfoldername |
while read list;do
vmkfstools -D "$list" >> duringmigration_10mins_"$list".$(date +"%Y.%m.%d.%H.%M.%S");
done
the output im hoping for is something like
duringmigration_10mins_blahblahblah.vmx.2016.09.25.21.26.35
of course it doesnt work, and im not exactly sure how to solve it. I know the problem outright is $list as the filename variable will reprint the fullpath, so I need some sort of way to tell the loop "hey just use the filename as the variable NOT the full path" but im not sure how to do that in this case. Im also hoping to be able to run this from any location not specific path.
There are two problems preventing the behaviour you are looking for:
As you saw the filenames returned by find include the full path.
Your find command will return all the files and the directory name.
We solve #1 by calling basename on $list in the output filename.
We solve #2 by adding -type f to the find command to only return files and not directories.
find /vmfs/volumes/unlistedpathname/unlistedfoldername -type f |
while read list ; do
vmkfstools -D "${list}" >> "duringmigration_10mins_$(basename "${list}").$(date +"%Y.%m.%d.%H.%M.%S")"
done

Exactly why does result=$($shell_command) fail?

Why does assigning command output work in some cases and seemingly not in others? I created a minimal script to show what I mean, and I run it in a directory with one other file in it, a.txt. Please see the ??? in the script below and let me know what's wrong, perhaps try it. Thanks.
#!/bin/bash
## setup so anyone can copy/paste/run this script ("complete" part of MCVE)
tempdir=$(mktemp -d "${TMPDIR:-/tmp}"/demo.XXXX) || exit # make a temporary directory
trap 'rm -rf "$tempdir"' 0 # delete temporary directory on exit
cd "$tempdir" || exit # don't risk changing non-temporary directories
touch a.txt # create a sample file
cmd1="find . -name 'a*' -print"
eval $cmd1 # this produces "./a.txt" as expected
res1=$($cmd1)
echo "res1=$res1" # ??? THIS PRODUCES ONLY "res1=" , $res1 is blank ???
# let's try this as a comparison
cmd2="ls a*"
res2=$($cmd2)
echo "res2=$res2" # this produces "res2=a.txt"
Let's look at exactly what this does:
cmd1="find . -name 'a*' -print"
res1=$($cmd1)
echo "res1=$res1" # ??? THIS PRODUCES ONLY "res1=" , $res1 is blank ???
As per BashFAQ #50, execution of res1=$($cmd1) does the following, assuming you have no files with names starting with 'a and ending with ' (yes, with single quotes as part of the name), and that you haven't enabled the nullglob shell option:
res1=$( find . -name "'a*'" -print )
Note the quoting around than name? That quoting represents that the 's are treated as data, rather than syntax; thus, rather than having any effect on whether the * is expanded, they're simply an additional element required to be part any filename for it to match, which is why you get a result with no matches at all. Instead, as the FAQ tells you, use a function:
cmd1() {
find . -name 'a*' -print
}
res1=$(cmd1)
...or an array:
cmd1=( find . -name 'a*' -print )
res1=$( "${cmd1[#]}" )
Now, why does this happen? Read the FAQ for a full explanation. In short: Parameter expansion happens after syntactic quotes have already been applied. This is actually a Very Good Thing from a security perspective -- if all expansions recursively ran through full parsing, it would be impossible to write secure code in bash handling hostile data.
Now, if you don't care about security, and you also don't care about best practices, and you also don't care about being able to correctly interpret results with unusual filenames:
cmd1="find . -name 'a*' -print"
res1=$(eval "$cmd1") # Force parsing process to restart from beginning. DANGEROUS if cmd1
# is not static (ie. constructed with user input or filenames);
# prone to being used for shell injection attacks.
echo "res1=$res1"
...but don't do that. (One can get away with sloppy practices only until one can't, and the point when one can't can be unpleasant; for the sysadmin staff at one of my former jobs, that point came when a backup-maintenance script deleted several TB worth of billing data because a buffer overflow had placed random garbage in the name of a file that was due to be deleted). Read the FAQ, follow the practices it contains.

Issue saving result of "find" function in a shell script

I'm pretty new to shell scripting, but it's been great in helping me automating cumbersome tasks in OS X.
One of the functions I'm trying to write in a new script needs to find the specific filename in a subdirectory given a regex string. While I do know that the file exists, the version (and therefore filename itself) is being continually updated.
My function is currently as follows:
fname(){
$2=$(find ./Folder1 -name "$1*NEW*")
}
Which I'm then calling later in my script with the following line:
fname Type1 filename1
What I'm hoping to do is save the filename I'm looking for in variable filename1. My find syntax seems to be correct if I run it in Terminal, but I get the following error when I run my script:
./myscript.sh: line 13: filename1=./Folder1/Type1-list-NEW.bin: No such file or directory
I'm not sure why the result of find is not just saving to the variable I've selected. I'd appreciate any help (regardless of trivial this question may end up being). Thanks!
EDIT: I have a bunch of files in the subdirectory, but with the way I'm setting that folder up I know my "find" query will return exactly 1 filename. I just need the specific filename to do various tasks (updating, version checking, etc.)
The syntax for writing output to a file is command > filename. So it should be:
fname() {
find ./Folder1 -name "$1*NEW*" > "$2"
}
= is for assigning to a variable, not saving output in a file.
Are you sure you need to put the output in a file? Why not put it in a variable:
fname() {
find ./Folder1 -name "$1*NEW*"
}
var=$(fname Type1)
If you really want the function to take the variable name as a parameter, you have to use eval
fname() {
eval "$2='$(find ./Folder1 -name "$1*NEW*")'"
}
Okay, so I'm reading this as, you want to take the output of the find and save it in a shell variable given by $2.
You can't actually use a shell variable to dynamically declare the name of a new shell variable to rename, when the shell sees an expansion at the beginning of a line it immediately begins processing the words as arguments and not as an assignment.
There might be some way of pulling this off with declare and export but generally speaking you don't want to use a shell variable to hold n file names, particularly if you're on OS X, those file names probably have whitespaces and you're not protecting for that in the way your find is outputting.
Generally what you do in this case is you take the list of files find is spitting out and you act on them immediately, either with find -exec or as a part of a find . -print0 | xargs -0 pipeline.

Recursively dumping the content of a file located in different folders

Still being a newbie with bash-programming I am fighting with another task I got. A specific file called ".dump" (yes, with a dot in the beginning) is located in each folder and always contains three numbers. I need to dump the third number in a variable in case it is greater than 1000 and then print this and the folder name locating the number. So the outcome should look like this:
"/dir1/ 1245"
"/dir1/subdir1/ 3434"
"/dir1/subdir2/ 10003"
"/dir1/subdir2/subsubdir3/ 4123"
"/dir2/ 45440"
(without "" and each of them in a new line (not sure, why it is not shown correctly here))
I was playing around with awk, find and while, but the results are that bad that I do not wanna post them here, which I hope is understood. So any code snippet helping is appreciated.
This could be cleaned up, but should work:
find /dir1 /dir2 -name .dump -exec sh -c 'k=$(awk "\$3 > 1000{print \$3; exit 1}" $0) ||
echo ${0%.dump} $k ' {} \;
(I'm assuming that all three numbers in your .dump files appear on one line. The awk will need to be modified if the input is in a different format.)

Bash: find references to filenames in other files

Problem:
I have a list of filenames, filenames.txt:
Eg.
/usr/share/important-library.c
/usr/share/youneedthis-header.h
/lib/delete/this-at-your-peril.c
I need to rename or delete these files and I need to find references to these files in a project directory tree: /home/noob/my-project/ so I can remove or correct them.
My thought is to use bash to extract the filename: basename filename, then grep for it in the project directory using a for loop.
FILELISTING=listing.txt
PROJECTDIR=/home/noob/my-project/
for f in $(cat "$FILELISTING"); do
extension=$(basename ${f##*.})
filename=$(basename ${f%.*})
pattern="$filename"\\."$extension"
grep -r "$pattern" "$PROJECTDIR"
done
I could royally screw up this project -- does anyone see a flaw in my logic; better: do you see a more reliable scalable way to do this over a huge directory tree? Let's assume that revision control is off the table ( it is, in fact ).
A few comments:
Instead of
for f in $(cat "$FILELISTING") ; do
...
done
it's somewhat safer to write
while IFS= read -r f ; do
...
done < "$FILELISTING"
That way, your code will have no problem with spaces, tabs, asterisks, and so on in the filenames (though it still won't support newlines).
Your goal in separating f into extension and filename, and then reassembling them with \., seems to be that you want the filename to be treated as a literal string; right? Like, you're worried that grep will treat the . as meaning "any character" rather than as "one dot". A more general solution is to use grep's -F option, which tells it to treat the pattern as a fixed string rather than a regex:
grep -r -F "$f" "$PROJECTDIR"
Your introduction mentions using basename, but then you don't actually use it. Is that intentional?
If your non-use of basename is intentional, then filenames.txt really just contains a list of patterns to search for; you don't even need to write a loop, in this case, since grep's -f option tells it to take a newline-separated list of patterns from a file:
grep -r -F -f "$FILELISTING" "$PROJECTDIR"
You should back up your project, using something like tar -czf backup.tar.gz "$PROJECTDIR". "Revision control is off the table" doesn't mean you can't have a rollback strategy!
Edited to add:
To pass all your base-names to grep at once, in the hopes that it can do something smarter with them than just looping over them just as though the calls were separate, you can write something like:
grep -r -F "$(sed 's#.*/##g' "$FILELISTING")" "$PROJECTDIR"
(I used sed rather than while+basename for brevity's sake, but you can an entire loop inside the "$(...)" if you prefer.)
This is a job for an IDE.
You're right that this is a perilous task, and unless you know the build process and the search directories and the order of the directories, you really can't say what header is with which file.
Let's take something as simple as this:
# include "sql.h"
You have a file in the project headers/sql.h. Is that file needed? Maybe it is. Maybe not. There's also a /usr/include/sql.h. Maybe that's the one that's actually used. You can't tell without looking at the Makefile and seeing the order of the include directories which is which.
Then, there are the libraries that get included and may need their own header files in order to be able to compile. And, once you get to the C preprocessor, you really will have a hard time.
This is a task for an IDE (Integrated Development Environment). An IDE builds the project and tracks file and other resource dependencies. In the Java world, most people use Eclipse, and there is a C/C++ plugin for those developers. However, there are over 2 dozen listed in Wikipedia and almost all of them are open source. The best one will depend upon your environment.

Resources