My task is log rotation and I can't find any command which can some number from filename with 1.
For example, I have some files with name: wrapper.log.1, wrapper.log.2.
I need to rename and move that files to other directory and get wrapper_1.log, wrapper_2.log. After file was moved it should be deleted from origin directory.
It is possible, that in new folder there are files with the same name.
So, I should get last file and plus 1 to its filename like wrapper_(2+1).log.
For whole my task I found something like
find . -name "wrapper.log.*"
mkdir $(date '+ %d.%m.%y')
find . -name "wrapper.log.*" |sort -r |head -n1 | sed -E 's/(.log)(.[0-9])/_$(2+1)\1/'
But, of course, it doesn`t work after second line.
And, in future, it needs to be in bash.
P.S: Also, I think, It is possible to create just new file in new folder with timestamp or smth like that as postfix.
For example:
folder file
01.01.19 wrapper_00_00_01
wrapper_00_01_07
wrapper_01_10_53
wrapper_13_07_11
02.01.19
wrapper_01_00_01
wrapper_03_01_07
wrapper_05_10_53
wrapper_13_07_11
To find the highest number of the wrapper_ log files:
find . -type f -name "*.log" -exec basename {} \; | ggrep -Po '(?<=wrapper_)[0-9]' | sort -rn | head -n 1
I'm using grep's pearl switch to do a look-behind for "wrapper_", then reverse sorting the numbers found and taking the first one. If you want to generate a new file name, I'd use awk, e.g:
find . -type f -name "*.log" -exec basename {} \; | ggrep -Po '(?<=wrapper_)[0-9]' | sort -rn | head -n 1 | awk '{print "wrapper_"$1 + 1".log" }'
This will produce a file name with the next number in the sequence.
I don't understand your question entirely but I know that using a dollar-sign and double brackets you can execute a calculation:
Prompt>echo $((1+1))
2
Finally, I found two solutions.
First, it`s bash, smth like this
#!/bin/bash
#DECLARE
FILENAME=$1
DATE=$(date '+%d.%m.%y')
SRC_DIR="/usr/local/apache-servicemix-6.1.0/data/log"
DEST_DIR="/mnt/smxlog/$HOSTNAME"
#START
mkdir -m 777 "$DEST_DIR/$DATE"
if [ -d "$DEST_DIR/$DATE" ]
then
for f in $(find "$SRC_DIR/" -name "$FILENAME.log.*")
do
TIME=$(date '+%H_%M_%S.%3N')
NEW_FILENAME="$FILENAME-$TIME.log"
NEW_DEST_WITH_FILE="$DEST_DIR/$DATE/$NEW_FILENAME"
mv $f $NEW_DEST_WITH_FILE
gzip "$NEW_DEST_WITH_FILE"
done
else
exit 1
fi
#END
And the second variant is using log4j logger properties, but it should to upload to servicemix system folder log4j-1.2.17_fragment.jar and apache-log4j-extras-1.2.17_fragment. May be it is possible to upload them as bundle, I didn`t try.
Both jar use different API. There are
https://logging.apache.org/log4j/1.2/apidocs/index.html?overview-summary.html and
http://logging.apache.org/log4j/companions/apidocs/index.html?overview-summary.html
And properties will be
log4j.logger.wrapper.log=DEBUG, wrapper
log4j.additivity.logger.wrapper.log=false
log4j.appender.wrapper=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.wrapper.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
#This setting should be used with commented line log4j.appender.wrapper.File=... if it needs to zip to target directory immediately
#log4j.appender.wrapper.rollingPolicy.FileNamePattern=/mnt/smxlog/${env:HOSTNAME}/wrapper.%d{HH:mm:ss}.log.gz
#Or it is possible to log and zip in the same folder, and after that with cron replace zipped files to required folder
log4j.appender.wrapper.rollingPolicy.FileNamePattern=${karaf.data}/log/wrapper.%d{HH:mm:ss}.log.gz
log4j.appender.wrapper.File=${karaf.data}/log/wrapper.log
log4j.appender.wrapper.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
#Size in bytes
log4j.appender.wrapper.triggeringPolicy.MaxFileSize=1000000
log4j.appender.wrapper.layout=org.apache.log4j.PatternLayout
log4j.appender.wrapper.layout.ConversionPattern=%d{dd-MM-yyyy_HH:mm:ss} %-5p [%t] - %m%n
log4j.appender.wrapper.Threshold=DEBUG
log4j.appender.wrapper.append=true
Related
I have a directory with more than 20K files all with a random number prefix (eg 12345--name.jpg). I want to find files with similar names and remove all but one. I don't care which one because they are duplicates.
To find duplicated names I've use
find . -type f \( -name "*.jpg" \) | | sed -e 's/^[0-9]*--//g' | sort | uniq -d
as the list of a for/next loop.
To find all but one to delete, I'm currently using
rm $(ls -1 *name.jpg | tail -n +2)
This operation is pretty slow. I want to speed this up. Any suggestions?
I would do it like this.
*Note that you are dealing with rm command, so make sure that you have backup of the existing directory in case something goes south.
Create a backup directory and take backup of existing files. Once done check if all the files are there.
mkdir bkp_dir;cp *.jpg /bkp_dir
Create another temp directory where we will keep all only 1 file for each similar name. So all unique file names will be here.
$ mkdir tmp
$ for i in $(ls -1 *.jpg|sed 's/^[[:digit:]].*--\(.*\.jpg\)/\1/'|sort|uniq);do cp $(ls -1|grep "$i"|head -1) tmp/ ;done
*Explanation of the command is at the last. Once executed, check in /tmp directory if you got unique instances of the files.
Remove all *.jpg files from main directory. Saying again, please verify that all files have been backed up before executing rm command.
rm *.jpg
Backup the unique instances from the temp directory.
cp tmp/*.jpg .
Explanation of command in step 2.
Command to get unique file names for step 2 will be
for i in $(ls -1 *.jpg|sed 's/^[[:digit:]].*--\(.*\.jpg\)/\1/'|sort|uniq);do cp $(ls -1|grep "$i"|head -1) tmp/ ;done
$(ls -1 *.jpg|sed 's/^[[:digit:]].*--\(.*\.jpg\)/\1/'|sort|uniq) will get the unique file names like file1.jpg , file2.jpg
for i in $(...);do cp $(ls -1|grep "$i"|head -1) tmp/ ;done will copy one file for each filename to tmp/ directory.
You should not be using ls in scripts and there is no reason to use a separate file list like in userunknown's reply.
keepone () {
shift
rm "$#"
}
keepone *name.jpg
If you are running find to identify the files you want to isolate anyway, traversing the directory twice is inefficient. Filter the output from find directly.
find . -type f -name "*.jpg" |
awk '{ f=$0; sub(/^[0-9]*--/, "", f); if (a[f]++) print }' |
xargs echo rm
Take out the echo if the results look like what you expect.
As an aside, the /g flag to sed is useless for a regex which can only match once. The flag says to replace all occurrences on a line instead of the first occurrence on a line, but if there can be only one, the first is equivalent to all.
Assuming no subdirectories and no whitespace-in-filenames involved:
find . -type f -name "*.jpg" | sed -e 's/^[0-9]*--//' | sort | uniq -d > namelist
removebutone () { shift; echo rm "$#"; }; cat namelist | while read n; do removebutone "*--$n"; done
or, better readable:
removebutone () {
shift
echo rm "$#"
}
cat namelist | while read n; do removebutone "*--$n"; done
Shift takes the first parameter from $* off.
Note that the parens around the name parmeter are superflous, and that there shouldn't be two pipes before sed. Maybe you had something else there, which needed to be covered.
If it looks promising, you have, of course, to remove the 'echo' in front of 'rm'.
Bash newbie here trying to insert the name of a folder into certain files inside that folder.
The problem is that these files are in subfolders of subfolders of the main directory, and the names of each level are different in each case.
For example, the main folder interviews may contain John Doe and under John Doe is a directory Images with a file Screenshot.jpg. But there might also be John Smith with a folder Etc in which is 12_Screenshot 2.jpg.
I want to rename all these files containing Screenshot inserting John Doe or John Smith before the filename.
I tried adapting a couple of scripts I found and ran them from the interviews directory:
for i in `ls -l | egrep '^d'| awk '{print $10}'`; do find . -type f -name "*Screenshot*" -exec sh -c 'mv "$0" "${i}${0}"' '{}' \; done
after which the terminal gives the caret prompt, as if I'm missing something. I also tried
find -regex '\./*' -type d -exec mv -- {}/*/*Screenshot* {}/{}.jpg \; -empty -delete
which returns find: illegal option -- r
The fact that the second one theoretically moves the file up to the parent folder is not a problem since I'll have to do this eventually anyways.
The following script will work as desired :
dir=$1
find $dir -name "*Screenshot*" -type f | while read file
do
base=$(basename $file)
dirpath=$(dirname $file)
extr=$(echo $file | awk -F/ '{print $(NF-2)}') #extracts the grandparent directory
mv $file $dirpath/$extr-$base
done
As #loneswap mentioned, this must be invoked as a script. So if your main directory is mainDir, then you would invoke it as so...
./script mainDir
For each directory in current working directory, recursively find files containing string "screenshot" (case insensitive due to OSX). Split the found path into parent part (always present at least in form './') and file name, produce two lines one original file path, second one original folder + modified target file name. Execute mv command via xargs using two arguments (separate by newline to allow whitespaces in paths):
for i in `ls -l | sed -n '/^d\([^[:space:]]\+[[:space:]]\+\)\+\([^[:space:]]\+\)$/s//\2/p'`; do
find "$i" -type f -iname "*Screenshot*" \
| sed -n '\!^\(\([^/]\+/\)\+\)\([^/]\+\)$!s!!\1\3\n\1'$i'\3!p' \
| xargs -d '\n' -n 2 mv;
done
Drawback: xargs on OSX does not know --no-run-if-empty, so for directories that do not contain files with "screenshot" string empty mv is invoked. Proper option needs to be added (don't have access to OSX man pages) or xargs ... 2>&/dev/null to ignore all errors...
Basically I have a directory and sub-directories that needs to be scanned to find .csv files. From there I want to copy all lines containing "foo" from the csv's found to new files (in the same directory as the original) but with the name reflecting the file it was found in.
So far I have
find -type f -name "*.csv" | xargs egrep -i "foo" > foo.csv
which yields one backup file (foo.csv) with everything in it, and the location it was found in is part of the data. Both of which I don't want.
What I want:
For example if I have:
csv1.csv
csv2.csv
and they both have lines containing "foo", I would like those lines copied to:
csv1_foo.csv
csv2_foo.csv
and I don't anything extra entered in the backups, other than the full line containing "foo" from the original file. I.e. I don't want the original file name in the backup data, which is what my current code does.
Also, I suppose I should note that I'm using egrep, but my example doesn't use regex. I will be using regex in my search when I apply it to my specific scenario, so this probably needs to be taken into account when naming the new file. If that seems too difficult, an answer that doesn't account for regex would be fine.
Thanks ahead of time!
try this if helps it anyway.
find -type f -name "*.csv" | xargs -I {} sh -c 'filen=`echo {} | sed 's/.csv//' | sed "s/.\///"` && egrep -i "foo" {} > ${filen}_foo.log'
You can try this:
$ find . -type f -exec grep -H foo '{}' \; | perl -ne '`echo $2 >> $1_foo` if /(.*):(.*)/'
It uses:
find to iterate over files
grep to print file path:line tuples (-H switch)
perl to echo those line to the output files (using backslashes, but it could be done prettier).
You can also try:
find -type f -name "*.csv" -a ! -name "*_foo.csv" | while read f; do
grep foo "$f" > "${f%.csv}_foo.csv"
done
I have a directory containing hundreds of files (each having several chars). I want to join them into a single file with a separator, "|".
I tried
find . -type f | (while read line; do; cat $line; echo "|"; done;) > output.txt
But that created an infinite loop.
You can exclude output.txt from the output of find using -not -name output.txt (or as you already pointed out in the comments below, simply place the output file outside the target directory).
For example:
find . -type f -not -name output.txt -exec cat {} \; -exec echo "|" \; > output.txt
I've also taken the liberty to replace your while/cat/echo with a couple of -exec params so we can do the whole thing using a single find call.
*To answer the title of the question, since it's the first in google results (the output.txt problem is actually unrelated):
This is what I use to join .jar files to run Java app with files in lib/:
EntityManagerStoreImpl
ondra#lenovo:~/work/TOOLS/JawaBot/core$ ls
catalog.xml nbactions.xml nb-configuration.xml pom.xml prepare.sh resources run.sh sql src target workdir
ondra#lenovo:~/work/TOOLS/JawaBot/core$ echo `ls -1` | sed 's/\W/:/g'
catalog:xml:nbactions:xml:nb:configuration:xml:pom:xml:prepare:sh:resources:run:sh:sql:src:target:workdir
The file listing may be of course replaced with find ... or anything.
The echo is there to replace newlines with spaces.
Final form:
java -cp $(echo `ls -1 *.jar` | sed 's/\W/:/g') com.foo.Bar
I reused Ondra's answer, but with absolute path instead.
Command :
echo $( \find '/home/user/[path-to-webapp]/WEB-INF/lib' -name '*.jar' -print0) | \
sed 's#\.jar/#.jar:#g'
Note: I use # as sed's separator to not match the last jar in the list.
Results:
/home/user/[path-to-webapp]/WEB-INF/lib/jar1.jar:home/user/[path-to-webapp]/WEB-INF/lib/jar2.jar[...
and so
on...]:/home/user/[path-to-webapp]/WEB-INF/lib/last-jar.jar
Then, I can use this output in a javac -classpath command.
I want to copy the files from the most recent directory created. How would I do so in unix?
For example, if I have the directories names as date stamp as such:
/20110311
/20110318
/20110325
This is the answer to the question I think you are asking.
When I deal with many directories that have date/time stamps in the name, I always take the approach that you have which is YYYYMMDD - the great thing about that is that the date order is then also the alphabetical order. In most shells (certainly in bash and I am 90% sure of the others), the '*' expansion is done alphabetically, and by default 'ls' return alphabetical order. Hence
ls | head -1
ls | tail -1
Give you the earliest and the latest dates in the directory.
This can be extended to only keep the last 5 entries etc.
lastdir=`ls -tr <parentdir> | tail -1`
I don't know how to make the backticks play nice with the commenting system here. Just replace those apostrophes with backticks.
After some experimenting, I came up with the following:
The unix stat command is useful here. The '-t' option causes stat to print its output in terse mode (all in one line), and the 13th element of that terse output is the unix timestamp (seconds since epoch) for the last-modified time. This command will list all directories (and sub-directories) in order from newest-modified to oldest-modified:
find -type d -exec stat -t {} \; | sort -r -n -k 13,13
Hopefully the "terse" mode of stat will remain consistent in future releases of stat !
Here's some explanation of the command-line options used:
find -type d # only find directories
find -exec [command] {} \; # execute given command against each *found* file.
sort -r # reverse the sort
sort -n # numeric sort (100 should not appear before 2!)
sort -k M,N # only sort the line using elements M through N.
Returning to your original request, to copy files, maybe try the following. To output just a single directory (the most recent), append this to the command (notice the initial pipe), and feed it all into your 'cp' command with backticks.
| head --lines=1 | sed 's/\ .*$//'
The trouble with the ls based solutions is that they are not filtering just for directories. I think this:
cp `find . -mindepth 1 -maxdepth 1 -type d -exec stat -c "%Y %n" {} \; |sort -n -r |head -1 |awk '{print $2}'`/* /target-directory/.
might do the trick, though note that that will only copy files in the immediate directory. If you want a more general answer for copying anything below your newest directory over to a new directory I think you would be better off using rsync like:
rsync -av `find . -mindepth 1 -maxdepth 1 -type d -exec stat -c "%Y %n" {} \; |sort -n -r |head -1 |awk '{print $2}'`/ /target-directory/
but it depends a bit which behaviour you want. The explanation of the stuff in the backticks is:
. - the current directory (you may want to specify an absolute path here)
-mindepth/-maxdepth - restrict the find command only to the immediate children of the current directory
-type d - only directories
-exec stat .. - outputs the modified time and the name of the directory from find
sort -n -r |head -1 | awk '{print $2}' - date orders the directory and outputs the name of the most recently modified
If your directories are named YYYYMMDD like your question suggests, take advantage of the alphabetic globbing.
Put all directories in an array, and then pick the first one:
dirs=(*/); first_dir="$dirs";
(This is actually a shortcut for first_dir="${dirs[0]}";.)
Similarly, for the last one:
dirs=(*/); last_dir="${dirs[$((${#dirs[#]} - 1))]}";
Ugly syntax, but this is what it breaks down to:
# Create an array of all directories inside the working directory.
dirs=(*/);
# Get the number of entries in the array.
num_dirs=${#dirs[#]};
# Calculate the index of the last entry.
last_index=$(($num_dirs - 1));
# Get the value at the last index.
last_dir="${dirs[$last_index]}";
I know this is an old question with an accepted answer, but I think this method is preferable as it does everything in Bash. No reason to spawn extra processes, let alone parse the output of ls. (Which, admittedly, should be fine in this particular case of YYYYMMDD names.)
please try with following command
ls -1tr | tail -1
find ~ -type d | ls -ltra
This one is simple and useful which I learned recently.
This command will show the results in reverse chronological order.
I wrote a command that can be used to identify which folder or files are created in a folder as a newest. That's seems pure :)
#/bin/sh
path=/var/folder_name
newest=`find $path -maxdepth 1 -exec stat -t {} \; |sed 1d |sort -r -k 14 | head -1 |awk {'print $1'} | sed 's/\.\///g'`
find $path -maxdepth 1| sed 1d |grep -v $newest