I have files that has following format in directory:
SLS20160112.001 (20160112 stands for YYYYMMDD)
I wish to archive all previous month files, for example:
SLS20160201.001
SLS20150201.001
SLS20160107.001
SLS20160130.001
For the above files listed, i will archive SLS20160107.001 and SLS20160130.001 because from the filename it stamps January.
For the SLS20160201.001 it still remains as i only want to archive previous month file. I can only extract date from the filename, not the mdate or adate.
My current logic is to loop through all files, then get previous month files and then pipe out the filename and tar it. But not sure how to do that part.
for file in SLS*; do
f="${file%.*}"
GET PREVIOUS MONTH FILES AND THEN ECHO
done | tar cvzf SlSBackup_<PREVIOUS_MONTH>.TAR.GZ -T-
It looks like you want to solve the problem by using a shell script. I do a lot of work on Mac so I use csh/tcsh (default shell for OSX), and my answer will be in csh/tcsh script. You can either translate it to bash (your shell) or you can easily spawn a new shell by just typing$ tcsh.
You can write a small shell script which can filter filelist for your desired month.
#!/bin/csh -f
set mon_wanted = '01'
foreach file (`ls -1 SLS*.*`)
set mon = `echo $file | awk '{print substr($0, 8, 2)}'`
if ($mon != $mon_wanted) continue
echo -n $file ' '
end
Let's say the filename is foo.csh. Make it executable by
$ chmod 755 foo.csh
Then,
$ tar cvzf `foo.csh` > out.tar.gz
Related
Everday we get a new backup dump of a database eg:
thisfile.0.db2v22.DODE0000.CATN00000.20180627132924.001
thisfile.0.db2v22.DODE0000.CATN00000.20180628132924.001
thisfile.0.db2v22.DODE0000.CATN00000.20180629132924.001
and from that dump is the date which is in the 6th position of the file name eg: 20180627132924.
I need to write a script that will strip the date and time eg: 20180627132924 from that file in the folder and insert into a restore script.
How do I grab this date so I can add it as a variable within my restore script?
maybe something like :
OUTPUT="$(ls -l *.001 | awk -F '[_.]' '{print $6}')"
echo " restore $(OUTPUT) to this" >>restore
chmod 700 restore
./restore
In case of the filename you propose, I would suggest to use pure Bash built-in functions :
$ file="thisfile.0.db2v22.DODE0000.CATN00000.20180627132924.001"
$ fname=${file%.*}
$ fname=${fname##*.}
$ echo $fname
20180627132924
$ echo "This is the file name ${fname} and it has been backup." > outputfile
I want to list specified files (files uploaded yesterday) from an amazon S3.
Then I want to loop on this list, and for every single element of the list I want to unzip the file.
My code is:
for file in s3cmd ls s3://my-bucket/`date +%Y%m%d -d "1 day ago"`*
do s3cmd get $file
arrIN=(${file//my-bucket\//})
gunzip ${arrIN[1]}
done
so basicaly arrIN=(${file//my-bucket//}); explodes my string and allow me to retrieve the name of the file I want to unzip.
Thing is, file are downloading but nothing is being unzip, so I tried:
for file in s3cmd ls s3://my-bucket/`date +%Y%m%d -d "1 day ago"`*
do s3cmd get $file
echo test1
done
Files are being downloaded but nothing is being echo. Loop is just working for the first line...
You need to use command substitution to iterate over the result of the desired s3smd ls command.
for file in $(s3cmd ls s3://my-bucket/$(date +%Y%m%d -d "1 day ago")*); do
However, this isn't the preferred way to iterate over the output of a command, since in theory the results could contain whitespace. See Bash FAQ 001 for the proper method.
I'm trying to automate some processes by using hazel app to move a file to a specific folder, execute a shell script on any csv in that folder, and then move it to another folder. Right now the part I'm working on is the shell script. I have been testing out the cut command in terminal on csvs but I'm not sure if its the same thing as a shell script since it doesnt seem to be working but what I have is:
cut -d',' -f2,12 test.csv > campaigns-12-31-13.csv
It looks for test.csv but I would like it to work with any csv, and it also exports it with the date 12-31-13 but I'm just trying to get it to export with whatever yesterdays date was.
How do I convert this to a shell script that will execute on any csv in the folder and so it adds the date for yesterday at the end of the filename?
You can try the following script:
#! /bin/bash
saveDir="saveCsv"
dd=$(date +%Y-%m-%d -d "yesterday")
for file in *.csv ; do
bname=$(basename "$file" .csv)
saveName="${saveDir}/${bname}-${dd}.csv"
cut -d',' -f2,12 "$file" > "$saveName"
done
I'm writing a deployment script that saves timestamped backup files to a backups directory. I'd like to do a rollback implementation that would roll back to the most recent file.
My backups directory:
$:ls
. 1341094065_public_html_bu 1341094788_public_html_bu
.. 1341094390_public_html_bu
1341093920_public_html_bu 1341094555_public_html_bu
I want to identify the most recent file (by timestamp in the filename) in the backup directory, and save its name to a variable, then cp it to ../public_html, and so on...
ls -t will sort files by mtime. ls -t | head -n1 will select the newest file. This is independent of any naming scheme you have, which may or may not be a plus.
...and a more "correct" way, which won't break when filenames contain newlines, and also not when there are no matching files (unexpanded glob results)
for newestfile in ./* ; do : ; done
if test -e "$newestfile"; then do something with "$newestfile" ; fi
The latest-timestamped filename should sort last alphabetically. So you can then use tail -n1 to extract it.
For files that don't have newlines in their names:
shopt -s nullglob
printf '%s\n' "$buDir"/* | tail -n 1
I am trying to write a bash script that goes through a file line by line (ignoring the header), extracts a file name from the beginning of each line, and then finds a file by this name in one directory and moves it to another directory. I will be processing hundreds of these files in a loop and moving over a million individual files. A sample of the file is:
ImageFileName Left_Edge_Longitude Right_Edge_Longitude Top_Edge_Latitude Bottom_Edge_Latitude
21088_82092.jpg: -122.08007812500000 -122.07733154296875 41.33763821961143 41.33557596965434
21088_82093.jpg: -122.08007812500000 -122.07733154296875 41.33970040427444 41.33763821961143
21088_82094.jpg: -122.08007812500000 -122.07733154296875 41.34176252364274 41.33970040427444
I would like to ignore the first line and then grab 21088_82092.jpg as a variable. File names may not always be the same length, but they will always have the format digits_digits.jpg
Any help for an efficient approach is much appreciated.
This should get you started:
$ tail -n +2 input | cut -f 1 -d: | while read file; do test -f $dir/$file && mv -v $dir/$file $destination; done
You can construct a script that will do something like this, then simply run the script. The following command will give you a script which will copy the files from one place to another, but you can make the script generation more complex simply by changing the awk output:
pax:~$ cat qq.in
ImageFileName Left_Edge_Longitude Right_Edge_Longitude
21088_82092.jpg: -122.08007812500000 -122.07733154296875
21088_82093.jpg: -122.08007812500000 -122.07733154296875
21088_82094.jpg: -122.08007812500000 -122.07733154296875
pax:~$ awk -F: '/^[0-9]+_[0-9]+.jpg:/ {
printf "cp /srcdir/%s /dstdir\n",$1
} {}' qq.in
cp /srcdir/21088_82092.jpg /dstdir
cp /srcdir/21088_82093.jpg /dstdir
cp /srcdir/21088_82094.jpg /dstdir
You capture the output of that script (the last three lines) to another file then that file is your script for doing the actual copies.