Parse and rename date in filename - shell

I have some files like this:
XXXXXXXX-15Jan2014.pdf
XXXXXXXX-15Jan2015.pdf
XXXXXXXX-22Aug2013.pdf
I'd like to rename them to
XXXXXXXX-2014-01-15.pdf
XXXXXXXX-2015-01-15.pdf
XXXXXXXX-2013-08-22.pdf
That is, rename <day><month><year> to <year>-<month-as-number>-<day>.
Is there a super easy linux/osx command to do this?

I cant say about any command.
Try this script.
#!/bin/bash
for file in `ls`
do
FirstPart=`echo $file|rev|cut -d"-" -f2|rev`
MON=`echo $file|rev|cut -d"." -f2|rev`
date=`echo $file|rev|cut -d"-" -f1|rev`
read date year <<<${date//[^0-9]/ }
echo "$MON" | grep -i "jan" && mon=1
echo "$MON" | grep -i "feb" && mon=2
echo "$MON" | grep -i "mar" && mon=3
echo "$MON" | grep -i "apr" && mon=4
echo "$MON" | grep -i "may" && mon=5
echo "$MON" | grep -i "jun" && mon=6
echo "$MON" | grep -i "jul" && mon=7
echo "$MON" | grep -i "aug" && mon=8
echo "$MON" | grep -i "sep" && mon=9
echo "$MON" | grep -i "oct" && mon=10
echo "$MON" | grep -i "nov" && mon=11
echo "$MON" | grep -i "dec" && mon=12
newName="$FirstPart-$year-$mon-$date"
mv $file $newName
done

awk can help:
awk -F'[-.]' '{c="date -d" $2 " +%F"; c|getline d;close(c);printf "mv %s %s-%s.pdf\n",$0,$1, d}'
the above one-liner will generate the mv command line to do the renaming. What you can try is:
ls -1 *.pdf|awk -F'[-.]' '{c="date -d" $2 " +%F"; c|getline d;close(c);printf "mv %s %s-%s.pdf\n",$0,$1, d}'
You will see the mv ... commands as output, to execute those statements to achieve your renaming goal, just pipe the command to |sh, that is:
ls ..|awk..|sh
I picked one of your file as example:
kent$ ls -1 *.pdf
XXXXXXXX-15Jan2014.pdf
kent$ ls -1 *.pdf|awk -F'[-.]' '{c="date -d" $2 " +%F"; c|getline d;close(c);printf "mv %s %s-%s.pdf\n",$0,$1, d}'|sh
kent$ ls -1 *.pdf
XXXXXXXX-2014-01-15.pdf

I don't have an OSX box to test on, but this should work:
while read filename; do
prefix=${filename%-*}
ext=${filename##*.}
date=${filename##*-}; date=${date%.*}
echo mv "$filename" "$prefix-$(date -jf "%d%b%Y" "$date" "+%Y-%m-%d").$ext"
done <<END
XXXXXXXX-15Jan2014.pdf
XXXXXXXX-15Jan2015.pdf
XXXXXXXX-22Aug2013.pdf
END

Related

How can I use wget to download specific files in a CSV file, and then store those files into specific directories?

I have been attempting to extract a CSV file full of URL's of images (about 1000).
Each row is a specific product with the first cell labelled "id".
I have taken the ID of each line in excel and created directories for them using a loop with mkdir.
My issue now is that I can't seem to figure out how to download the image, and then immediately store it into these folder's.
What I am attempting here is to use wget by concatenating "fold_name" and "EXT" to get it like a directory "/name_of_folder", and then getting the links to the images (in cell 5,6,7 and 8) and then using wget from these cells, into the directory.
Can anyone assist me with this?
I think this should be straight forward enough.
Thank you!
#!/usr/bin/bash
EXT='/'
while read line
do
fold_name= cut -d$',' -f1
concat= "%EXT" + "%fold_name"
img1= cut -d$',' -f5
img2= cut -d$',' -f6
img3= cut -d$',' -f7
img4= cut -d$',' -f8
wget -O "%img1" "%concat"
wget -O "%img2" "%concat"
wget -O "%img1" "%concat"
wget -O "%img2" "%concat"
done < file.csv
You might use -P switch to designate target directory, consider following simple example using some files from test-images/png repository
mkdir -p black
mkdir -p gray
mkdir -p white
wget -P black https://raw.githubusercontent.com/test-images/png/main/202105/cs-black-000.png
wget -P gray https://raw.githubusercontent.com/test-images/png/main/202105/cs-gray-7f7f7f.png
wget -P white https://raw.githubusercontent.com/test-images/png/main/202105/cs-white-fff.png
will lead to following structure
black
cs-black-000.png
gray
cs-gray-7f7f7f.png
white
cs-white-fff.png
You should use variables names that are less ambiguous.
You need to provide the directory as part of the output filename.
"%" is not a bash variable designator. That is a formatting directive (for bash, awk, C, etc.).
The following will provide what you want.
#!/usr/bin/bash
DBG=1
INPUT="${1}"
INPUT="file.csv"
cat >"${INPUT}" <<"EnDoFiNpUt"
#topic_1,junk01,junk02,junk03,img_101.png,img_102.png,img_103.png,img_104.png
#topic_2,junk04,junk05,junk06,img_201.png,img_202.png,img_203.png,img_204.png
#
topic_1,junk01,junk02,junk03,https://raw.githubusercontent.com/test-images/png/main/202105/cs-black-000.png,https://raw.githubusercontent.com/test-images/png/main/202105/cs-gray-7f7f7f.png,https://raw.githubusercontent.com/test-images/png/main/202105/cs-white-fff.png
EnDoFiNpUt
if [ ${DBG} -eq 1 ]
then
echo -e "\n Input file:"
cat "${INPUT}" | awk '{ printf("\t %s\n", $0 ) ; }'
echo -e "\n Hit return to continue ..." ; read k
fi
REPO_ROOT='/tmp'
grep -v '^#' "${INPUT}" |
while read line
do
topic_name=$(echo "${line}" | cut -f1 -d\, )
test ${DBG} -eq 1 && echo -e "\t topic_name= ${topic_name} ..."
folder="${REPO_ROOT}/${topic_name}"
test ${DBG} -eq 1 && echo -e "\t folder= ${folder} ..."
if [ ! -d "${folder}" ]
then
mkdir "${folder}"
else
rm -f "${folder}/"*
fi
if [ ! -d "${folder}" ]
then
echo -e "\n Unable to create directory '${folder}' for saving downloads.\n Bypassing 'wget' actions ..." >&2
else
test ${DBG} -eq 1 && ls -ld "${folder}" | awk '{ printf("\n\t %s\n", $0 ) ; }'
url1=$(echo "${line}" | cut -d\, -f5 )
url2=$(echo "${line}" | cut -d\, -f6 )
url3=$(echo "${line}" | cut -d\, -f7 )
url4=$(echo "${line}" | cut -d\, -f8 )
test ${DBG} -eq 1 && {
echo -e "\n URLs extracted:"
echo -e "\n\t ${url1}\n\t ${url2}\n\t ${url3}\n\t ${url4}"
}
#imageFile1=$( basename "${url1}" | sed 's+^img_+yourImagePrefix_+' )
#imageFile2=$( basename "${url2}" | sed 's+^img_+yourImagePrefix_+' )
#imageFile3=$( basename "${url3}" | sed 's+^img_+yourImagePrefix_+' )
#imageFile4=$( basename "${url4}" | sed 's+^img_+yourImagePrefix_+' )
imageFile1=$( basename "${url1}" | sed 's+^cs-+yourImagePrefix_+' )
imageFile2=$( basename "${url2}" | sed 's+^cs-+yourImagePrefix_+' )
imageFile3=$( basename "${url3}" | sed 's+^cs-+yourImagePrefix_+' )
test ${DBG} -eq 1 && {
echo -e "\n Image filenames assigned:"
#echo -e "\n\t ${imageFile1}\n\t ${imageFile2}\n\t ${imageFile3}\n\t ${imageFile4}"
echo -e "\n\t ${imageFile1}\n\t ${imageFile2}\n\t ${imageFile3}"
}
test ${DBG} -eq 1 && {
echo -e "\n WGET process log:"
}
### This form of wget does NOT work for me, although man page says it should.
#wget -P "${folder}" -O "${imageFile1}" "${url1}"
### This form of wget DOES work for me
wget -O "${folder}/${imageFile1}" "${url1}"
wget -O "${folder}/${imageFile2}" "${url2}"
wget -O "${folder}/${imageFile3}" "${url3}"
#wget -O "${folder}/${imageFile3}" "${url3}"
test ${DBG} -eq 1 && {
echo -e "\n Listing of downloaded files:"
ls -l /tmp/topic* 2>>/dev/null | awk '{ printf("\t %s\n", $0 ) ; }'
}
fi
done
The script is adapted for what I had to work with. :-)

Read data from log file and put to file name

#!/bin/bash
title=$(echo "$1" | sed "s/.*\///" | cut -f 1 -d '.')
function _ask() {
while [[ $url == "" ]]; do
echo ; echo -e "Wklej link do filmu:" ; read -e url
done
}
napi.sh search -k movie "$title"
_ask
napi.sh subtitles "$url" > napi.log
echo Pobieram napisy:
napi.sh download -e srt `grep -o 'napiprojekt:.*' napi.log`
exit
napi.log
00:0001 - wywolano o pią, 7 maj 2021, 22:04:42 CEST
00:0002 - system: linux, forkow: 32, wersja: v2.0.0
00:0003 - Przetwarzam: [http://napiprojekt.pl/napisy-910-Sissi-młoda-cesarzowa-(1956)]
Rozmiar: 732258304 bajtow | fps: 25 | napiprojekt:d40a1ef492e0dd094bc42141fd6e2dba
Rozmiar: 733724672 bajtow | fps: 25 | napiprojekt:38631781c8d9420eadfa13e9fe5d803b
Rozmiar: 733888512 bajtow | fps: 25 | napiprojekt:626e99378154981617418da4b39a098f
Rozmiar: 733685760 bajtow | fps: 25 | napiprojekt:9b9787e3e57754fb4301d10e865efdad
Rozmiar: 733751296 bajtow | fps: 25 | napiprojekt:f066e7a2da79ef924793212f52f0afae
Rozmiar: 855366544 bajtow | fps: 25 | napiprojekt:bbc652046e94f1802d6bea0e7d21643e
Rozmiar: 4469515020 bajtow | fps: 23.976 | napiprojekt:9470761b635b733c2befab2df62f2672
Rozmiar: 796309304 bajtow | fps: 25 | napiprojekt:a385be50baeffaad680fb03c8c0e8ede
My script and another napi.sh https://gitlab.com/hesperos/napi I use to download all subs for selected movie.
napi.sh download -e srt `grep -o 'napiprojekt:.*' napi.log`
this code help me to read all ids from log file.
napi.sh download -e srt napiprojekt:a385be50baeffaad680fb03c8c0e8ede napiprojekt:9470761b635b733c2befab2df62f2672 ...
and napi.sh save files name as id name:
d40a1ef492e0dd094bc42141fd6e2dba.srt
626e99378154981617418da4b39a098f.srt
a385be50baeffaad680fb03c8c0e8ede.srt
Someone can help me to add to file name information about fps, this data can be received from napi.log compare id with file name and add?
Greetings.
Final script thx for help #Zilog80
#!/bin/bash
title=$(echo "${1%.*}" | sed "s/.*\///" )
function _ask() {
while [[ $url == "" ]]; do
echo ; echo -e "Wklej link do filmu:" ; read -e url
done
}
napi.sh search -k movie "$title"
_ask
mkdir "$title"
cd "$title"
napi.sh subtitles "$url" > napi.log
echo Pobieram napisy:
napi.sh download -e srt `grep -o 'napiprojekt:.*' napi.log`
# Command to rename the files with the fps
awk -F'|' '/napiprojekt:/ { fps=$2;gsub(/^[^0-9\.]*/,"", fps); gsub(/ /,"", fps);
filename=$3; gsub(/^ *napiprojekt:/,"",filename); gsub(/ /,"\\ ",filename);
print "mv "filename".srt FPS_"fps"_"filename".srt"}' napi.log | while read move_cmd; do
[ ! -z "${move_cmd}" ] && ${move_cmd};
done
for f in *.srt
do
if [ "$charset" != 'utf-8' ]
then
echo Koduje do UTF-8:
iconv -f windows-1250 -t UTF-8 "$f" -o "$f"_utf8
mv -f "$f"_utf8 "$f"
fi
done
echo Dodaje tytuł do nazwy:
for f in *.srt; do mv -f "$f" ../"$title"_"$f"; done
cd ..
rm -r -f "$title"
exit
I guess you're looking for an extract of napi.log that will return the napi project including the fps number.
Use then awk -F'\|' '/napiprojekt:/ { fps=$2;gsub(/^[^0-9\.]*/,"", fps); print $3"_"fps}' napi.log instead of grep -o 'napiprojekt:.*' napi.log :
napi.sh download -e srt `awk -F'\|' '/napiprojekt:/ { fps=$2;gsub(/^[^0-9\.]*/,"", fps);
print $3"_"fps}' napi.log`
With your example napi.log, this will return :
napiprojekt:d40a1ef492e0dd094bc42141fd6e2dba_25
napiprojekt:38631781c8d9420eadfa13e9fe5d803b_25
napiprojekt:626e99378154981617418da4b39a098f_25
napiprojekt:9b9787e3e57754fb4301d10e865efdad_25
napiprojekt:f066e7a2da79ef924793212f52f0afae_25
napiprojekt:bbc652046e94f1802d6bea0e7d21643e_25
napiprojekt:9470761b635b733c2befab2df62f2672_23.976
napiprojekt:a385be50baeffaad680fb03c8c0e8ede_25
EDIT The problem comes from the fact that the napi will use the hash to make download and will use them as is to name the file. So you have to rename the files after the napi.sh donwload command, like with these command :
awk -F'|' '/napiprojekt:/ { fps=$2;gsub(/^[^0-9\.]*/,"", fps);gsub(/ /,"", fps);
filename=$3; gsub(/^ *napiprojekt:/,"",filename); gsub(/ /,"\\ ",filename);
print "mv "filename".srt "filename"_"fps".srt"}' napi.log | while read move_cmd; do
[ ! -z "${move_cmd}" ] && ${move_cmd};
done
Here is your script including that command to do that :
#!/bin/bash
title=$(echo "$1" | sed "s/.*\///" | cut -f 1 -d '.')
function _ask() {
while [[ $url == "" ]]; do
echo ; echo -e "Wklej link do filmu:" ; read -e url
done
}
napi.sh search -k movie "$title"
_ask
napi.sh subtitles "$url" > napi.log
echo Pobieram napisy:
napi.sh download -e srt `grep -o 'napiprojekt:.*' napi.log`
# Command to rename the files with the fps
awk -F'|' '/napiprojekt:/ { fps=$2;gsub(/^[^0-9\.]*/,"", fps); gsub(/ /,"", fps);
filename=$3; gsub(/^ *napiprojekt:/,"",filename); gsub(/ /,"\\ ",filename);
print "mv "filename".srt "filename"_"fps".srt"}' napi.log | while read move_cmd; do
[ ! -z "${move_cmd}" ] && ${move_cmd};
done
exit

Same KSH script run in test env but fails in production

I run the below script in a reference env and get the correct output in the output file, but when I run the same in production the output file is empty.
I tried debugging the using set -x, and understand that the for loop is not getting executed in the prod env.
please suggest what might be the issue.
#!/bin/ksh
DIR=/some/log/dir/error
DATE=$1
OUTPUT=/some/logs/dir/scripts/output/output.csv
. /calling/env/setupscript.ksh
for file in $(find $DIR 2>/dev/null| grep $ASOF | grep -i something1 | grep -vi someotherthing2 | grep -iv someotherthing3 | grep -vi someotherthing4 | grep -vi someotherthing4 )
do
echo "Checking file $file..."
if [[ -z "$DONE" ]] then
head -1 $file | read line
echo "File name;Filter name;Type;Scenario;$line" > $OUTPUT.tmp
DONE=1
fi
fullfile=$file
file=$(basename $file)
file=${file#SPR_RPT_}
file=${file#SPR_BY__}
echo $file | awk -F_ '{ print $(NF-2)" "$(NF-1) }' | read type scen
filter=${file%%_$type\_*}
grep ERROR $fullfile | while read line
do
echo "$(basename $fullfile);$filter;$type;$scen;$line" >> ${OUTPUT}.tmp
done
done

How to pass a variable string to a file txt at the biginig of test?

I have a problem
I Have a program general like this gene.sh
that for all file (es file: geneX.csv) make a directory with the name of gene (example: Genex/geneX.csv) next this program compile an other program inside gene.sh but this progrm need a varieble and I dont know how do it.
this is the program gene.sh
#!/bin/bash
# Create a dictory for each file *.xls and *.csv
for fname in *.xlsx *csv
do
dname=${fname%.*}
[[ -d $dname ]] || mkdir "$dname"
mv "$fname" "$dname"
done
# For each gene go inside the directory and compile the programs getChromosomicPositions.sh to have the positions, and getHapolotipeStings.sh to have the variants
for geni in */; do
cd $geni
z=$(tail -n 1 *.csv | tr ';' "\n" | wc -l)
cd ..
cp getChromosomicPositions.sh $geni --->
cp getHaplotypeStrings.sh $geni
cd $geni
export z
./getChromosomicPositions.sh *.csv
export z
./getHaplotypeStrings.sh *.csv
cd ..
done
This is the program getChromosomichPositions.sh:
rm chrPosRs.txt
grep '^Haplotype\ ID' $1 | cut -d ";" -f 4-61 | tr ";" "\n" | awk '{print "select chrom,chromStart,chromEnd,name from snp147 where name=\""$1"\";"}' > listOfQuery.txt
while read l; do
echo $l > query.txt
mysql -h genome-mysql.cse.ucsc.edu -u genome -A -D hg38 --skip-column-names < query.txt > queryResult.txt
if [[ "$(cat queryResult.txt)" == "" ]];
then
cat query.txt |
while read line; do
echo $line | awk '$6 ~/rs/ {print $6}' > temp.txt;
if [[ "$(cat temp.txt)" != "" ]];
then cat temp.txt | awk -F'name="' '{print $2}' | sed -e 's/";//g' > temp.txt;
./getHGSVposHG19.sh temp.txt ---> Hear the problem--->
else
echo $line | awk '{num=sub(/.*:g\./,"");num+=sub(/\".*/,"");if(num==2){print};num=""}' > temp2.txt
fi
done
cat query.txt >> varianti.txt
echo "Missing Data" >> chrPosRs.txt
else
cat queryResult.txt >> chrPosRs.txt
fi
done < listOfQuery.txt
rm query*
hear the problem:
I need to enter in the file temp.txt and put automatically at the beginning of the file the variable $geni of the program gene.sh
How can I do that?
Why not pass "$geni" as say the first argument when invoking your script, and treating the rest of the arguments as your expected .csv files.
./getChromosomicPositions.sh "$geni" *.csv
Alternatively, you can set it as environment variable for the script, so that it can be used there (or just export it).
geni="$geni" ./getChromosomicPositions.sh *.csv
In any case, once you have it available in the second script, you can do
if passed as the first argument:
echo "${1}:$(cat temp.txt | awk -F'name="' '{print $2}' | sed -e 's/";//g')
or if passed as environment variable:
echo "${geni}:$(cat temp.txt | awk -F'name="' '{print $2}' | sed -e 's/";//g')

BASH: Remove newline for multiple commands

I need some help . I want the result will be
UP:N%:N%
but the current result is
UP:N%
:N%
this is the code.
#!/bin/bash
UP=$(pgrep mysql | wc -l);
if [ "$UP" -ne 1 ];
then
echo -n "DOWN"
else
echo -n "UP:"
fi
df -hl | grep 'sda1' | awk ' {percent+=$5;} END{print percent"%"}'| column -t && echo -n ":"
top -bn2 | grep "Cpu(s)" | \sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | \awk 'END{print 100 - $1"%"}'
You can use command substitution in your first sentence (notice you're creating a subshell in this way):
echo -n $(df -hl | grep 'sda1' | awk ' {percent+=$5;} END{print percent"%"}'| column -t ):

Resources