I am new to shell scripting and I am trying to automatise a process.
I think the structure is ok, but i get notice which i think that it should not be there
# STEP 1#
echo " Untar the file s"
time tar -xf $tarfiles
time tar -xf *.tar
#STEP 2#
fname=(wnd10m.gdas.2010*.grb2)
echo " convert into .nc "
cdo -f nc copy $fname $fname.nc
#STEP 3#
ofile1=$fname.nc
echo "re-format into structured grid 0.5x0.5"
cdo remapbil,r720x361 $ofile1 remap$R.nc
#STEP 4#
ofile2=$remapR.nc
echo "produce desired mesh"
cdo sellonlatbox,lon1,lon2,lat1,lat2 $ofile2 $grid.nc
#STEP 5#
ofile3=$grid.nc
echo "merge the files based on tstep"
cdo mergetime $ofile3 final$R.nc
STEP 1 and STEP 2 are performed although it only manipulates and alters one of the file that it was included in the .tar file.
The process so far gives out the STEP 1 & 2, but during STEP 3 I do not get the new file as remap$R.nc with R being an additional ending so generated is not confused with other produced in the previous step.
After I run it this is what i get
convert into .nc
cdo copy: Processed 987365376 values from 2 variables over 744 timesteps ( 234.66s )
re-format into structured grid 0.5x0.5
cdo remapbil: Processed 987365376 values from 2 variables over 744 timesteps ( 53.59s )
produce desired mesh
Error (cdo sellonlatbox) : Output file name .nc is equal to input file name on position 1!
merge the files based on tstep
cdo mergetime: Open failed on >.nc<
No such file or directory
I am trying to de-bug my file but don't know what I have done wrong, because the process is supposed to handle a lot of files converting them into every stage I paid significant attention to assigning correct and different fnames.
I would be grateful if you could have a look and give me your inputs.
p.s. cdo is a code I am using so you can disregard it, my focus is on the process itself
thanks
$remapR is different to remap$R. Move your dollar sign.
You can use set -xv to debug your script.
compare these:
cdo remapbil,r720x361 $ofile1 remap$R.nc
and
ofile2=$remapR.nc
this looks like a classical typo
Hello all as a continuation to the issue i had and the user #Bushmills help me a lot (thanks!!!)
I attach the code that I did with advice i got, i tested and it works, so hopefully another user (novice like me) may find this useful.
file="wnd10m.gdas."
year='2010'
month='01 02'
ext="grb2"
for((y=$year;y<=$year;y++));
do
for m in $month
do
fname=$file$y$m.$ext
#STEP 1#
cdo -f nc copy $fname $fname.nc
#STEP 2#
ofile1=$fname.nc
cdo remapbil,r720x361 $ofile1 R$ofile1 #produces a file with the name of the R+ofile1
example if the previous file was named FILE the command will produce a new file named RFILE
#STEP 3#
ofile2=($R*.nc)
use all the file starting with R
cdo sellonlatbox,$lon1,$lon2,$lat1,$lat2 $ofile2 grid_$ofile2
done
done
thank you for the help
Related
I have lot of files in source directory with different name and ending with some number as suffix name. I need to copy all the files into 3 different directory. While copying, each file should be unique to other directory.
Example :
Source directory
1) Test01.csv
2) Test02.csv
3) Test03.csv
4) Nontesting01.csv
5) Nontesting02.csv
6) Nontesting03.csv
Destination directory
Directory 1 : Test01.csv
Nontesting01.csv
Directory 2 : Test02.csv
Nontesting02.csv
Directory 3 : Test03.csv
Nontesting03.csv
I have tried below code but it's copying 1 file per directory.
#!/bin/bash
dest=/Users/myprofile/Testing_
count=0
for f in *.csv ; do (cp $f/*.csv* "${dest}"${count}/$f ) ;
((count++))
done
Could someone help how to achieve this scenario using shell scripting.
Piggy-backing on what Jetchisel said, the code should be adapted, as a loop on patterns, to look like this:
#!/bin/bash
destPref="/Users/myprofile/Testing_"
for pattern in 01 02 03
do
files=(*${pattern}.csv)
cp -v -- "${files[#]}" "${destPref}${pattern}/"
done
As a general guidance, if you start by writing the code logic
in pseudo-code,
fine-grained for each task that you are trying to accomplish,
in the correct sequence and
in the correct context,
then having that worded so that it does exactly what you want it to do ... will, almost explicitly, tell you WHAT you need to code for each of those, not the HOW. The HOW is the nitty gritty of coding.
If you give that a try, the solution will almost pop out of the page at you.
Good luck with your apprenticeship!
I am new to unix and could really use your help.
I want to rename a lot of photographs so they correspond with codes of items that are on the picture. I have a .csv file that has the original .jpg name and then the codes I want the photos to be renamed to, following in consecutive columns. For example:
IMG_1234.JPG,AB001,AB003,AB004
IMG_1345.JPG,AB011,AB012,AB013,AB014,AB015
IMG_1456.JPG,AB112
IMG_1678.JPG,AB125,AB126
So I want IMG_1234.JPG copied 3 times and renamed to AB001, AB003, and AB004 etc.
I know I need a script and that I can copy and rename files, but I can't figure out how to make a script run through the csv file and copy & rename the .jpg to the names following until an empty cell and then move on to the next row and copy & rename that .jpg etc etc.
I hope my question is clear and I apologize for my limited knowledge.
Thanks in advance!
edit: The image names have directories (with spaces) in front of them as the photographs are in different folders. For example:
./Photos sorted/Samples1-100/IMG_1134.JPG
This should do what you want. The filename of the csv file is given as parameter to the script. You might adjust the paths inside the copy command, currently everything must be in the same directory. If you are using this on a mac or linux, you can also use "ln -s" instead if "cp" to create a symbolic link to the original file to save disk space.
CSVFILE=$1
cat $CSVFILE |\
while read LINE; do
SPLIT=`echo $LINE | tr "," " "`
FIRST=0
for NAME in $SPLIT; do
if [ $FIRST -eq 0 ]; then
SRCNAME=$NAME
else
DSTNAME=$NAME
cp ${SRCNAME}.jpg ${DSTNAME}.jpg
fi
((FIRST++))
done
done
I am trying to create a loop in Bash script for a series of data migrations:
At the beginning of every step, the script should get the name of the newest file in a folder
called "migrationfiles/ and store it in the variable "migbefore" and create a new variable called "migbefore+1":
Example: if the "migrationfiles/" folder contains the following files:
migration.pickle1 migration.pickle2 migration.pickle3
The variable "migbefore" and migafter should have the following value:
migbefore=migration.pickle3
migafter=migration.pickle4
At the end of every step, the function "metl", which is in charge of making the data migration, uses the file "migbefore" to load the data and creates 1 new file called "migafter" and stores it in the "migrationfiles/" folder, so in this case, the new file created will be called:
"migration.pickle4"
The code I pretend using is the following:
#!/bin/bash
migbefore=0
migafter=0
for y in testappend/*
for x in migrationfiles/*
do
migbefore=migration.pickle(oldest)
migafter=migbefore+1
done
do
metl -m migrationfiles/"${migbefore}"
-t migrationfiles/"${migafter}"
-s "${y}"
config3.yml
done
Does anyone know how I could make the first loop (The one that searches for the newest file in the "migrationfiles/" folder) and then assigns the name of the variable "migafter" as "migbefore+1"?
I think this might do what you want.
#!/bin/bash
count=0
prefix=migration.pickle
migbefore=$prefix$((count++))
migafter=$prefix$((count++))
for y in testappend/*; do
echo metl -m migrationfiles/"${migbefore}" \
-t migrationfiles/"${migafter}" \
-s "${y}" \
config3.yml
migbefore=$migafter
migafter=$prefix$((count++))
done
Copy with Numbered Backups
It's really hard to tell what you're really trying to do here, and why. However, you might be able to make life simpler by using the --backup flag from the cp command. For example:
cp --backup=numbered testappend/migration.pickle migrationfiles/
This will ensure that you have a sequence of migration files like:
migration.pickle
migration.pickle.~1~
migration.pickle.~2~
migration.pickle.~3~
where the older versions have larger ordinal numbers, while the latest version has no ordinal extension. It's a pretty simple system, but works well for a wide variety of use cases. YMMV.
# configuration:
path=migrationfiles
prefix=migration.pickle
# determine number of last file:
last_number=$( find ${path} -name "${prefix}*" | sed -e "s/.*${prefix}//g" | sort -n | tail -1 )
# put together the file names:
migbefore=${prefix}${last_number}
migafter=${prefix}$(( last_number + 1 ))
# test it:
echo $migbefore $migafter
This should work even if there are no migration files yet. In that case, the value of migbefore is just the prefix and does not point to a real file.
First post, so Hi! Let me start by saying I'm a total noob regarding programming. I understand very basic stuff, but when it comes to checking exit codes or what the adequate term is, I'm at a loss. Apparently my searchfoo is really weak in this area, I guess it's a question of terminology.
Thanks in advance for taking your time to reading this/answering my question!
Description: I found a script that converts/repack .cbr files to .cbz files. These files are basically your average rar and zip files, however renamed to another extension as they are used for (comic)book applications such as comicrack, qcomicbook and what not. Surprisingly enough there no cbr -> cbz converters out there. The advantages of .cbz is besides escaping the proprietary rar file format, that one can store the metadata from Comic Vine with e. g comictagger.
Issue: Sometimes the repackaging of the files doesn't end well and would hopefully be alleviated by a integrity check & another go. I modified said script slightly to use p7zip as it can both pack/unpack 7z, zip-files and some others, i. e great for options. p7zip can test the archive by:
7z t comicfile.cbz tmpworkingdir
I guess it's a matter of using if & else here(?) to check the integrity and then give it another go, if there are any error.
Question/tl;dr: What would be the "best"/adequate approach to add a integrity file check to the script below?
#!/bin/bash
#Source: http://comicrack.cyolito.com/forum/13-scripts/30013-cbr3cbz-rar-to-zip-conversion-for-linux
echo "Converting CBRs to CBZs"
# Set the "field separator" to something other than spaces/newlines" so that spaces
# in the file names don't mess things up. I'm using the pipe symbol ("|") as it is very
# unlikely to appear in a file name.
IFS="|"
# Set working directory where to create the temp dir. The user you are using must have permission
# to write into this directory.
# For performance reasons I'm using ram disk (/dev/shm/) in Ubuntu server.
WORKDIR="/dev/shm/"
# Set name for the temp dir. This directory will be created under WORDDIR
TEMPDIR="cbr2cbz"
# The script should be invoked as "cbr2cbz {directory}", where "{directory}" is the
# top-level directory to be searched. Just to be paranoid, if no directory is specified,
# then default to the current working directory ("."). Let's put the name of the
# directory into a shell variable called SOURCEDIR.
# Note: "$1" = "The first command line argument"
if test -z "$1"; then
SOURCEDIR=`pwd`
else
SOURCEDIR="$1"
fi
echo "Working from directory $SOURCEDIR"
# We need an empty directory to work in, so we'll create a temp directory here
cd "$WORKDIR"
mkdir "$TEMPDIR"
# and step into it
cd "$TEMPDIR"
# Now, execute a loop, based on a "find" command in the specified directory. The
# "-printf "$p|" will cause the file names to be separated by the pipe symbol, rather than
# the default newline. Note the backtics ("`") (the key above the tab key on US
# keyboards).
for CBRFILE in `find "$SOURCEDIR" -name "*.cbr" -printf "%p|while read line; do
# Now for the actual work. First, extract the base file name (without the extension)
# using the "basename" command. Warning: more backtics.
BASENAME=`basename $CBRFILE ".cbr"`
# And the directory path for that file, so we know where to put the finished ".cbz"
# file.
DIRNAME=`dirname $CBRFILE`
# Now, build the "new" file name,
NEWNAME="$BASENAME.cbz"
# We use RAR file's name to create folder for unpacked files
echo "Processing $CBRFILE"
mkdir "$BASENAME"
# and unpack the rar file into it
7z x "$CBRFILE" -O"$BASENAME"
cd "$BASENAME"
# Lets ensure the permissions allow us to pack everything
sudo chmod 777 -R ./*
# Put all the extracted files into new ".cbz" file
7z a -tzip -mx=9 "$NEWNAME" *
# And move it to the directory where we found the original ".cbr" file
mv "$NEWNAME" $DIRNAME/"$NEWNAME"
# Finally, "cd" back to the original working directory, and delete the temp directory
# created earlier.
cd ..
rm -r "$BASENAME"
# Delete the RAR file also
rm "$CBRFILE"
done
# At the end we cleanup by removing the temp folder from ram disk
cd ..
echo "Conversion Done"
rm -r "$TEMPDIR"
Oh the humanity, not posting more than two links before 10 reputation and I linked the crap out of OP.. [edit]ah.. mh-mmm.. there we go..
[edit 2] I removed unrar as an dependency and use p7zip instead, as it can extract rar-files.
You will need two checks:
7z t will test the integrity of the archive
You should also test the integrity of all the image files in the archive. You can use at tools like ImageMagick for this.
A simple test would be identify file but that might read only the header. I'd use convert file -resize 5x5 png:- > /dev/null
This scales the image down to 5x5 pixels, converts it to PNG and then pipes the result to /dev/null (discarding it). For the scaling, the whole image has to be read. If this command fails with an error, something is wrong with the image file.
Here's what I want to do:
I have thousand of split rar archives on folder name Archives.
Name of the files 0001.part1.rar 0002.part2.rar 0003.part3.rar etc.
read 0001.part1.rar
create a directory based on prefix of the file above e.g. 0001
move all files with the same prefix on the directory created above.
Extract the files within that directory
delete all rar files in that directory
Rename the extracted file based on a list of names from a text file.
Rar the renamed file with different arguments.
Move the renamed file (from step 6) to a new directory called Done.
Proceed to file 0002.part1.rar then do steps 2-8 and so forth.
Additional how would I incorporate it with cron???
This should be run once only...
After extracting the first set of rar's files change to:
file.001
file.002
file.003
etc. which I need to extract too.
Clarification on Step 6:
After extracting the second set of rar's (file.001 , file.002 etc.)
I want to rename it based on a list of names from a text file.
e.g. List of files from a text file:
0001 - GB Funds.DAT
0002 - US Bonds.DAT
0003 - SG Securities.DAT
0004 - LU Credits.DAT
Clarification on Step 7:
After renaming the file I want to move it on a new folder called "Done"
Clarification on Step 9:
Go back to the main folder with all the other archives
and continue extracting the next set of archives and
do the same steps from 1 to 8.
You can write a shell script including something like this:
# foo.sh
set -e
set -u
for i in `find -max-depth 1 -type f -name '*.rar' | sed 's/\.part*\.rar$//' | sort -u`; do
mkdir $i
mv $i.part*rar $i
cd $i
unrar x $i.part1.rar
DST=`grep $i ../rename.txt | sed 's/^[0-9]\+ - //'`
mv $i "$DST"
# and so on, rar it together again, move it done directory etc.
cd ..
done
Run it then via:
bash foo.sh
You have to clarify 6./8./9.
I don't know why do you want to run it via cron, since you only want to run it once. at is designed for running one-time jobs, or run it in a screen session.
I suggest that you do a few tests with 1-3 files from your collection and the script you end up with, before starting the whole job.