I am trying to loop through all of the files in a directory and
move them to a workspace (I need to do this to do this because the workspace doesn't have much storage).
Run a program which produces an output directory that contains all the files I will want to work with in the future
Delete the original file from the workspace (to save space in the workspace)and
Move the output directory out of the workspace and back to the storage space
I am able to do this for each file singly (i.e. each line works if I actually use the name of the files), but I can't get the for loop to work. I am quite new to this, so I probably did something simple wrong.
Can anyone see where I am going wrong?
for i in path_to_files; do
#copy to home directory (from scratch)
cp $i .
#Run IDBA
idba_ud -l $i -o '$i'_out
#remove file from work directory (limited space)
rm $i
#copy out directory back to scratch
cp -r '$i'_out path_to_files
done
I keep getting an error that says
syntax error near unexpected token `cp'.
I have also tried replacing cp with copy and i/$i with file/$file with no luck.
If this is indeed POSIX compatible shell (your code looks suspiciously like that, but you haven't specified the actually used shell), then:
You should always quote filenames, in case it contains spaces or other weird characters:
But you should not use single-quotes, as this will prevent shell from expanding your variables.
when appending text to substituted variables, use ${} notation (e.g. if $i expands to "murgel", then ${i}foo will expand to "murgelfoo", whereas $ifoo will expand to "" (an empty string) if there is no variable ifoo)
Thus try:
filepath=/path/to/files
for i in "${filepath}"/*; do
#copy to home directory (from scratch)
cp "${i}" .
#Run IDBA
idba_ud -l "${i}" -o "${i}_out"
#remove file from work directory (limited space)
rm $i
#copy out directory back to scratch
cp "${i}_out"/* "${filepath}"/
done
Related
So I want to copy some files from one directory to another.
Essentially, I want to capture the directory path to a variable, say, "pathname" and use
"cp-r $pathname ." to copy file1 and file2 to to a new folder in which I made using mkdir and have cd'ed into (hence the "." as the second command line argument).
source_to_copy_from:
home/folder1/folder2/file1
home/folder1/folder2/file2
destination_to_copy_to:
home/newfolder1/newfolder2/file1
home/newfolder1/newfolder2/file2
I did:
pathname=$"$(pwd)"
//code to make the newfolder1 here
cp -r $"$pathname" .
But there appears to be nothing in the new pathname that was supposed to be copied in.
Also am using Mac bash
also quite beginner to bash
I want to move a couple of files from point a to point b
but I have to manually specify
mv /full/path/from/a /full/path/to/b
but some times there are 20 files which I have to move manually. Instead of /full/path/form/a, can't I just enter the a function which returns all the files which I want to move in my case;
/full/path/to/b is a directory, it's the target directory which all the files with extenstions mp3, exe and mp4 must go to:
mv ls *.{mp3,exe,mp4} /full/path/to/b
If I have to move a couple of files and I don't want to do it one by one, how can I optimize the problem?
The command mv ls *.{mp3,exe,mp4} /full/path/to/b in your question is not correct.
As pointed out in comments by #janos, the correct command is
mv *.{mp3,exe,mp4} /full/path/to/b
mv can complain about missing file if the file is really missing and/or the path is not accessible or is not valid.
As i can understand by your question description, if you go manually to the source path you can move the file to the desired directory.
Thus it seems that path is valid, and file exists.
In order mv to keeps complaining about *.mp3 not found (having a valid path and file) the only reason that pops up in my head is the Bash Pathname Expansion feature (enabled by default in my Debian).
Maybe for some reason this pathname expansion bash feature is disabled in your machine.
Try to enable this feature using command bellow and provide the correct command to mv and you should be fine.
$ set +f
PS: Check man bash about pathname expansion.
This is not a duplicate as I want to achieve the result without using Windows tools as mentioned in this post https://superuser.com/questions/78434/tool-for-deleting-directories-with-path-names-too-long-for-normal-delete
I am deleting SVN ignored files that Maven has created (something like this)
jaxb/com.mycompany.test.jaxb.upload-document/target
jaxb/com.mycompany.test.jaxb.fpml/target
jaxb/com.mycompany.test.jaxb.fixml/target
jaxb/com.mycompany.test.jaxb.trade-header/target
jaxb/com.mycompany.test.jaxb.schema-versioning/target
test-risk/target
I am using the following command but it fails when the path is too long on Windows.
echo "Deleting ignored files"
svn stat --no-ignore | awk '$1=="I" { print $0 }'|awk '{$1=""; print $0}'|tr '\\' '/'|while read file;do rm -rf "$file" ;done;
This is run as part of a script that runs on Unix as well as Windows which is why I can't use any of the previously mentioned tools.
What I need to do is write a routine that checks to see if there are any errors due to long file length and recursively rename the directories from the top down to the bottom with single characters in an attempt to shorten the path.
I'm not sure how to check for the errors ("Directory not empty" and " File or path name too long").
Ideally I'd like to fail the "rm -rf" immediately rather than throwing the error hundreds of times.
Finally, I'm not sure what the most efficient way to recursively rename the directories is.
EDIT: If I just assume that all directories are going to cause the issue I could get the roots cd to them in turn and recursively rename the sub-dirs. This way I don't have to worry about catching errors from rm.
I have cobbled together a shell script to submit multiple jobs on a cluster, which it appears to without giving me an error message, but the output files are missing and the error log files are also empty. What the script supposed to do is 1.) make a bunch of new directories, 2.) copy four files to each (mainparams, extraparams, infile, and structurejobsubmissionfile) 3.) then submit each one to the cluster for it to run structure while changing one parameter in the mainparams file every tenth directory (that's the 's/changethis/'$k'/g' line).
Test running it on the front end gives no errors, the structure program is up to date on the cluster, and the cluster administrators don't see anything wrong. Thanks!
#!/bin/bash
reps=10
numK=10
for k in $(seq $numK);
do
for i in $(seq $reps);
do
#make folder name (ex. k4rep7)
tmpstr="k${k}rep${i}"
#echo "Making folder and filename $tmpstr"
#make the new folder
mkdir $tmpstr
#go to that folder
cd ./$tmpstr
#copy in the input files
cp ../str_in/* ./
#modify the recently copied input file here so source file remains the same
cp ./mainparams ./temp.txt
#change maxpops to current value of k and the directory for the files to the current directory
sed -e 's/changethis/'$k'/g' -e "s:pathforrunningstructurehere:$PWD:g" ./temp.txt > ./mainparams
#get rid of temporary file
rm ./temp.txt
#inside $i so run STRUCTURE here
qsub -q fnrgenetics -l nodes=1:ppn=1,walltime=20:00:00 structurejobsubmissionfile
#go back to parent directory
cd ../
done
done
I can't see anything obviously wrong, but I think the place that you'll find the answer lies in better logging and better error checking. Some of the things that you're not checking that you should:
Is $tmpstr created correctly? (will fail on disk full or if permissions are not set correctly)
does str_in/ exist, and is it a directory?
does it contain files?
does it contain mainparams?
is qsub in $PATH?
does the call to qsub return an error?
You can roll an error logging function of your own, or use a package like log4bash
I have a lot of files named the same, with a directory structure (simplified) like this:
../foo1/bar1/dir/file_1.ps
../foo1/bar2/dir/file_1.ps
../foo2/bar1/dir/file_1.ps
.... and many more
As it is extremely inefficient to view all of those ps files by going to the
respective directory, I'd like to copy all of them into another directory, but include
the name of the first two directories (which are those relevant to my purpose) in the
file name.
I have previously tried like this, but I cannot get which file is from where, as they
are all named consecutively:
#!/bin/bash -xv
cp -v --backup=numbered {} */*/dir/file* ../plots/;
Where ../plots is the folder where I copy them. However, they are now of the form file.ps.~x~ (x is a number) so I get rid of the ".ps.~*~" and leave only the ps extension with:
rename 's/\.ps.~*~//g' *;
rename 's/\~/.ps/g' *;
Then, as the ps files have hundreds of points sometimes and take a long time to open, I just transform them into jpg.
for file in * ; do convert -density 150 -quality 70 "$file" "${file/.ps/}".jpg; done;
This is not really a working bash script as I have to change the directory manually.
I guess the best way to do it is to copy the files form the beginning with the names
of the first two directories incorporated in the copied filename.
How can I do this last thing?
If you just have two levels of directories, you can use
for file in */*/*.ps
do
ln "$file" "${file//\//_}"
done
This goes over each ps file, and hard links them to the current directory with the /s replaced by _. Use cp instead of ln if you intend to edit the files but don't want to update the originals.
For arbitrary directory levels, you can use the bash specific
shopt -s globstar
for file in **/*.ps
do
ln "$file" "${file//\//_}"
done
But are you sure you need to copy them all to one directory? You might be able to open them all with yourreader */*/*.ps, which depending on your reader may let browse through them one by one while still seeing the full path.
You should run a find command and print the names first like
find . -name "file_1.ps" -print
Then iterate over each of them and do a string replacement of / to '-' or any other character like
${filename/\//-}
The general syntax is ${string/substring/replacement}. Then you can copy it to the required directory. The complete script can be written as follows. Haven't tested it (not on linux at the moment), so you might need to tweak the code if you get any syntax error ;)
for filename in `find . -name "file_1.ps" -print`
do
newFileName=${filename/\//-}
cp $filename YourNewDirectory/$newFileName
done
You will need to place the script in the same root directory or change the find command to look for the particular directory if you are placing the above script in some other directory.
References
string manipulation in bash
find man page