Trying to write a bash script to copy a large number of files from an external drive into separate directories based on a subject id.
I've included the script I've written below.
I get the following error:
cat: /Volumes/Seagate: No such file or directory
cat: Backup: No such file or directory
cat: Plus: No such file or directory
cat: Drive/Subject_List.txt: No such file or directory
When I try to copy a single file at a time using the terminal, it copies using the exact command I've put in this script. I'm not sure why it's not recognizing the directory when I try and use it in the script below. Any help is greatly appreciated!
#!/bin/bash
#A bash script to copy the structural and functional files into the HCP_Entropy folder
#subject list
SUBJECT_LIST="/Volumes/Seagate/Backup/Plus/Drive/Subject_List.txt
for j in $(cat ${SUBJECT_LIST}); do
echo ${j}
cp /Volumes/Seagate\ Backup\ Plus\ Drive/HCP_DATA/Structural/{j}/unprocessed/3T/T1w_MPR1/${j}_3T_T1w_MPR1.nii.gz /Users/myname/Box/HCP_Entropy/BEN/${j}/anat
done
the line
$SUBJECT_LIST=/Volumes/Seagate\ Backup\ Plus\ Drive/Subject_List.txt
is bogus.
to assign values to a variable, you must not add the $ specifier.
a token starting with $ will be expanded, so $SUBJECT_LIST=... will first be expanded to =... (since you haven't assigned anything to the SUBJECT_LIST variable yet it is empty).
the proper way would be:
SUBJECT_LIST="/Volumes/Seagate Backup Plus Drive/Subject_List.txt"
(this uses quotes instead of escaping each space, which i find much more readable)
you also need to quote variables in case they contain spaces, else they might be interpreted by the command (cp) as multiple arguments.
for j in $(cat "${SUBJECT_LIST}"); do
# ...
done
and of course, you should check whether the source file actually exists, just like the destination directory.
indir="/Volumes/Seagate Backup Plus Drive"
SUBJECT_LIST="${indir}/Subject_List.txt"
cat "${SUBJECT_LIST}" | while read j; do
infile="${indir}/HCP_DATA/Structural/${j}/unprocessed/3T/T1w_MPR1/${j}_3T_T1w_MPR1.nii.gz"
outdir="/Users/myname/Box/HCP_Entropy/BEN/${j}/anat"
mkdir -p "${outdir}"
if [ -e "${infile}" ]; then
cp -v "${infile}" "${outdir}"
else
echo "missing file ${infile}" 1>&2
fi
done
Related
I'm attempting to write a program that moves zipped files that arrive in a directory, unzips them and then outputs the contents.
#!/bin/bash
shopt -s extglob
echo "Press [CTRL+C] to stop.."
#begin loop
while :
do
#search folder test_source for files and append names to array
queue+=($(ls /home/ec2-user/glutton/test_source | egrep 'test[0-9]{1}.gz'))
for i in $queue; do
#move file in test_source to temp folder
mv /home/ec2-user/glutton/test_source/${queue[i]} /home/ec2-user/glutton/temp
#unzip file
gunzip /home/ec2-user/glutton/temp/${queue[i]}
#add new file name to variable unzipped
unzipped=($(ls /home/ec2-user/glutton/temp | egrep 'test[0-9]{1}'))
cat temp/$unzipped
#Test for successful run
exit_status=$?
if [ $exit_status -eq 1 ]; then
#If fail move file to bank and output
mv /home/ec2-user/glutton/temp/$unzipped /home/ec2-user/glutton/logs/bank
echo "Error! $unzipped moved to /home/ec2-user/glutton/logs/bank"
#write to error.log
echo "moved ${queue[i]} to bank due to error" >> /home/ec2-user/glutton/logs/error.log
else
#If success delete file
rm /home/ec2-user/glutton/temp/$unzipped
fi
#wipe variable
unset unzipped
i=$i+1
done
#wipe array
unset queue
i=0
#go to top of loop
done
This has worked pretty well up until I added the unzipping feature and now my program outputs this error when attempting to move the .gz file:
./glutton.sh: line 11: test0.gz: syntax error: invalid arithmetic operator (error token is ".gz")
When I run the first part of my script in the command line it seems to work perfectly, but doesn't when I run it on its own, I'm pretty confused.
Your main issue is that when you iterate an array like you are doing, you get the first item of the array, not the index. So in your case, $i is not a number, it is the filename (i.e. test1.gz) and it will only see the first file. The cleanest way I have seen to iterate the items in an array would be for i in "${arrayName[#]}".
Also, using '{1}' in your regex is redundant, the character class will already match only 1 character if you don't specify a modifier.
It shouldn't matter depending on the contents of your 'temp' folder, but I would be more specific on your egreps too, if you add -x then it will have to match the whole string. As it is currently, a file called 'not_a_test1' would match.
I am new to programming and just starting in bash.
I'm trying to print a list of directories and files to a txt file, and remove some of the path that gets printed to make it cleaner.
It works with this:
TODAY=$(date +"%Y-%m-%d")
cd
cd Downloads
ls -R ~/Music/iTunes/iTunes\ Media/Music | sed 's/\/Users\/BilPaLo\/Music\/iTunes\/iTunes\ Media\/Music\///g' > music-list-$TODAY.txt
But to clean it up I want to use variables like so,
# Creates a string of the date, format YYYY-MM-DD
TODAY="$(date +"%Y-%m-%d")"
# Where my music folders are
MUSIC="$HOME/Music/iTunes/iTunes\ Media/Music/"
# Where I want it to go
DESTINATION="$HOME/Downloads/music-list-"$TODAY".txt"
# Path name to be removed from text file
REMOVED="\/Users\/BilPaLo\/Music\/iTunes\/iTunes\ Media\/Music\/"
ls -R "$MUSIC" > "$DESTINATION"
sed "s/$REMOVED//g" > "$DESTINATION"
but it gives me a 'no such file or directory' error that I can't seem to get around.
I'm sure there are many other problems with this code but this one I don't understand.
Thank you everyone! I followed the much needed formatting advice and #amo-ej1's answer and now this works:
# Creates a string of the date format YYYY-MM-DD
today="$(date +"%Y-%m-%d")"
# Where my music folders are
music="$HOME/Music/iTunes/iTunes Media/Music/"
# Where I want it to go
destination="$HOME/Downloads/music-list-$today.txt"
# Temporary file
temp="$HOME/Downloads/temp.txt"
# Path name to be removed of text file to only leave artist name and album
remove="\\/Users\\/BilPaLo\\/Music\\/iTunes\\/iTunes\\ Media\\/Music\\/"
# lists all children of music and writes it in temp
ls -R "$music" > "$temp"
# substitutes remove by nothing and writes it in destination
sed "s/$remove//g" "$temp" > "$destination"
rm $temp #deletes temp
First when debugging bash it can be helpful to start bash with the -x flags (bash -x script.sh) or within the script enter set -x, that way bash will print out the commands it is executing (with the variable expansions) and you can more easily spot errors that way.
In this specific snippet our ls output is being redirected to a file called $DESTINATION and and sed will read from standard input and write also to $DESTINATION. So however you wanted to replace the pipe in your oneliner is wrong. As a result this will look as if your program is blocked but sed will simply wait for input arriving on standard input.
As for the 'no such file or directory', try executing with set -x and doublecheck the paths it is trying to access.
So I have a conversion program (works through command line) that takes a data file and converts it into a new format while also putting it in a folder with various subfolders. I wanted to create a script that would check for duplicates before executing this conversion program.
So far, I have
#!/bin/bash
for subj in `ls <directory to data files>`
do
subj_name=$subj
subj_path=<directory to data files>/$subj_name #I need this for my program, can ignore
cd <directory with output folders>
if [ -e “$subj” ]; then
echo “This file already exists” #This will restart the loop and move to the next file
else
echo “This folder does not exist”
My_Program #I can handle this one
fi
done
The program works fine with files of the same format (ie .txt and .txt) but cannot check for a folder and .txt for the same name. Are there any changes I can make to check for the same name regardless of file format?
Edit: I did a little experimenting, and I put a duplicate data file into the directory with the output folders and it still didn't recognize it. I think the cd line or the if line is wrong then.. anyone have any tips on how I could fix this?
Use the syntax bellow to remove ".txt" from the end of value of $subj, returning the resulting string . (more info on "Bash String Manipulation")
${subj%.txt}
Then check the existence of files/directories with or without .txt:
if [ -e "$subj" ] || [ -e "${subj%.txt}" ]; then
....
If you want to remove any suffix (.txt, .tgz, ...) use ${subj%.*} to delete all characters after (and including) the last '.' Example:
[bash]$ subj=file.txt
[bash]$ echo ${subj%.*}
[bash]$ file
Or use ${subj%%.*} to delete all characters after (and including) the first '.':
[bash]$ subj=file.txt.tgz
[bash]$ echo ${subj%%.*}
[bash]$ file
I have made a directory with lots of files with:
samplefile_111222015_reporting_{1..13}
I am trying to create a vi script where when I enter the directory as an argument to the command e.g.
sh myScript $HOME/theDir/*
then it copies all the files in that directory to a new one I made. Although right now, I'm having problems with the for loop alone.
This is what I have in my script:
for f in $1;
do
echo "$f"
done
but when i enter sh myScript $HOME/theDir, I get back the name of the first file only (samplefile_111222015_reporting_1). why the first file? Is this not a for loop>
# Because of the wild card expansion, all the files in the directory are
# already made available to the script through arguments
# So do the following to get all the file listing
for f ; do echo $f; done
This is because each file is passed as a separate argument and you're only looping over $1, which is the first argument.
Instead, you most likely want to loop over "$#", which is every argument starting from $1.
The man page for bash, under the Special Parameters section, details the special parameters available in more detail.
I'm trying to write a script that does something a bit more sophisticated than what I'm going to show you, but I know that the problem is in this part.
I want each name of a list of files in a directory to be assigned to a variable (the same variable, one at a time) through a for loop, then do something inside of the loop with this, see what mean:
for thing in $(ls $1);
do
file $thing;
done
Edit: let's say this scrypt is called Scrypt and I have a folder named Folder, and it has 3 files inside named A,B,C. I want it to show me on the terminal when I write this:
./scrypt Folder
the following:
A: file
B: file
C: file
With the code I've shown above, I get this:
A: ERROR: cannot open `A' (No such file or directory)
B: ERROR: cannot open `B' (No such file or directory)
C: ERROR: cannot open `C' (No such file or directory)
that is the problem
One way is to use wildcard expansion instead of ls, e.g.,
for filename in "$1"/*; do
command "$filename"
done
This assumes that $1 is the path to a directory with files in it.
If you want to only operate on plain files, add a check right after do along the lines of:
[ ! -f "$filename" ] && continue
http://mywiki.wooledge.org/ParsingLs
Use globbing instead:
for filename in "$1"/* ; do
<cmd> "$filename"
done
Note the quotes around $filename
It's a bit unclear what you are trying to accomplish, but you can essentially do the same thing with functionality that already exists with find. For example, the following prints the contents of each file found in a folder:
find FolderName -type f -maxdepth 1 -exec cat {} \;
well,
i think that what you meant is that the loop will show the filenames in the desired dir.
so, i would do it like that:
for filename in "$1"/*; do
echo "file: $filename"
done
that way the result should be (in case in the dir are 3 files and the names are A B C:
`file: A
`file: B
`file: C