I've written a script to go through all the files in the directory the script is located in, identify if a file name contains a certain string and then modify the filename. When I run this script, the files that are supposed to be modified are disappearing. It appears my usage of the mv command is incorrect and the files are likely going to an unknown directory.
#!/bin/bash
string_contains="dummy_axial_y_position"
string_dontwant="dummy_axial_y_position_time"
file_extension=".csv"
for FILE in *
do
if [[ "$FILE" == *"$string_contains"* ]];then
if [[ "$FILE" != *"$string_dontwant"* ]];then
filename= echo $FILE | head -c 15
combined_name="$filename$file_extension"
echo $combined_name
mv $FILE $combined_name
echo $FILE
fi
fi
done
I've done my best to go through the possible errors I've made in the MV command but I haven't had any success so far.
There are a couple of problems and several places where your script can be improved.
filename= echo $FILE | head -c 15
This pipeline runs echo $FILE adding the variable filename having the null string as value in its environment. This value of the variable is visible only to the echo command, the variable is not set in the current shell. echo does not care about it anyway.
You probably want to capture the output of echo $FILE | head -c 15 into the variable filename but this is not the way to do it.
You need to use command substitution for this purpose:
filename=$(echo $FILE | head -c 15)
head -c outputs only the first 15 characters of the input file (they can be on multiple lines but this does not happen here). head is not the most appropriate way for this. Use cut -c-15 instead.
But for what you need (extract the first 15 characters of the value stored in the variable $FILE), there is a much simpler way; use a form of parameter expansion called "substring expansion":
filename=${FILE:0:15}
mv $FILE $combined_name
Before running mv, the variables $FILE and $combined_name are expanded (it is called "parameter expansion"). This means that the variable are replaced by their values.
For example, if the value of FILE is abc def and the value of combined_name is mnp opq, the line above becomes:
mv abc def mnp opq
The mv command receives 4 arguments and it attempts to move the files denoted by the first three arguments into the directory denoted by the fourth argument (and it probably fails).
In order to keep the values of the variables as single words (if they contain spaces), always enclose them in double quotes. The correct command is:
mv "$FILE" "$combined_name"
This way, in the example above, the command becomes:
mv "abc def" "mnp opq"
... and mv is invoked with two arguments: abc def and mnp opq.
combined_name="$filename$file_extension"
There isn't any problem in this line. The quotes are simply not needed.
The variables filename and file_extension are expanded (replaced by their values) but on assignments word splitting is not applied. The value resulted after the replacement is the value assigned to variable combined_name, even if it contains spaces or other word separator characters (spaces, tabs, newlines).
The quotes are also not needed here because the values do not contain spaces or other characters that are special in the command line. They must be quoted if they contain such characters.
string_contains="dummy_axial_y_position"
string_dontwant="dummy_axial_y_position_time"
file_extension=".csv"
It is not not incorrect to quote the values though.
for FILE in *
do
if [[ "$FILE" == *"$string_contains"* ]];then
if [[ "$FILE" != *"$string_dontwant"* ]]; then
This is also not wrong but it is inefficient.
You can use the expression from the if condition directly in the for statement (and get rid of the if statement):
for FILE in *"$string_contains"*; do
if [[ "$FILE" != *"$string_dontwant"* ]]; then
...
If you have read and understood the above (and some of the linked documentation) you will be able to figure out yourself where were your files moved :-)
Related
I am trying to write a shell script so that I can move school files from one destination to another based on the input. I download these files from a source like canvas and want to move them from my downloads based on the tag I assign, to the path for my course folder which is nested pretty deep thanks to how I stay organized. Unfortunately, since I store these files in my OneDrive school account, I am unable to eliminate some spacing issues but I believe I have accounted for these. Right now the script is the following:
if [ "$1" = "311" ];
then
course="'/path/to/311/folder/$2'"
elif [ "$1" = "411" ];
then
course="'/path/to/411/folder/$2'"
elif [ "$1" = "516" ];
then
course="'/path/to/516/folder/$2'"
elif [ "$1" = "530" ];
then
course="'/path/to/530/folder/$2'"
elif [ "$1" = "599" ];
then
course="'/path/to/599/folder/$2'"
fi
files=$(mdfind 'kMDItemUserTags='$1'' -onlyin /Users/user/Downloads)
#declare -a files=$(mdfind 'kMDItemUserTags='$1'' -onlyin /Users/user/Downloads)
#mv $files $course
#echo "mv $files $course"
#echo $course
for file in $files
#for file in "${files[#]}"
do
#echo $file
#echo $course
mv $file $course
done
Where $1 is the tag ID and first part of path selection, and $2 is what week number folder I want to move it to. The single quotation marks are there to take care of the spacing in the filepath. I could very easily do this in python but I'm trying to expand my capabilities some. Every time I run this script I get the following message:
usage: mv [-f | -i | -n] [-v] source target
mv [-f | -i | -n] [-v] source ... directory
I initially tried to just move them all at once (per the first mv command that's commented out) and got this error, then tried the for loop, and array but get the same error each time. However, when I uncomment the echo statements in the for loop and manually try to move each one by copying and pasting the paths to the command line, it works perfectly. My best guess is something to do with the formatting of the variable "files", since
echo "mv $files $course"
indicates the presence of a newline character or separator between each file it saves.
I'm sure it's something super simple that I'm missing since I just started trying to pick up shell scripting last week, but nothing I have been able to find online has helped me resolve this. Any help would be greatly appreciated. Thanks
You can replace the files variable assignment and for loop with one command make this the script:
if [ "$1" = "311" ];
then
course="'/path/to/311/folder/$2'"
elif [ "$1" = "411" ];
then
course="'/path/to/411/folder/$2'"
elif [ "$1" = "516" ];
then
course="'/path/to/516/folder/$2'"
elif [ "$1" = "530" ];
then
course="'/path/to/530/folder/$2'"
elif [ "$1" = "599" ];
then
course="'/path/to/599/folder/$2'"
fi
mv -t $course $(mdfind 'kMDItemUserTags='$1'' -onlyin /Users/user/Downloads | sed ':a;N;$!ba;s/\n/ /g)
The sed ':a;N;$!ba;s/\n/ /g command simply replaces the newline characters with spaces, and the -t option for mv simply makes mv take the destination as the first argument.
You're getting rather confused about how quoting works in the shell. First rule: quotes go around data, not in data. For example, you use:
course="'/path/to/311/folder/$2'"
...
mv $file $course
When you set course this way, the double-quotes are treated as shell syntax (i.e. they change how what's between them is parsed), but the single-quotes are stored as part of the variable's value, and will thereafter be treated as data. When you use this variable in the mv command, it's actually looking for a directory literally named single-quote, and under that a directory named "path", etc. Instead, just put the appropriate quotes for how you want it parsed at that point, and then double-quotes around the variable when you use it (to prevent probably-unwanted word splitting and wildcard expansion). Like this:
course="/path/to/311/folder/$2"
...
mv "$file" "$course" # This needs more work -- see below
Also, where you have:
mdfind 'kMDItemUserTags='$1'' -onlyin /Users/user/Downloads
that doesn't really make any sense. You've got a single-quoted section, 'kMDItemUserTags=' where the quotes have no effect at all (single-quotes suppress all special meanings that characters have, like $ introducing variable substitution, but there aren't any characters there with special meanings, so no reason for the quotes), followed by $ without double-quotes around it, meaning that some special characters (whitespace and wildcards) in its value will get special parsing (which you probably don't want), followed by a zero-length single-quoted string, '', which parses out to exactly nothing. You want the $1 part in double-quotes; some people also include the rest of the string in the double-quoted section, which has no effect at all. In fact, other than the $2 part (and the spaces between parameters), you can quote or not however you want. Thus, any of these would work equivalently:
mdfind kMDItemUserTags="$1" -onlyin /Users/user/Downloads
mdfind "kMDItemUserTags=$1" -onlyin /Users/user/Downloads
mdfind "kMDItemUserTags=$1" '-onlyin' '/Users/user/Downloads'
mdfind 'kMDItemUserTags'="$1" '-'"only"'in' /'Users'/'user'/'Down'loads
...etc
Ok, next problem: parsing the output from mdfind from a series of characters into separate filepaths. This is actually tricky. If you put double-quotes around the resilting string, it'll get treated as one long filepath that happens to contain some newlines in it (which is totally legal, but not what you want). If you don't double-quote it, it'll be split into separate filepaths based on whitespace (not just newlines, but also spaces and tabs -- and spaces are common within macOS filenames), and anything that looks like a wildcard will get expanded to a list of matching filenames. This tends to cause chaos.
The solution: there's one character than cannot occur in a filepath, the ASCII NULL (character code 0), and mdfind -0 will output its list delimited with null characters. You can't put the result in a shell variable (they can't hold nulls either), but you can pass it through a pipe to, say, xargs -0, which will (thanks to the -0 option) parse the nulls as delimiters, and build commands out of the results. There is one slightly tricky thing: you want xargs to put the filepaths it gets in the middle of the argument list to mv, not at the end like it usually does. The -J option lets you tell it where to add arguments. I'll also suggest two safety measures: the -p option to xargs makes it ask before actually executing the command (use this at least until you're sure it's doing the right thing), and the -n option to mv, which tells it not to overwrite existing files if there's a naming conflict. The result is something like this:
mdfind -0 kMDItemUserTags="$1" -onlyin /Users/user/Downloads | xargs -0 -p -J% mv -n % "$course"
It is a good point to consider about filenames with whitespaces.
However the problem is that you are not quoting the filename in the mv command. Please take a look of a simple example below:
filename="with space.txt"
=> assign a variable to a filname with a space
touch "$filename"
=> create a file "with space.txt"
str="'$filename'"
=> wrap with single quotes (as you do)
echo $str
=> yields 'with space.txt' and may look good, which is a pitfall
mv $str "newname.txt"
=> causes an error
The mv command above causes an error because the command is invoked with
three arguments as: mv 'with space.txt' newname.txt. Unfortunately
the pre-quoting with single quotes is meaningless.
Instead, please try something like:
if [ "$1" = "311" ]; then
course="/path/to/311/folder/$2"
elif [ "$1" = "411" ]; then
course="/path/to/411/folder/$2"
elif [ "$1" = "516" ]; then
course="/path/to/516/folder/$2"
elif [ "$1" = "530" ]; then
course="/path/to/530/folder/$2"
elif [ "$1" = "599" ]; then
course="/path/to/599/folder/$2"
else
# illegal value in $1. do some error handling
fi
# the lines above may be simplified if /path/to/*folder/ have some regularity
mdfind "kMDItemUserTags=$1" -onlyin /Users/user/Downloads | while read -r file; do
mv "$file" "$course"
done
# the syntax above works as long as the filenames do not contain newline characters
I was looking for the best way to find iterate over files in a variables path and came across this question.
However, this and every other solution I've found uses a literal path rather than a variable, and I believe this is my problem.
for file in "${path}/*"
do
echo "INFO - Checking $file"
[[ -e "$file" ]] || continue
done
Even though there are definitely files in the directory (and if i put one of the literal paths in place of ${path} I get the expected result), this always only iterates once, and the value of $file is always the literal value of ${path}/* without any globbing.
What am I doing wrong?
Glob expansion doesn't happen inside quotes (both single and double) in shell.
You should be using this code:
for file in "$path"/*; do
echo "INFO - Checking $file"
[[ -e $file ]] || continue
done
I am working on a script and I need to compare a filename to another one and look for specific changes (in this case a "(x)" added to a filename when OS X needs to add a file to a directory, when a filename already exists) so this is an excerpt of the script, modified to be tested without the rest of it.
#!/bin/bash
p2_s2="/Path/to file (name)/containing - many.special chars.docx.gdoc"
next_line="/Path/to file (name)/containing - many.special chars.docx (1).gdoc"
file_ext=$(echo "${p2_s2}" | rev | cut -d '.' -f 1 | rev)
file_name=$(basename "${p2_s2}" ".${file_ext}")
file_dir=$(dirname "${p2_s2}")
esc_file_name=$(printf '%q' "${file_name}")
esc_file_dir=$(printf '%q' "${file_dir}")
esc_next_line=$(printf '%q' "${next_line}")
if [[ ${esc_next_line} =~ ${esc_file_dir}/${esc_file_name}\ \(?\).${file_ext} ]]
then
echo "It's a duplicate!"
fi
What I'm trying to do here is detect if the file next_line is a duplicate of p2_s2. As I am expecting multiple duplicates, next_line can have a (1) appended at the end of a filename or any other number in brackets (Although I am sure no double digits). As I can't do a simple string compare with a wildcard in the middle, I tried using the "=~" operator and escaping all the special chars. Any idea what I'm doing wrong?
You can trim ps2_s2's extension, trim next_line's extension including the number inside the parenthesis and see if you get the same file name. If you do - it's a duplicate. In order to do so, [[ allows us to perform a comparison between a string and a Glob.
I used extglob's +( ... ) pattern, so I can use +([0-9]) to match the number inside the parenthesis. Notice that extglob is enabled by shopt -s extglob.
#!/bin/bash
p2_s2="/Path/to/ps2.docx.gdoc"
next_line="/Path/to/ps2(1).docx.gdoc"
shopt -s extglob
if [[ "${p2_s2%%.*}" = "${next_line%%\(+([0-9])\).*}" ]]; then
printf '%s is a duplicate of %s\n' "$next_line" "$p2_s2"
fi
EDIT:
I now see that you've edited your question, so in case this solution is not enough, I'm positive that it'll be a good template to work with.
The (1) in next_line doesn't come before the final . it comes before the second to final . in the original filename but you only strip off a single . as the extension.
So when you generate the comparison filename you end up with /Path/to\ file\ \(name\)/containing\ -\ many.special\ chars.docx\ \(?\).gdoc which doesn't match what you expect.
If you had added set -x to the top of your script you'd have seen what the shell was actually doing and seen this.
What does OS X actually do in this situation? Does it add (#) before .gdoc? Does it add it before.docx`? Does it depend on whether OS X knows what the filename is (it is some type it can open natively)?
I am looking to search for strings within a file using variables.
I have a script that will accept 3 or 4 parameters: 3 are required; the 4th isn't mandatory.
I would like to search the text file for the 3 parameters matching within the same line, and if they do match then I want to remove that line and replace it with my new one - basically it would update the 4th parameter if set, and avoid duplicate entries.
Currently this is what I have:
input=$(egrep -e '$domain\s+$type\s+$item' ~/etc/security/limits.conf)
if [ "$input" == "" ]; then
echo $domain $type $item $value >>~/etc/security/limits.conf
echo \"$domain\" \"$type\" \"$item\" \"$value\" has been successfully added to your limits.conf file.
else
cat ~/etc/security/limits.conf | egrep -v "$domain|$type|$item" >~/etc/security/limits.conf1
rm -rf ~/etc/security/limits.conf
mv ~/etc/security/limits.conf1 ~/etc/security/limits.conf
echo $domain $type $item $value >>~/etc/security/limits.conf
echo \"$domain\" \"$type\" \"$item\" \"$value\" has been successfully added to your limits.conf file.
exit 0
fi
Now I already know that the input=egrep etc.. will not work; it works if I hard code some values, but it won't accept those variables. Basically I have domain=$1, type=$2 and so on.
I would like it so that if all 3 variables are not matched within one line, than it will just append the parameters to the end of the file, but if the parameters do match, then I want them to be deleted, and appended to the file. I know I can use other things like sed and awk, but I have yet to learn them.
This is for a school assignment, and all help is very much appreciated, but I'd also like to learn why and how it works/doesn't, so if you can provide answers to that as well that would be great!
Three things:
To assign the output of a command, use var=$(cmd).
Don't put spaces around the = in assignments.
Expressions don't expand in single quotes: use double quotes.
To summarize:
input=$(egrep -e "$domain\s+$type\s+$item" ~/etc/security/limits.conf)
Also note that ~ is your home directory, so if you meant /etc/security/limits.conf and not /home/youruser/etc/security/limits.conf, leave off the ~
You have several bugs in your script. Here's your script with some comments added
input=$(egrep -e '$domain\s+$type\s+$item' ~/etc/security/limits.conf)
# use " not ' in the string above or the shell can't expand your variables.
# some versions of egrep won't understand '\s'. The safer, POSIX character class is [[:blank:]].
if [ "$input" == "" ]; then
# the shell equality test operator is =, not ==. Some shells will also take == but don't count on it.
# the normal way to check for a variable being empty in shell is with `-z`
# you can have problems with tests in some shells if $input is empty, in which case you'd use [ "X$input" = "X" ].
echo $domain $type $item $value >>~/etc/security/limits.conf
# echo is unsafe and non-portable, you should use printf instead.
# the above calls echo with 4 args, one for each variable - you probably don't want that and should have double-quoted the whole thing.
# always double-quote your shell variables to avoid word splitting ad file name expansion (google those - you don't want them happening here!)
echo \"$domain\" \"$type\" \"$item\" \"$value\" has been successfully added to your limits.conf file.
# the correct form would be:
# printf '"%s" "%s" "%s" "%s" has been successfully added to your limits.conf file.\n' "$domain" "$type" "$item" "$value"
else
cat ~/etc/security/limits.conf | egrep -v "$domain|$type|$item" >~/etc/security/limits.conf1
# Useless Use Of Cat (UUOC - google it). [e]grep can open files just as easily as cat can.
rm -rf ~/etc/security/limits.conf
# -r is for recursively removing files in a directory - inappropriate and misleading when used on a single file.
mv ~/etc/security/limits.conf1 ~/etc/security/limits.conf
# pointless to remove the file above when you're overwriting it here anyway
# If your egrep above failed to create your temp file (e.g. due to memory or permissions issues) then the "mv" above would zap your real file. the correct way to do this is:
# egrep regexp file > tmp && mv tmp file
# i.e. use && to only do the mv if creating the tmp file succeeded.
echo $domain $type $item $value >>~/etc/security/limits.conf
# see previous echo comments.
echo \"$domain\" \"$type\" \"$item\" \"$value\" has been successfully added to your limits.conf file.
# ditto
exit 0
# pointless and misleading having an explicit "exit <success>" when that's what the script will do by default anyway.
fi
This line:
input=$(egrep -e '$domain\s+$type\s+$item' ~/etc/security/limits.conf)
requires double quotes around the regex to allow the shell to interpolate the variable values.
input=$(egrep -e "$domain\s+$type\s+$item" ~/etc/security/limits.conf)
You need to be careful with backslashes; you probably don't have to double them up in this context, but you should be sure you know why.
You should be aware that your first egrep commands is much more restrictive in what it selects than the second egrep which is used to delete data from the file. The first requires the entry with the three fields in the single line; the second only requires a match with any one of the words (and that could be part of a larger word) to delete the line.
Since ~/etc/security/limits.conf is a file, there is no need to use the -r option of rm; it is advisable not to use the -r unless you intend to remove directories.
I am writing a simple shell script to make automated backups, and I am trying to use basename to create a list of directories and them parse this list to get the first and the last directory from the list.
The problem is: when I use basename in the terminal, all goes fine and it gives me the list exactly as I want it. For example:
basename -a /var/*/
gives me a list of all the directories inside /var without the / in the end of the name, one per line.
BUT, when I use it inside a script and pass a variable to basename, it puts single quotes around the variable:
while read line; do
dir_name=$(echo $line)
basename -a $dir_name/*/ > dir_list.tmp
done < file_with_list.txt
When running with +x:
+ basename -a '/Volumes/OUTROS/backup/test/*/'
and, therefore, the result is not what I need.
Now, I know there must be a thousand ways to go around the basename problem, but then I'd learn nothing, right? ;)
How to get rid of the single quotes?
And if my directory name has spaces in it?
If your directory name could include spaces, you need to quote the value of dir_name (which is a good idea for any variable expansion, whether you expect spaces or not).
while read line; do
dir_name=$line
basename -a "$dir_name"/*/ > dir_list.tmp
done < file_with_list.txt
(As jordanm points out, you don't need to quote the RHS of a variable assignment.)
Assuming your goal is to populate dir_list.tmp with a list of directories found under each directory listed in file_with_list.txt, this might do.
#!/bin/bash
inputfile=file_with_list.txt
outputfile=dir_list.tmp
rm -f "$outputfile" # the -f makes rm fail silently if file does not exist
while read line; do
# basic syntax checking
if [[ ! ${line} =~ ^/[a-z][a-z0-9/-]*$ ]]; then
continue
fi
# collect targets using globbing
for target in "$line"/*; do
if [[ -d "$target" ]]; then
printf "%s\n" "$target" >> $outputfile
fi
done
done < $inputfile
As you develop whatever tool will process your dir_list.tmp file, be careful of special characters (including spaces) in that file.
Note that I'm using printf instead of echo so that targets whose first character is a hyphen won't cause errors.
This might work
while read; do
find "$REPLY" >> dir_list.tmp
done < file_with_list.txt