I have a question concerning why this doesn't work. Probably, it's a simple answer, but I just can't seem to figure it out.
I want to move a couple of files I have. They all have the same filename (let's say file1) but they are all in different directories (lets say /tmp/dir1,dir2 and dir3). If I were to move these individually I could do something along the lines of:
mv /tmp/dir1/file1 /tmp
That works. However, I have multiple directories and they're all going to end up in the same spot....AND I don't want to overwrite. So, I tried something like this:
mv /tmp/{dir1,dir2,dir3}/file1 /tmp/file1.{a,b,c}
When I try this I get:
/tmp/file1.c is not a directory
Just to clarify...this also works:
mv /tmp/dir1/file1 /tmp/file1.c
Pretty sure this has to do with brace expansion but not certain why.
Thanks
Just do echo to understand how the shell expands:
$ echo mv /tmp/{dir1,dir2,dir3}/file1 /tmp/file1.{a,b,c}
mv /tmp/dir1/file1 /tmp/dir2/file1 /tmp/dir3/file1 /tmp/file1.a /tmp/file1.b /tmp/file1.c
Now you can see that your command is not what you want, because in a mv command, the destination (directory or file) is the last argument.
That's unfortunately now how the shell expansion works.
You'll have to probably use an associative array.
!/bin/bash
declare -A MAP=( [dir1]=a [dir2]=b [dir3]=c )
for ext in "${!MAP[#]}"; do
echo mv "/tmp/$ext/file1" "/tmp/file1.${MAP[$ext]}"
done
You get the following output when you run it:
mv /tmp/dir2/file1 /tmp/file1.b
mv /tmp/dir3/file1 /tmp/file1.c
mv /tmp/dir1/file1 /tmp/file1.a
Like with many other languages key ordering is not guaranteed.
${!MAP[#]} returns an array of all the keys, while ${MAP[#]} returns the an array of all the values.
Your syntax of /tmp/{dir1,dir2,dir3}/file1 expands to /tmp/dir1/file /tmp/dir2/file /tmp/dir3/file. This is similar to the way the * expansion works. The shell does not execute your command with each possible combination, it simply executes the command but expands your one value to as many as are required.
Perhaps instead of a/b/c you could differentiate them with the actual number of the dir they came from?
$: for d in 1 2 3
do echo mv /tmp/dir$d/file1 /tmp/file1.$d
done
mv /tmp/dir1/file1 /tmp/file1.1
mv /tmp/dir2/file1 /tmp/file1.2
mv /tmp/dir3/file1 /tmp/file1.3
When happy with it, take out the echo.
A relevant point - brace expansion is not a wildcard. It has nothing to do with what's on disk. It just creates strings.
So, if you create a bunch of files named with single letters or digits, echo ? will wildcard and list them all, but only the ones actually present. If there are files for vowels but not consonants, only the vowels will show. But -
if you say echo {foo,bar,nope} it will output foo bar nope regardless of whether or not any or all of those exist as files or directories, etc.
Related
I have a folder structure where two files are in a folder. The files have long names, yet are distinguished by R1 and R2. Note I am running this over many folders using the for loop but keeping it simple for this example. I am running a loop and am wonder how to correctly call the files with a (*) star character to autocomplete without having to type in all file name. My attempt is below:
#!/bin/bash
for item in Folder_Directory:
do
forward=$item/*R1*
reverse=$item/*R2*
bbmap.sh ref=reference.fna in1=$forward in2=$reverse outu=Unmapped.fasta
done
The output I am getting is an error because the variable is not identifying the desired file:
Error:
align2.BBMap build=1 overwrite=true fastareadlen=500 ref=reference.fna
in1=Folder_Dictory/*R1* in2=Folder_Dictory/*R2* outu=Folder_Dictory/Unmapped.fastq
In this example I could autocomplete the files, however, when I expand this loop to include multiple folders that is no longer ideal. Autocompleting using (*) characters was my first approach, any other suggestions or fixes to my issue are greatly appreciated.
The problem is that the shell sees in1=Folder_Dictory/*R1* and notices that there are no files which match the glob with the literal in1= prefix, and so the wildcard does not get expanded at all.
You probably want to evaluate the wildcard before passing it to the command, like for instance
for item in Folder_Directory:
do
forward=$item/*R1*
reverse=$item/*R2*
bbmap.sh ref=reference.fna in1="$(echo $forward)" in2="$(echo $reverse)" outu=Unmapped.fasta
done
This will of course still be erratic if the wildcard expands to more than one file.
If you want only two files from your folder_structure, then i believe it would be good to use find to search for the files and assign then into separate variables as per your requirement...don't see use of for loop here.
forward=$(find Folder_Directory -type f -name "*R1*")
reverse=$(find Folder_Directory -type f -name "*R2*")
bbmap.sh ref=reference.fna in1="$forward" in2="$reverse" outu=Unmapped.fasta
It works like this:
test=f*
$ echo $test
file
But
$ echo "$test"
f*
And
test2=$test
$ echo "$test" $test2
f* file
$ echo "$test" "$test2"
f* f*
To make it work, you have to do something like this:
test3="$(echo $test)"
$ echo "$test" "$test2" "$test3"
f* f* file
I'm writing a small piece of code that checks for .mov files in a specific folder over 4gb and writes it to a log.txt file by name (without an extension). I'm then reading the names into a while loop line by line which signals some archiving and copying commands.
Consider a file named abcdefg.mov (new) and a corresponding folder somewhere else named abcdefg_20180525 (<-*underscore timestamp) that also contains a file named abcedfg.mov (old).
When reading in the filename from the log.txt, I strip the extension to store the variable "abcdefg" ($in1) and i'm using that variable to locate a folder elsewhere that contains that matching string at the beginning.
My problem is with how the mv command seems to support a wild card in the "source" string, but not in the "destination" string.
For example i can write;
mv -f /Volumes/Myshare/SourceVideo/$in1*/$in1.mov /Volumes/Myshare/Archive
However a wildcard on the destination doesn't work in the same way. For example;
mv -f /Volumes/Myshare/Processed/$in1.mov Volumes/Myshare/SourceVideo/$in1*/$in1.mov
Is there an easy fix here that doesn't involve using another method?
Cheers for any help.
mv accepts a single destination path. Suppose that $in1 is abcdfg, and that $in1* expands to abcdefg_20180525 and abcdefg_20180526. Then the command
mv -f /dir1/$in1 /dir2/$in1*/$in1.mov
will be equivalent to:
mv -f /dir1/abcdefg.mov /dir2/abcdefg_20180526/abcdefg.mov
mv -f /dir1/abcdefg.mov /dir2/abcdefg_20180526/abcdefg.mov
mv -f /dir2/abcdefg_20180525/abcdefg.mov /dir2/abcdefg_20180526/abcdefg.mov
Moreover, because the destination file is the same in all three cases, the first two files will be overwritten by the third.
You should create a precise list and do a precise copy instead of using wild cards.
This is what I would probably do, generate a list of results in a file with FULL path information, then read those results in another function. I could have used arrays but I wanted to keep it simple. At the bottom of this script is a function call to scan for files of EXT mp4 (case insensitive) then writes the results to a file in tmp. then the script reads the results from that file in another function and performs some operation (mv etc.). Note, if functions are confusing , you can just remove the function name { } and name calls and it becomes a normal script again. functions are really handy, learn to love them!
#!/usr/bin/env bash
readonly SIZE_CHECK_LIMIT_MB="10M"
readonly FOLDER="/tmp"
readonly DESTINATION_FOLDER="/tmp/archive"
readonly SAVE_LIST_FILE="/tmp/$(basename $0)-save-list.txt"
readonly EXT="mp4"
readonly CASE="-iname" #change to -name for exact ext type upper/lower
function find_files_too_large() {
> ${SAVE_LIST_FILE}
find "${FOLDER}" -maxdepth 1 -type f "${CASE}" "*.${EXT}" -size +${SIZE_CHECK_LIMIT_MB} -print0 | while IFS= read -r -d $'\0' line ; do
echo "FOUND => $line"
echo "$line" >> ${SAVE_LIST_FILE}
done
}
function archive_large_files() {
local read_file="${SAVE_LIST_FILE}"
local write_folder="$DESTINATION_FOLDER"
if [ ! -s "${read_file}" ] || [ ! -f "${read_file}" ] ;then
echo "No work to be done ... "
return
fi
while IFS= read -r line ;do
echo "mv $line $write_folder" ;sleep 1
done < "${read_file}"
}
# MAIN (this is where the script starts) We just call two functions.
find_files_too_large
archive_large_files
it might be easier, i think, to change the filenames to the folder name initially. So abcdefg.mov would be abcdefg_timestamp.mov. I can always strip the timestamp from the filename easy enough after its copied to the right location. I was hoping i had a small syntax issue but i think there is no easy way of doing what i thought i could...
I think you have a basic misunderstanding of how wildcards work here. The mv command doesn't support wildcards at all; the shell expands all wildcards into lists of matching files before they get passed to the mv command as wildcards. Furthermore, the mv command doesn't know if the list of arguments it got came from wildcards or not, and the shell doesn't know anything about what the command is going to do with them. For instance, if you run the command grep *, the grep command just gets a list of names of files in the current directory as arguments, and will treat the first of them as a regex pattern ('cause that's what the first argument to grep is) to search the rest of the files for. If you ran mv * (note: don't do this!), it will interpret all but the last filename as sources, and the last one as a destination.
I think there's another source of confusion as well: when the shell expands a string containing a wildcard, it tries to match the entire thing to existing files and/or directories. So when you use Volumes/Myshare/SourceVideo/$in1*/$in1.mov, it looks for an already-existing file in a matching directory; AIUI the file isn't there yet, there's no match. What it does in that case is pass the raw (unexpanded) wildcard-containing string to mv as an argument, which looks for that exact name, doesn't find it, and gives you an error.
(BTW, should there be a "/" at the front of that pattern? I assume so below.)
If I understand the situation correctly, you might be able to use this:
mv -f /Volumes/Myshare/Processed/$in1.mov /Volumes/Myshare/SourceVideo/$in1*/
Since the filename isn't supplied in the second string, it doesn't look for existing files by that name, just directories with the right prefix; mv will automatically retain the filename from the source.
However, I'll echo #Sergio's warning about chaos from multiple matches. In this case, it won't overwrite files (well, it might, but for other reasons), but if it gets multiple matching target directories it'll move all but the last one into the last one (along with the file you meant to move). You say you're 100% certain this won't be a problem, but in my experience that means that there's at least a 50% chance that something you'd never have thought of will go ahead and make it happen anyway. For instance, is it possible that $in1 could wind up empty, or contain a space, or...?
Speaking of spaces, I'd also recommend double-quoting all variable references. You want the variables inside double-quotes, but the wildcards outside them (or they won't be expanded), like this:
mv -f "/Volumes/Myshare/Processed/$in1.mov" "/Volumes/Myshare/SourceVideo/$in1"*/
My script reads the name of folders in current directory:
for D in */ ; do
echo $D
done
The result is:
folder1/
folder2/
folder3/
But when I want to use $D like this:
mv *.zip /home/user/$D/zipfolder
it makes two /-s like /home/user/folder1//zipfolder
and it fails
How do I remove the extra /?
If you are on bash version 4.2 or higher you can use ${D:0:-1} as support for negative length was added.
If you are on an earlier version you would need to write ${D:0:${#D}-1}
But as suggested correctly in a comment below, it might be the case you don't have a slash as a last character, still with that solution above the last character will be removed...
If this is not acceptable, you might want to use ${D%/} as nothing happens when the last character isn't a /
for D in */
do
echo $D
ls /home/user/$D/zipfolder
done
The example with mv *.zip doesn't make much sense, since once the files are moved, why would you need a for-loop? So I changed the sample to something better reproducible.
Aaron made the suggestions I would have given, but since he didn't answer, and Kristianmitk's answer isn't satisfying, I repeat the comment as an answer.
a) Two slashes don't hurt.
ls a/b
ls a//b
ls a/./b
all evaluate to the same directory. This is handy in your case, since the slash is a natural terminator for the variable name
ls a/b/c
ls a/bb/c
ls a/bb/./c
D=bb/
ls a/$D/c
ls a/$D./c
Even the dot terminates a variable name, so the last line will work, too.
b) If you're irritated by two slashes, you may use curly braces, which is the canonical way, to isolate variable names from surrounding characters:
ls a/${D}c
or
for D in */
do
echo $D
ls /home/user/${D}zipfolder
done
I have a set of files that I download that contain files that I want to remove. I would like to create a list of some form, the script should support blobbing so I can be pretty aggressive with file removal without getting into the complexities of using regex within the list of files.
I am also stumped in that I put a sleep command within the loop of my script, and that is not getting run after each iteration, but only once at the end of run.
Here is the script
# Get to the place where all the durty work happens
cd /Volumes/Videos
FILES=".DS_Store
*.txt
*.sample
*.sample.*
*.samples"
if [ "$(pwd)" == "/Volumes/Videos" ]; then
echo "You are currently in $(pwd)"
echo "You would not have read the above if this script were operating anywhere else"
# Dekete fikes from list above
for f in "$FILES"
do
echo "Removing $f";
rm -f "$f";
echo "$f has been deleted";
sleep 10;
echo "";
echo "";
done
# See if dir is empty, ask if we want to delete it or keep it
# Iterate evert movie file, see if we want to nuke contents. Maybe use part of last openned to help find those files fast
else
# Not in the correct directory
echo "This script is trying to alter files in a location that it should not be working"
echo "Script is currently trying to work in $(pwd)"
exit 1
fi
The main thing that has be completely stumped is the sleep command. It is run once, not once per file iteration. If I have 100 files to go through I get 10 seconds of sleep, not 100*10.
I will be adding in some other features, like if a file is smaller than x bytes, go ahead and delete it too. These files will have spaces and other odd characters in the filenames, am I creating my variables correctly to make this script handle those scenarios as well as be as POSIX compliant as possible. I will change the shebang to sh over bash and try to add in set -o noun set and set -o err exit though I tend to have a lot of trouble when I do that.
Is there a better form of list I should be using? I am not objectionable to storing the pattern match list in a separate file. I can include it, or read it in with any of a few commands.
These are also nested files, a dir, that contains files, or a dir that contains a dir that contains some files. Something like this:
/Volumes/Videos:
The Great guy in a tree
The Great guy in a tree S01e01
sample.avi
readme.txt
The Great guy in a tree S01e01.mpg
The Great guy in a tree S01e02
The Great guy in a tree S01e02.mpg
The Great guy in a tree S01e03
The Great guy in a tree S01e03.mpg
The Great guy in a tree S01e04
The Great guy in a tree S01e04.mpg
Thank you.
The reason that your script is not working as you expect is because your for loop is written incorrectly. This example shows what is going on:
$ i=0
$ FILES=".DS_Store
*.txt
*.sample
*.sample.*
*.samples"
$ for f in "$FILES"; do echo $((++i)) "$f"; done
1 .DS_Store
*.txt
*.sample
*.sample.*
*.samples
Note that only one number is output, indicating that the loop is only going around once. Also, no pathname expansion has occurred.
In order to make your script work as you expect, you can remove the quotes around "$FILES". This means that each word in your string will be evaluated separately, rather than all at once. It also means that pathname expansion of the wildcards that you are using will occur, so all files ending in .txt will be removed, which I guess is what you meant.
Instead of using a string to store your list of expressions, you might prefer to make use of an array:
FILES=( '.DS_Store' '*.txt' '*.sample' '*.sample.*' '*.samples' )
The quotes around each element prevent expansion (so the array only has 5 elements, not the fully expanded list). You could then change your loop to for f in ${FILES[#]} (again, no double quotes results in each element of the list being expanded).
Although removing the quotes fixes your script, I would agree with #hek2mgl's suggestion of using find. It allows you to find files by name, size, date modified and a lot more in one line. If you want to pause between the deletion of each file, you could use something like this:
find \( -name "*.sample" -o -name "*.txt" \) -delete -exec sleep 10 \;
You can use find:
find -type f -name '.DS_Store' -o -name '*.txt' -o -name '*.sample.*' -o -name '*.samples' -delete
I have a script where inotifywait is piped into a while loop that executes the following logic.
cp "$S3"/2/post2.png "$S3";
mv "$S3"/1/post1.png "$S3"/2/post2.png;
cp "$S3"/3/post3.png "$S3";
mv "S3"/post2.png "$S3"/3/post3.png;
so forth and so on..... then at the end of the script...
mv "$dir"/$file "$S3"/1/post1.png
That line represents a fresh post, the above is the rotation of older post.
I can can hand code the iterations all the way down to 100+, but I would like to program more efficiently and save time.
So, what's some correct ways to loop this?
I think a better mechanism would list the directories in "$S3" in reverse numeric order, and arrange to process them like that. It isn't clear if the 100 directories are all present or whether they need to be created. We'll assume that directories 1..100 might exist, and directory N will always and only contain postN.png.
I'm assuming that there are no spaces, newlines or other awkward characters in the file paths; this means that ls can be used without too much risk.
for dirnum in $(cd "$S3"; ls */*.png | sed 's%/.*%%' | sort -nr)
do
next=$(($dirnum + 1))
mv "$S3/$dirnum/post$dirnum.png" "$S3/$next/post$next.png"
done
The cd "$S3" means I don't get a possibly long pathname included in the output; the ls */*.png lists the files that exist; the sed removes the file name and slash, leaving just a list of directory numbers containing files; and the sort puts the directories in reverse numeric order.
The rest is straight-forward, given the assumption that the necessary directories already exist. It would not be hard to add [ -d "$S3/$next" ] || mkdir -p "$S3/$next" before moving the file. Clearly, after the loop you can use your final command:
mv "$dir/$file" "$S3/1/post1.png"
Note that I've enclosed complete names in double quotes; it generally leads to fewer nasty surprises if something acquires spaces unexpectedly.
Try this:
for i in $(ls -r1 "$3"); do
mkdir -p "$3/$((i+1))"
mv "$3/$i/post$i.png" "$3/$((i+1))/post$((i+1)).png"
done
mv "$dir"/$file "$S3"/1/post1.png
The loop will iterate through all directories in reverse order and move the files.