I have a Bash script that takes in a directory as a parameter, and after some processing will do some output based on the files in that directory.
The command would be like the following, where dir is a directory with the following structure inside
dir/foo
dir/bob
dir/haha
dir/bar
dir/sub-dir
dir/sub-dir/joe
> myscript ~/files/stuff/dir
After some processing, I'd like the output to be something like this
foo
bar
sub-dir/joe
The code I have to remove the path passed in is the following:
shopt -s extglob
for file in $files ; do
filename=${file#${1}?(/)}
This gets me to the following, but for some reason the optional / is not being taken care of. Thus, my output looks like this:
/foo
/bar
/sub-dir/joe
The reason I'm making it optional is because if the user runs the command
> myscript ~/files/stuff/dir/
I want it to still work. And, as it stands, if I run that command with the trailing slash, it outputs as desired.
So, why does my ?(/) not work? Based on everything I've read, that should be the right syntax, and I've tried a few other variations as well, all to no avail.
Thanks.
that other guy's helpful answer solves your immediate problem, but there are two things worth nothing:
enumerating filenames with an unquoted string variable (for file in $files) is ill-advised, as sjsam's helpful answer points out: it will break with filenames with embedded spaces and filenames that look like globs; as stated, storing filenames in an array is the robust choice.
there is no strict need to change global shell option shopt -s extglob: parameter expansions can be nested, so the following would work without changing shell options:
# Sample values:
file='dir/sub-dir/joe'
set -- 'dir/' # set $1; value 'dir' would have the same effect.
filename=${file#${1%/}} # -> '/sub-dir/joe'
The inner parameter expansion, ${1%/}, removes a trailing (%) / from $1, if any.
I suggested you change files to an array which is a possible workaround for non-standard filenames that may contain spaces.
files=("dir/A/B" "dir/B" "dir/C")
for filename in "${files[#]}"
do
echo ${filename##dir/} #replace dir/ with your param.
done
Output
A/B
B
C
Here's the documentation from man bash under "Parameter Expansion":
${parameter#word}
${parameter##word}
Remove matching prefix pattern. The word is
expanded to produce a pattern just as in pathname
expansion. If the pattern matches the beginning of
the value of parameter, then the result of the
expansion is the expanded value of parameter with
the shortest matching pattern (the ``#'' case) or
the longest matching pattern (the ``##'' case)
deleted.
Since # tries to delete the shortest match, it will never include any trailing optional parts.
You can just use ## instead:
filename=${file##${1}?(/)}
Depending on what your script does and how it works, you can also just rewrite it to cd to the directory to always work with paths relative to .
Related
So I am wondering if there is any way to organize a directory on a mac with the terminal. I am a beginner with using the terminal and just seeing if this is possible.
I have a script that will scrape various pages and save certain data to a file (data irrelevant), such as this picture.
directory that needs organizing
I would like to know if I can write something that will read the file names and create directories that correspond. For example, it runs a loop that will read all files with "Year2014", create a folder named "Year2014", then place the files inside.
If you have any other questions, let me know!
The short answer is "Yes", and the longer answer is there are many ways to do it. Since you are using bash (or any POSIX shell), you have parameter expansion with substring removal available to help you trim text from the end of each filename to isolate the "YearXXXX" part of the filename that you can then use to (1) create the directory, and (2) move the file into the newly created directory.
Presuming Filenames Formatted WeekXXYearXXXX.txt
Take for example a simple for loop where the loop variable f will contain each filename in turn. You can isolate the "WeekXX" part of the name by using a parameter expansion that trims from the right of the string trough 'Y' leaving whatever "WeekXX" is. (save the result in a temporary variable) You can then use that temp variable to remove the "WeekXX" text from the original filename leaving "YearXXXX.txt". You then simply remove ".txt" from the first to arrive at the directory name to put the file in.
Scriptwise it would look like:
for f in *.txt; do ## loop over .txt files using variable $f
tmp="${f%%Y*}" ## remove though 'Y' from right
dname="${f#$tmp}" ## remove contents of tmp from left
dname="${dname%.txt}" ## remove .txt
mkdir -p "$dname" ## create dname (no error if exists)
mv "$f" "$dname" ## move $f to $dname
done
Where the temporary variable used is tmp and the final directory name is stored in the variable dname.
(note: you may want to use mv -i if you want mv to prompt before overwriting if the filename already exists in the target directory)
You can refer to man bash under the Parameter Expansion heading to read the specifics of each expansion which (among many more) are described as:
${var#pattern} Strip shortest match of pattern from front of $var
${var##pattern} Strip longest match of pattern from front of $var
${var%pattern} Strip shortest match of pattern from back of $var
${var%%pattern} Strip longest match of pattern from back of $var
Note this set of parameter expansions is POSIX so it will work with any POSIX shell, while most of the remaining expansions are bashisms (bash-only)
Let me know if you have further questions.
To rephrase - I want to use Bash command substitution and string substitution in the same line.
My actual commands are longer, but the ridiculous use of echo here is just a "substitution" for shortness and acts the same - with same errors ;)
I know we can use a Bash command to produce it's output string as a parameter for another command like this:
echo "$(echo "aahahah</ddd>")"
aahahah</ddd>
I also know we can remove last known part of a string like this:
var="aahahah</ddd>"; echo "${var%</ddd>}"
aahahah
I am trying to write a command where one command gives a string output, where I want to remove last part, which is known.
echo "${$(echo "aahahah</ddd>")%</ddd>}"
-bash: ${$(echo "aahahah</ddd>")%</ddd>}: bad substitution
It might be the order of things happening or substitution only works on variables or hardcoded strings. But I suspect just me missing something and it is possible.
How do I make it work?
Why doesn't it work?
When a dollar sign as in $word or equivalently ${word} is used, it asks for word's content. This is called parameter expansion, as per man bash.
You may write var="aahahah</ddd>"; echo "${var%</ddd>}": That expands var and performs a special suffix operation before returning the value.
However, you may not write echo "${$(echo "aahahah</ddd>")%</ddd>}" because there is nothing to expand once $(echo "aahahah</ddd>") is evaluated.
From man bash (my emphasis):
${parameter%word}
Remove matching suffix pattern. The word is expanded to produce a
pattern just as in pathname expansion. If the pattern
matches a trailing portion of the expanded value of parameter, then
the result of the expansion is the expanded value of parameter
with the shortest matching pattern (the ''%'' case) or the longest matching pattern (the ''%%'' case) deleted.
Combine your commands like this
var=$(echo "aahahah</ddd>")
echo ${var/'</ddd>'}
I have files, for example
- public_00000.jpg
- public_00001.jpg
- ...
- public_00535.jpg
But I want to make these files as
- public_05674.jpg
- public_05675.jpg
- ...
- public_06209.jpg
I mean, I want to increase the number in the filename by +5674 on the whole.
How can I do this by Shell or Command??
Thanks ahead:)
Could you please try following.
for file in *.jpg
do
first_filename_part="${file%_*}"
last_filename_part="${file#*.}"
var="${file#*_}"
count="${var%.*}"
((count = count + 5674))
printf "%s %s %s_%05d.%s\n" "mv" $file $first_filename_part $count $last_filename_part
done
Above will only print the commands on screen like:
mv public_00000.jpg public_05674.jpg
Try running only 1 command First from above printed output on your terminal, once you are Happy with results try following then, since this will rename all the files.
for file in *.jpg
do
first_filename_part="${file%_*}"
last_filename_part="${file#*.}"
var="${file#*_}"
count="${var%.*}"
((count = count + 5674))
printf "%s %s %s_%05d.%s\n" "mv" $file $first_filename_part $count $last_filename_part | bash
done
From man page: I have used parameter expansion mechanism.
${parameter#word}
${parameter##word} Remove matching prefix pattern. The word is
expanded to produce a pattern just as in pathname expansion. If the
pattern matches the beginning of the value of parameter, then the
result of the expansion is the expanded value of parameter with the
shortest matching pattern (the #'' case) or the longest matching
pattern (the##'' case) deleted. If parameter is # or *, the
pattern removal operation is applied to each positional parame- ter in
turn, and the expansion is the resultant list. If parameter is an
array variable subscripted with # or *, the pattern removal
operation is applied to each member of the array in turn, and the
expansion is the resultant list.
${parameter%word}
${parameter%%word} Remove matching suffix pattern. The word is
expanded to produce a pattern just as in pathname expansion. If the
pattern matches a trailing portion of the expanded value of parameter,
then the result of the expansion is the expanded value of parameter
with the shortest matching pattern (the %'' case) or the longest
matching pattern (the%%'' case) deleted. If parameter is # or *,
the pattern removal operation is applied to each positional parameter
in turn, and the expansion is the resultant list. If parameter is an
array variable subscripted with # or *, the pattern removal operation
is applied to each member of the array in turn, and the expansion is
the resultant list.
You can use Perl rename like this to do an "evaluated substitution" - that's the e right at the end:
rename --dry-run 's|(\d+)|sprintf("%05d",$1+5674)|e' pub*jpg
Sample Output
'public_00000.jpg' would be renamed to 'public_05674.jpg'
'public_00001.jpg' would be renamed to 'public_05675.jpg'
In case you are unfamiliar with Perl, the command basically says:
rename "substitute|THIS|with THAT|" IN_THESE_FILENAMES
In your case, THIS is \d+ which means "one or more digits" and that is enclosed within parentheses to make a "capture group". That group can then be referred to in the substitution on the right side by $1 since it is the first capture group.
The THAT in your case is simply a print statement that prints the first capture group $1 incremented by 5674 in a field that is zero-padded to be 5 digits wide using %05d.
Using Perl rename has the benefits that:
you can do a "dry run" to see what it would do without actually doing anything
it will not clobber (overwrite) files without warning
it is fast - it doesn't create a process for sed and another for mv for every single file, it just starts a single Perl interpreter and makes a library call to rename each file
it will automagically create any intermediate directories needed, if you wish
you can use the full power of Perl to do as much substitution or calculation as you wish
Note for macOS users... Perl is installed on macOS by default, so if you use homebrew to install your packages, you just need:
brew install rename
Note for Linux users... there are several rename packages, the one I am referring to is sometimes called prename, or "Perl rename". That means, if you run file on the rename command, it should say it's a Perl script like this:
file $(which rename)
/usr/local/bin/rename: Perl script text executable
I have a programme that is generating files like this "Incoming11781Arp", and there is always Incoming, and there is always 5 numbers, but there are 3 letters/upper-case/lower-case/numbers/special case _ in any way. Like Incoming11781_pi, or Incoming11781rKD.
How can I delete them using a script run from a cron job please? I've tried -
#!/bin/bash
file=~/Mail/Incoming******
rm "$file";
but it failed saying that there was no matching file or directory.
You mustn't double-quote the variable reference for pathname expansion to occur - if you do, the wildcard characters are treated as literals.
Thus:
rm $file
Caveat: ~/Mail/Incoming****** doesn't work the way you think it does and will potentially match more files than intended, as it is equivalent to ~/Mail/Incoming*, meaning that any file that starts with Incoming will match.
To only match files starting with Incoming that are followed by exactly 6 characters, use ~/Mail/Incoming??????, as #Jidder suggests in a comment.
Note that you could make your glob (pattern) even more specific:
file=~/Mail/Incoming[0-9][0-9][0-9][0-9][0-9][[:alpha:]_][[:alpha:]_][[:alpha:]_]
See the bash manual for a description of pathname expansion and pattern syntax: http://www.gnu.org/software/bash/manual/bashref.html#index-pathname-expansion.
You can achieve the same effect with the find command...
$ directory='~/Mail/'
$ file_pattern='Incoming*'
$ find "${directory}" -name "${file_pattern}" -delete
The first two lines define the directory and the file pattern separately, the find command will then proceed to delete any matching files inside that directory.
I have a script which contains the following line:
propFile="${0%/*}/anteater.properties"
What does "${0%/*}" mean?
This command gives a path to the script - but there is a spaces at path and script can't find this file - how to deal with it?
The % operator in variable expansion removes the matching suffix pattern given to it. So ${0%/*} takes the variable $0, and removes all matching /* at the end. This is equivalent to the command dirname, which, when given a path, returns the parent directory of that path.
In order to deal with spaces in bash variable, whenever expanding the variable (i.e. whenever you write $var), you should quote it. In short, always use "$var" instead of just $var.
Consider reading shell parameter expansion and variable quoting in the bash manual to learn more about these two subjects.
strips the suffix matching /*, i.e. everything after last slash including the slash itself.
quote it wherever you use it (cat "$propFile").