What scripts do you regularly use to improve your productivity?
Over the last year I have been trying to use bash scripts and commands to improve my productivity as a developer (Web and applications). Here is a list of a few simple ones that I use:
Make files lower case:
for i in *.txt; do mv "$i" "`echo $i| tr [A-Z] [a-z]`"; done
Test whether a tag exists in subversion:
if [ "`svn ls http://www.mysvnserver.co.uk/myproject/tags | grep it-0.7.0.1/`" = "it-0.7.0.1/" ]; then echo YES; else echo NO; fi
Rename all JPG files in the current directory and add an increment:
j=16;for i in *.jpg; do mv "$i" "gallery_"$j".jpg"; j=$(($j+1)); done;ls
Fix a misspelling in a group of filenames:
for i in aples*.jpg; do mv $i ${i/aples/apples} ; done
For more see here:
http://blog.emson.co.uk/2009/06/18-useful-bash-scripts-for-web-developers
What scripts do you use?
Thanks...
You have a good source of such command at CommandLineFu.
Related
I take pictures for my job on three different cameras. I am looking to automate the process of organising these to folders based on the first 3 letters and then the filetype.
Unfortunately my bash script syntax knowledge is non existent and so I can't figure out more than creating the directories..
An eg of the incoming files:
HAM1234.JPG
HAM1234.RAW
HDR1234.JPG
HDR1234.RAW
STL1234.JPG
STL1234.RAW
These would go into 3 folders
HAM - REF/{RAW,JPG}
HDR - HDRI/{RAW,JPG}
STL - STILLS/{RAW,JPG}
With the filetypes being aligned.
Any help would be much appreciated!
Jason
With your shown examples:
#!/bin/bash
mkdir -p {HAM\ -\ REF,HDR\ -\ HDRI,STL\ -\ STILLS}/{RAW,JPG}
shopt -s failglob
mv HAM*.JPG "HAM - REF/JPG"
mv HAM*.RAW "HAM - REF/RAW"
mv HDR*.JPG "HDR - HDRI/JPG"
mv HDR*.RAW "HDR - HDRI/RAW"
mv STL*.JPG "STL - STILLS/JPG"
mv STL*.RAW "STL - STILLS/RAW"
From man bash:
failglob: If set, patterns which fail to match filenames during pathname expansion result in an expansion error.
Please refer a very basic bash script based on your requirements and information you provided:
#!/bin/bash
for files in *
do
if [ -f "$files" ]; then
fileprefix=${files%%[0-9]*.*}
case $fileprefix in
HAM)
if [ ! -d REF ]; then
mkdir REF;
fi
mv "$fileprefix"*.* REF
;;
HDR)
if [ ! -d HDRI ]; then
mkdir HDRI;
fi
mv "$fileprefix"*.* HDRI
;;
STL)
if [ ! -d STILLS ]; then
mkdir STILLS;
fi
mv "$fileprefix"*.* STILLS
;;
esac
fi
done
The first logic for files in * assumes the script itself and all the image files resides in the same directory i.e. PWD.
if [ -f "$files" ]; then
Only run the logic if the matching item in the iteration is a file
fileprefix=${files%%[0-9]*.*}
Extract the letters (first 3) and store it in variable fileprefix
After that a simple switch-case follows.
You can modify the script as per your needs.
Hope this answers your question.
I have several thousand files with name like this:
PIN_PMN_PT_010_02_00331_0004_018edf
and need to rename them all something like this:
PIN_PMN_PT_010_02_00331_0004_018.edf
I have used simple mv scripts like this:
for f in *; do echo mv "$f" "`echo $f | tr 'edf' '.edf'`"; done
For some reason it creates names like this:
PIN_PMN_PT_010_02_00331_0004_018.ed
They are missing the last f. I am running the script using echo to dry run. Any ideas please?
Using MACBook Pro running Mohave 10.14.6 and Bash.
Use shell Parameter Expansion!
Simply under any shell:
For the test, echo to show what's will go:
for file in *edf ;do echo mv "$file" "${file%edf}.edf" ;done
Then, for doing the job:
for file in *edf ;do mv "$file" "${file%edf}.edf" ;done
(This must work same on MacOs, than under Linux.)
... And to prevent renaming of already correctly named files:
for file in *edf ;do test -f "${file##*.edf}" && mv "$file" "${file%edf}.edf" ;done
Syntax ${file##*.edf} will replace any string, terminated by .edf, by an empty string. So test -f "" will fail.
... Still: I don't have any Mac for doing the test, but as this is POSIX Standard, this must work on any shell. (Let my know, please comment!)
More infos?
Have a look at man sh or man bash and search for Parameter Expansion
man -P"less +'/Parameter Expansion'" bash
Using sed, this should work
for f in *; do echo mv "$f" "`echo $f | sed 's/.\{3\}$//`.edf"; done
You are just removing the last 3 characters of a string, and adding your file extension.
Personally, I find the rename command invaluable for this sort of thing:
rename 's/edf$/.edf/' *edf
If you want to do a dry-run, you can do:
rename --dry-run 's/edf$/.edf/' *edf
Sample Output
'PIN_PMN_PT_010_02_00331_0004_018edf' would be renamed to 'PIN_PMN_PT_010_02_00331_0004_018.edf'
The benefits of using rename are:
it can do a dry-run to test before you run for real
it will create all necessary directories with the -p option
it will not clobber (overwrite) files without warning
you have the full power of Perl available to you and can make your renaming as sophisticated as you wish.
As helpfully suggested by F. Hauri in the comments, you may have some files that have already had the dot inserted before the extension in your directory. To protect against insertion of a second dot, you could either be more specific in the files you select for renaming and only rename those ending in a digit followed by edf:
rename 's/edf$/.edf/' *[0-9]edf
Or, as F.Hauri suggested:
rename 's/([^.])edf$/$1.edf/' *edf
Note that you can install on macOS with homebrew:
brew install rename
In a specific Unix directory on a server. I would like to list down files and delete them in a sequence one by one.
Below program is showing errors in for loop. Can anybody help to make it work?
#!/bin/sh
address=/apps/applications/jboss-as/servers/scripts/
files=$(ls $address)
echo Below files will be deleted-
echo $files
k=0
for i in "$files" do {
rm $i
k++
}
done
echo ${k} files are deleted.
I fixed the script, but as others pointed out, you may want to use something other than the ls, but I left it in. There were other syntax errors as well. Here it is:
#!/bin/bash
address=/apps/applications/jboss-as/servers/scripts/
files=("$address"/*)
echo Below files will be deleted-
echo "${files[#]}"
k=0
for i in "${files[#]}" ; do
echo "removing $i ..."
rm "$i"
((k++))
done
echo ${k} files are deleted.
UPDATE:
Since you're still having trouble, you might add declare -a files just above the files=( "$address"/* ). It might fix the problem for you (but, the script runs fine without it here), and it doesn't hurt anything when I run the script here with it. So, either way ...
The first question is, what kind of error do you got?
Try this:
for file in $(ls ${files}/*); do
...
done
Best regards
Robert
I was using a code to rename files and a very interesting problem emerged: It worked in a computer running Mac OSX Lion 10.7.5, but it failed on Mac OSX 10.6.8 Snow Leopard.
The code is:
for i in *; do mv $i `echo $i | sed 's/..//'`; done
and the error I got is:
usage: mv [-f | -i | -n] [-v] source target
mv [-f | -i | -n] [-v] source ... directory
It's very weird because mv is working normally if not in a for loop...
Anyone know what should I do to get it working?
You should use bash string substitution:
for i in *; do mv $i ${i/??/}; done
or
for i in *; do mv $i ${i#??}; done
If you have files with spaces (or certain other shell metachaarcters) it's critical that you put all variable references in double-quotes. Otherwise, if you have e.g. i="File Name.txt", you'll be running a command like mv File Name.txt le Name.txt and it'll think you're specifying 4 filenames not just 2. The other standard mistake (which you're not making) is using for file in $(ls), which gets confused by spaces before the names even make it into the variable.
Also, I'm a bit concerned about short filenames and name conflicts. Even if you think you don't have any short filenames, if you have the dotglob shell option set you'll wind up trying to rename the pseudo-files "." and "..", which won't go well at all. Also, if you have e.g. files named "abcdefg" and "cdefg", the script will rename the first over the second (silently erasing the second), then rename that to "efg".
So, here's my proposed rewrite (also using #TrueY's suggestion for shortening the filename):
for i in *; do
if [ ${#i} -le 2 ]; then
echo "$i: not renamed (too short)" >&2
elif [ -e "${i:2}" ]; then
echo "$i: not renamed (${i:2} already exists)" >&2
else
mv "$i" "${i:2}"
fi
done
Say I have three files (template_*.txt):
template_x.txt
template_y.txt
template_z.txt
I want to copy them to three new files (foo_*.txt).
foo_x.txt
foo_y.txt
foo_z.txt
Is there some simple way to do that with one command, e.g.
cp --enableAwesomeness template_*.txt foo_*.txt
for f in template_*.txt; do cp $f foo_${f#template_}; done
[01:22 PM] matt#Lunchbox:~/tmp/ba$
ls
template_x.txt template_y.txt template_z.txt
[01:22 PM] matt#Lunchbox:~/tmp/ba$
for i in template_*.txt ; do mv $i foo${i:8}; done
[01:22 PM] matt#Lunchbox:~/tmp/ba$
ls
foo_x.txt foo_y.txt foo_z.txt
My preferred way:
for f in template_*.txt
do
cp $f ${f/template/foo}
done
The "I-don't-remember-the-substitution-syntax" way:
for i in x y z
do
cp template_$i foo_$
done
This should work:
for file in template_*.txt ; do cp $file `echo $file | sed 's/template_\(.*\)/foo_\1/'` ; done
for i in template_*.txt; do cp -v "$i" "`echo $i | sed 's%^template_%foo_%'`"; done
Probably breaks if your filenames have funky characters in them. Remove the '-v' when (if) you get confidence that it works reliably.
The command mmv (available in Debian or Fink or easy to compile yourself) was created precisely for this task. With the plain Bash solution, I always have to look up the documentation about variable expansion. But mmv is much simpler to use, quite close to "awesomeness"! ;-)
Your example would be:
mcp "template_*.txt" "foo_#1.txt"
mmv can handle more complex patterns as well and it has some sanity checks, for example, it will make sure none of the files in the destination set appear in the source set (so you can't accidentally overwrite files).
I don't know of anything in bash or on cp, but there are simple ways to do this sort of thing using (for example) a perl script:
($op = shift) || die "Usage: rename perlexpr [filenames]\n";
for (#ARGV) {
$was = $_;
eval $op;
die $# if $#;
rename($was,$_) unless $was eq $_;
}
Then:
rename s/template/foo/ *.txt
Yet another way to do it:
$ ls template_*.txt | sed -e 's/^template\(.*\)$/cp template\1 foo\1/' | ksh -sx
I've always been impressed with the ImageMagick convert program that does what you expect with image formats:
$ convert rose.jpg rose.png
It has a sister program that allows batch conversions:
$ mogrify -format png *.jpg
Obviously these are limited to image conversions, but they have interesting command line interfaces.