So I've got a movie collection that's dumped into a single folder (I know, bad practice in retrospect.) I want to organize things a bit so I can use Radarr to grab all the appropriate metadata, but I need all the individual files in their own folders. I created the script below to try and automate the process a bit, but I get the following error.
Script
#! /bin/bash
for f in /the/path/to/files/* ;
do
[[ -d $f ]] && continue
mkdir "${f%.*}"
mv "$f" "${f%.*}"
done
EDIT
So I've now run the script through Shellcheck.net per the suggestion of Benjamin W. It doesn't throw any errors according to the site, though I still get the same errors when I try running the command.
EDIT 2*
No errors now, but the script does nothing when executed.
Assignments are evaluated only once, and not whenever the variable being assigned to is used, which I think is what your script assumes.
You could use a loop like this:
for f in /path/to/all/the/movie/files/*; do
mkdir "${f%.*}"
mv "$f" "${f%.*}"
done
This uses parameter expansion instead of cut to get rid of the file extension.
Related
I am building a little script to update application files on a raspberry pi.
It will do the following:
Download a zip file of the application files
Unzip them
Copy each one to the right place and make it executable etc as needed.
The problem i'm having is that one of the files is updatescript.sh.
I've read that it is dangerous to update / change a bash script while it is executing. See Edit shell script while it's running
Is there a good way to achieve what I'm trying to do?
What you've read is badly overblown.
It's completely safe to overwrite a shell script in-place by mving a different file over it. When you do this, the old file handle is still valid, referring to the original unmodified file contents. What you can't safely do is edit the existing file in-place.
So, the below is fine (and is what all your OS-vendor update tools like RPM do in effect):
#!/usr/bin/env bash
tempfile=$(mktemp "$BASH_SOURCE".XXXXXX)
if curl https://example.com/whatever >"$tempfile" &&
curl https://example.com/whatever.sig >"$tempfile.sig" &&
gpgv "$tempfile.sig" "$tempfile"; then
chown --reference="$BASH_SOURCE" -- "$tempfile"
chmod --reference="$BASH_SOURCE" -- "$tempfile"
sync # force your filesystem to fully flush file contents to disk
mv -- "$tempfile" "$BASH_SOURCE" && rm -f -- "$tempfile.sig"
else
rm -f -- "$tempfile" "$tempfile.sig"
exit 1
fi
...whereas this is risky:
curl https://example.com/whatever >/usr/local/bin/whatever
So do the first, thing, not the second one: When downloading a new version of your script, write that to a different file, and only rename it over the original when the download succeeded. That's what you want to do anyhow to ensure atomicity.
(There are also some demonstrations of code-signing validation practices above because, well, you need them when building an updater. You wouldn't be trying to distribute code via an automated download without verifying a signature, right? Because that's how one simple breakin to your web server results in every single one of your customers being 0wned. The above expects the public side of your code-signing keys to be in ~/.gnupg/trustedkeys.gpg, but you can put trustedkeys.gpg in any directory and point to it with the environment variable GNUPGHOME).
Even if you don't write your update code safely, the risk is still trivial to mitigate. If you move the body of your script into a function, such that it has to be completely read before any part of it can be executed, then there's no part of the file that isn't already read at the time when execution begins.
#!/usr/bin/env bash
main() {
echo "Logic all goes here"
}; { main "$#"; exit; }
Because { main "$#"; exit; } is part of a compound command, the parser reads the exit before it starts executing the main, so it's guaranteed that no further source-file content will be read after main exits, even if some future bash release didn't handle input line-by-line in the first place.
Basically do something along:
shouldbe="/tmp/$(basename "$0")"
if [ "$0" != "$shouldbe" ]; then
cp "$0" "$shouldbe"
exec env REALPATH="$0" "$shouldbe" "$#"
fi
Check if you are running from a temporary directory
If you are not, copy yourself and rerun from the temporary directory
You can even pass some variables/state along, by using environmental variables or arguments. Then you can update yourself by using simple cp, as the old path isn't sourced (or even opened) anymore.
cp "new_script_version.sh" "$REALPATH"
The script simply looks like this:
#!/bin/bash
# we need to be run from /tmp directory
shouldbe="/tmp/$(basename "$0")"
if [ "$0" != "$shouldbe" ]; then
cp "$0" "$shouldbe"
exec env REALPATH="$0" "$shouldbe" "$#"
fi
echo "Updatting...."
echo "downloading zip files"
echo "unziping zip files..."
echo "Copying each zip files etc."
cp directory"new_updatescript.sh "$REALPATH"
echo "Update succedded"
Live/test version available at tutorialspoint.
One would also implement some flock locking to the scripts just in case.
I have a very simple renaming script I'm running in OSX Terminal. It looks like this:
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1140122_alternate1.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1140122_alternate1A.tif
I usually have several hundred lines of rename code like this one for all the files I have to rename.
However I think the network security at work is messing with the code because it will randomly jack up the file names. I think it's interrupting the code, the code is so simple I can't think of another reason why it wouldn't work.
I want to try adding a 1sec delay between each line, but how? I've read that something like sleep 1s might work but do I have to add that between every single line? That's going to be a headache if that's the case. If it is, is there another way?
UPDATE: I have a delay working but still getting the same problems as before. This is what Terminal returns:
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate1.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate1A.tif
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate2.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate2A.tif
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate3.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Remv -nvest/1247136_alternate3A.tif
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate4.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Remv -nTest/1247136_alternate4A.tif
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_lifestyle.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Renmv -nv /Volume36_lifestyleA.tif
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_standard.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_standardA.tifç^C^C^C^C^C
It's throwing up all kinds of junk in the rename part. It's messing with the file names and the directory names and I can't figure out why.
If you are planing to perform all those mv commands from terminal you can make a bash alias:
alias mvd='sleep 2s && mv'
In terms of a script, since scripts do not understand bash alias (at least easily) you can built a similar function in the beginning of your script:
function mvd { sleep 2s && mv "$#"; }
The only thing you need to do is to use the new mvd command instead of mv.
Tip: In case of alias you can also name your alias mv (same name as the command).
If you already have a script that has hardcoded paths (eg, the script looks like:
mv -nv /path1 /path2
mv -nv /path3 /path4
...
Then probably the simplest thing to do would be to define a function at the top of the script by adding:
mv() { command mv "$#"; sleep 1; }
the following script just reads in your file of commands and inserts a sleep after each command
while read curr_line; do
echo curr_line $curr_line
return_msg=$( $curr_line ) # execute cmd
# may want to do error checking on value of error variable $? and return_msg
sleep 1
done < ./input_file_of_original_cmds.txt # read in that file
My script downloads files from a web server in an infinite loop. My script calls wget to get the newest files (ones I haven't gotten before), then each new file needs to be processed. The problem is that after running wget, the files have been properly downloaded (based on an ls in a separate window), but sometimes my script (specifically, the line beginning for curFile in) sees them and sometimes it doesn't, which makes me think it is sometimes looking at an outdated cache.
while [ 5 -lt 10 ]; do
timestamp=(date +%s)
wget -mbr -l0 --no-use-server-timestamps --user=username --password=password ftp://ftp.mysite.com/public_ftp/incoming/*.txt
for curFile in ftp.mysite.com/public_ftp/incoming/*.txt; do
curFileMtime=$(stat -c %W "$curFile")
if((curFileMtime > timestamp)); then
echo "$curFile"
cp "$curFile" CommLink/MDLFile
cd CommLink
SendMDLGetTab
cd ..
fi
done
sleep 120
done
The first few times through the loop this seems to work fine, then it becomes sporadic afterwards (sometimes it sees the new files and sometimes it doesn't). I've done a lot of googling, and found that bash does cache pathnames for use in running executables (so sometimes it tries to execute things that aren't there, if the executable has been recently removed) but I haven't found anything on caching non-executable filenames, which would result in it not seeing things that are there. Any ideas? If it is a caching problem, how can I force it to not look at the cache?
As the most immediate issue -- the -b argument to wget tells it to run in the background. Thus, with this flag set, the first subsequent command takes place while wget is still running.
Beyond that: Results of glob expressions -- such as ftp.mysite.com/public_ftp/incoming/*.txt -- are not cached by the shell. However, this glob is only evaluated once per loop: If a new text file isn't present at the start of that loop, it won't be picked up until the next iteration.
However, the mechanism the code in the question uses for excluding files already present before wget was run is prone to race conditions. I would suggest the following instead:
while IFS= read -r -d '' filename; do
[[ "$filename" -nt CommLink/MDLFile ]] || continue # retest in case MDLFile has changed
cp -- "$filename" CommLink/MDLFile && {
touch -r "$filename" CommLink/MDLFile # copy mtime to destination
(cd CommLink && exec SendMDLGetTab) # scope cd to subshell
}
done < <(find ftp.mysite.com/public_ftp/incoming/ \
-name '*.txt' \
-newer CommLink/MDLFile \
-print0)
Some of the finer points:
The code above compares timestamps to the current copy of MDLFile, rather than to the beginning of the current iteration of the loop. This is more robust in terms of ensuring that updates are processed if a prior invocation of this script was interrupted after the wget but before the cp.
Using touch -r ensures that the new copy of MDLFile retains the mtime of the original. (One might replace the cp with ln -f to hardlink the inode to get this same effect without any race conditions and while only storing MDLFile once on-disk, if the side effects are acceptable).
The code above only performs operations intended to be run inside a subdirectory if the cd into that subdirectory succeeded, and scopes that cd by performing operations intended to be performed in a separate directory in a subshell. (The cost of this subshell is offset by using exec when running the external command its ultimate intent is to trigger).
Using a NUL-delimited stream, as in find -print0 | while IFS= read -r -d '' filename, ensures that all possible names -- including names with literal newlines -- can be correctly handled.
Whether timestamps are stored at or beyond integer-level resolution varies by filesystem; however, bash only supports integer math. The above -- wherein no numeric comparisons are performed by the shell -- is thus somewhat more robust.
Here is the condition:
I have a file with all packages installed.
I have a folder with all kinds of other packages, but they include all of the ones in the list, plus more.
I need a bash script that will read the file and check a folder for packages that don't exist in the list then remove them, they are not needed, but keep the packages that are on the list in that folder.
Or perhaps the bash should read folder then if packages in the folder aren't on the list them rm -f that or those packages.
I am familiar with writing if then conditional statements, I just don't know how to do if making the items in the list a variable or variables (in a loop).
thanks!
I would move the packages on the list to a new folder, delete the original folder, and move the temporary folder back:
DIR=directory-name
mkdir "$DIR-tmp"
while read pkgname; do
if [[ -f "$DIR/$pkgname" ]]; then
mv "$DIR/$pkgname" "$DIR-tmp"
fi
done < package-list.txt
# Confirm $DIR-tmp has the files you want first!
rm -rf "$DIR"
mv "$DIR-tmp" "$DIR"
I think you want something like this:
for file in $(ls folder) ; do
grep -E "$file" install-list-file >/dev/null || \
echo $file
done > rm-list
vi rm-list # view file to ensure correct
rm $(<rm_list)
There are ways to make this faster (using parameter substitution to avoid fork/exec's), but I recommend avoiding fancy shell stuff [${file##*/}] until you've got the basics down. Also, this script basically translates the description into a script and is not intended to be much more than a guide on how to approach the problem.
I want to backup my ubuntu filesystem, and I wrote this little script. It is very basic, but being my first try I am afraid to do mistakes. And since it will take few hours to complete to see results, I think it is better to ask you as experienced programmers if I did something wrong.
I'm particularly interested in > will that record output of mv or will it output also results of tar?
Also variables inside tar command is it correct way?
#!/bin/bash
mybackupname="backup-fullsys-$(date +%Y-%m-%d).tar.gz"
{ time tar -cfpzv $mybackupname --exclude=/$mybackupname --exclude=/proc --exclude=/lost+found --exclude=/sys --exclude=/mnt --exclude=/media --exclude=/dev / && ls -gh $mybackupname && mv -v $mybackupname backups/filesystem/ ; } > backup-system.log
exit
Anything I should know before I run this?
Sandro, you might want to consider spacing things out in your script and producing individual errors. Makes things much easier to read.
#!/bin/bash
mybackupname="backup-fullsys-$(date +%Y-%m-%d).tar.gz"
# Record start time by epoch second
start=$(date '+%s')
# List of excludes in a bash array, for easier reading.
excludes=(--exclude=/$mybackupname)
excludes+=(--exclude=/proc)
excludes+=(--exclude=/lost+found)
excludes+=(--exclude=/sys)
excludes+=(--exclude=/mnt)
excludes+=(--exclude=/media)
excludes+=(--exclude=/dev)
if ! tar -czf "$mybackupname" "${excludes[#]}" /; then
status="tar failed"
elif ! mv "$mybackupname" backups/filesystem/ ; then
status="mv failed"
else
status="success: size=$(stat -c%s backups/filesystem/$mybackupname) duration=$((`date '+%s'` - $start))"
fi
# Log to system log; handle this using syslog(8).
logger -t backup "$status"
If you wanted to keep debug information (like the stderr of tar or mv), that could be handled with redirection to a tmpfile or debug file. But if the command is being run via cron and has output, cron should send it to you via email. A silent cron job is a successful cron job.
The series of ifs causes each program to be run as long as the previous one was successful. It's like chaining your commands with &&, but lets you run other code in case of failure.
Note that I've changed the order of options for tar, because the thing that comes after -f is the file you're saving things to. Also, the -p option is only useful when extracting files from a tar. Permissions are always saved when you create (-c) a tar.
Others might wish to note that this usage of the stat command works in GNU/Linux, but not other unices like FreeBSD or Mac OSX. In BSD, you'd use stat -f%z $mybackupname.
The file redirection as you have it will only record the output of mv.
You can do
{ tar ... && mv ... ; } > logfile 2>&1
to capture the output of both, plus any errors that may occur.
It's a good idea to always be in the habit of quoting variables when they are expanded.
There's no need for the exit.