Symlink (ln) folders doesn't work as expected - bash

I am creating a script that basically look for all the things that ends with .symlink and try to create a symlink to the $HOME directory removing the symlink part and adding dot in front of the name.
Below is the line showing how the destination is created.
dst="$HOME/.$(basename "${src%.*}")"
There are the two functions created for that
create_symlink () {
local src=$1 dst=$2
# Create the symlink
ln -s -i "$src" "$dst"
message "Linked $1 -------> $2" # This just use echo
}
run_symlinks() {
message "Creating Symlinks"
local root=$1
# Find the files/folder and try to do the symlink
for src in $(find -H $root -name '*.symlink')
do
dst="$HOME/.$(basename "${src%.*}")"
create_symlink "$src" "$dst"
done
}
The problem
I have a folder called atom.symlink that folder basically have some configuration for my environment with the files of Atom text editor inside. When I run run_symlinks function that folder is being synced but in the wrong place.
The output of message "Linked $1 -------> $2" is:
Linked $HOME/.dotfiles/src/symlinks/atom.symlink -------> $HOME/.atom
But when I look at the folder the symlink is actually to $HOME/.atom/atom.symlink instead of just to the .atom folder.
Note: the $HOME/.atom folder is not empty and I need to figure out how to make this script without worry about having an empty folder.
I tried to find the answer on Google but I could't even know how to ask this specific question about it.

From man ln:
ln [OPTION]... [-T] TARGET LINK_NAME (1st form)
ln [OPTION]... TARGET (2nd form)
ln [OPTION]... TARGET... DIRECTORY (3rd form)
ln [OPTION]... -t DIRECTORY TARGET... (4th form)
In the 1st form, create a link to TARGET with the name LINK_NAME. In the
2nd form, create a link to TARGET in the current directory. In the 3rd
and 4th forms, create links to each TARGET in DIRECTORY. Create hard
links by default, symbolic links with --symbolic. By default, each destination
(name of new link) should not already exist. When creating hard
links, each TARGET must exist. Symbolic links can hold arbitrary text; if
later resolved, a relative link is interpreted in relation to its parent
directory.
After first iteration of your script, you have
$HOME/.atom/ -> $HOME/.dotfiles/src/symlinks/atom.symlink # This is the first form in above man page snippet.
In the second iteration, you fall in the 3rd form, because the target already existed & after symlink dereferencing, it's a directory.
So, the command run is same:
ln -s -i $HOME/.dotfiles/src/symlinks/atom.symlink $HOME/atom
Only difference is that in the second iteration, the target is an existing directory (after dereferencing).
So you should first delete the target (rm -f "$dst") & then create a symlink. Luckily, ln can do it by itself:
Change your code to:
ln -sfn "$src" "$dst" # Note that with -f & -n in place, usage of -i is moot.

To complement anishsane's helpful answer, which explains the problem with your approach well:
Your desire to retain possibly preexisting (non-symlink) target folders in $HOME and add to their content requires a fundamentally different approach:
The only way to solve this is to avoid symlinking the *.symlink directories themselves; instead, the files in these directories must be individually symlinked to the target folder, which is either a preexisting folder or one that must be created as a regular folder on demand.
That is the only way to guarantee that the existing content of the target folder is not (invariably) lost:
while IFS= read -r f; do
# Strip $root from the file path, then remove suffix '.symlink' and add prefix '.'
# to the path commponent(s) to get the link path.
link="$HOME/$(sed 's#\([^/]\{1,\}\)\.symlink\(/\{0,1\}\)#.\1\2#g' <<<"${f#$root/}")"
# Make sure that the parent path exists, creating it on demand (as *regular* folder(s)).
mkdir -p -- "$(dirname -- "$link")"
# Now we can create the symlink.
echo "symlinking [$link] <- [$f]"
# Note the need to redirect from `</dev/tty` so as not
# to suppress the interactive prompt (`-i`).
ln -s -i "$f" "$link" </dev/tty
done < <(find -H "$root" -type f \( -path '*.symlink' -or -path '*.symlink/*' \))
The approach is a follows:
The find command finds only files, namely those themselves named *.symlink, and those inside directories named *.symlink (whatever suffix the files themselves have).
For each file, the target symlink path is determined by removing the $root path prefix, and then removing suffix .symlink and adding prefix . to matching path components.
The existence of each target symlink path's parent path is ensured with mkdir -p: any existing path components are retained as-is, and any non-existent ones are created as regular folders.
Once the existence of the target folder for the symlink is ensured / established, the ln command can be invoked.
Note that -i - to present an interactive prompt asking for replacement in case the link's path already exists - requires stdin to be a terminal in order to kick in; thus, given that stdin is redirected to the process substitution providing the output from find, </dev/tty is needed to show the prompt.

Related

How to remove a file named '.'?

Ok so i did something very stupid (copying a file and renaming it '.') since I thought it would just copy it as .uniprot_sprot.fasta.gz.icloud.
cp /path/.uniprot_sprot.fasta.gz.icloud .
and now I don't know how to remove it from current directory as it would be removing '.' itself.
What can I do?
This doesn't work. It says: No such file or directory
rm .uniprot_sprot.fasta.gz.icloud
On the other hand:
ls -a
gives this:
.
..
uniprot_sprot.fasta.gz.icloud
You have not copied a file and renamed it . (at any rate if you're running a sane *nix). Instead you have copied the file to the current directory with the name of the original file. (If you pass a directory to cp as the destination, files will be placed in that directory. . is the current directory, so this is all that has happened.) If you want to remove it you can just rm uniprot_sprot.fasta.gx.iscloud or explicitly rm ./uniprot_sprot.fasta.gx.iscloud. What you have tried to do is to remove a file whose name starts with ., which is a different thing.
Edit: I was unaware when I wrote this, but this is in fact simply down to . existing as a real, regular hardlink. At syscall level you can create a file whose name contains anything except / and \x00 (yep, including \n), assuming your filesystem allows it. However, the links . and .. are already present and thus unavailable as a file name. #thatotherguy links to the kernel source for the rmdir syscall, showing that in modern Linux at least it is the kernel itself which ultimately prevents you from deleting . and ...
Note that in bash, . at the beginning of a line by itself means source.
See this question on unix.se and its linked dupe for more information on the filename problem.

Script to change photos folder hierarchy

I'm looking for a script to change an existing folder structure quickly on my synology nas using ssh from YYYY/MM/DD/ to YYYY-MM-DD/ so nested to flat but struggling to find one or any examples probably due to me not searching for the correct terminology.
I did start to use exiftool and go through moving each item but the collection is taking ages.
for example an image01 that currently resides in say 2020/01/01/images01.jpg needs to move to 2020-01-01/images.jpg
image and video files only currently live in the day folders.
For example, you can do this with this command:
find . -mindepth 3 -type d |awk -F"/" 'system("mv " $2"/"$3"/"$4" "$2"-"$3"-"$4" && rmdir "$2"/"$3" && rmdir "$2" 2>/dev/null")'
Example input folder structure:
./2020/06/12/img1.jpg
./2020/06/12/file2.mpg
./2020/05/10/img2.jpg
./2020/05/10/img1.jpg
Result:
./2020-05-10/img1.jpg
./2020-05-10/img2.jpg
./2020-06-12/img1.jpg
./2020-06-12/file2.mpg
Explanation:
find . searches in the current directory
-mindepth 3 only searches at level 3 nesting
-type d only search for directories
| creates a pipe, directing the find result to AWK
-F"/" sets the AWK field separator to /
system command is executed for each line which transfers files to the new directory and deletes unnecessary old directories
&& do next command only when previous ones were successful
2>/dev/null directs stderr to the void so that you won't see any errors trying to delete a non-empty "year" directory
You cannot use rm -rf because the year directory may still contain other directories, hence using rmdir twice.
You can resolve the stderr redirection more elegantly by testing whether the directory of the year is empty before deleting it,
I do not put this test so as not to obscure the idea of ​​action unnecessarily.

Split a folder which has 100s of subfolders with each having few files into one more level of subfolder using shell

I have a following data dir:
root/A/1
root/A/2
root/B/1
root/B/2
root/B/3
root/C/1
root/C/2
And I want to convert it into following file structure:
root2/I/A/1
root2/I/A/2
root2/I/B/1
root2/I/B/2
root2/I/B/3
root2/II/C/1
root2/II/C/2
Purpose of doing it is I want to run some script which takes home folder (root here) and runs on it. And I want to run it in parallel on many folders(I, II) to speed up the process.
Simple assumption about file and folder name is that all are alphanumeric, even no period or underscore.
Edit: I tried following:
for i in `seq 1 30`; do mkdir -p "root2/folder$i"; find root -type f | head -n 4000 | xargs -i cp "{}" "root2/folder$i"; done
Problem is that it creates something like following, which is not what i wanted.
root2/I/1
root2/I/2
root2/I/1
root2/I/2
root2/I/3
root2/II/1
root2/II/2
You may wish to use a lesser known command called dirsplit, the usual application of which is to split a directory into multiple directories for burning purposes.
Use it like below :
dirsplit -m -s 300M /root/ -p /backup/folder1
Options implies below stuff :
-m|--move Move files to target dirs
-e 2 special exploration mode, 2 means files in directory are put together
-p prefix to be attached to each directory created, in you case I, II etc
-s Maximum size allowed for each new folder created.
For more information see :
dirsplit -H

looping files with bash

I'm not very good in shell scripting and would like to ask you some question about looping of files big dataset: in my example I have alot of files with the common .pdb extension in the work dir. I need to loop all of them and i) to print name (w.o pdb extension) of each looped file and make some operation after this. E.g I need to make new dir for EACH file outside of the workdir with the name of each file and copy this file to that dir. Below you can see example of my code which are not worked- it's didn't show me the name of the file and didn't create folder for each of them. Please correct it and show me where I was wrong
#!/bin/bash
# set the work dir
receptors=./Receptors
for pdb in $receptors
do
filename=$(basename "$pdb")
echo "Processing of $filename file"
cd ..
mkdir ./docking_$filename
done
Many thanks for help,
Gleb
If all your files are contained within the .Repectors folder, you can loop each of them like so:
#!/bin/bash
for pdb in ./Receptors/*.pdb ; do
filename=$(basename "$pdb")
filenamenoextention=${filename/.pdb/}
mkdir "../docking_${filenamenoextention}"
done
Btw:
filenamenoextention=${filename/.pdb/}
Does a search replace in the variable $pdb. The syntax is ${myvariable/FOO/BAR}, and replaces all "FOO" substrings in $myvariable with "BAR". In your case it replaces ".pdb" with nothing, effectively removing it.
Alternatively, and safer (in case $filename contains multiple ".pdb"-substrings) is to remove the last four characters, like so: filenamenoextention=${filename:0:-4}
The syntax here is ${myvariable:s:e} where s and e correspond to numbers for the start and end index (not inclusive). It also let's you use negative numbers, which are offsets from the end. In other words: ${filename:0:-4} says: extract the substring from $filename starting from index 0, until you reach fourth-to-the-last character.
A few problems you have had with your script:
for pdb in ./Receptors loops only "./Receptors", and not each of the files within the folder.
When you change to parent directory (cd ..), you do so for the current shell session. This means that you keep going to the parent directory each time. Instead, you can specify the parent directory in the mkdir call. E.g mkdir ../thedir
You're looping over a one-item list, I think what you wanted to get is the list of the content of ./Receptors:
...
for pdb in $receptors/*
...
to list only file with .pdb extension use $receptors/*.pdb
So instead of just giving the path in for loop, give this:
for pdb in $receptors/*.pdb
To remove the extension :
set the variable ext to the extension you want to remove and using shell expansion operator "%" remove the extension from your filename eg:
ext=.pdb
filename=${filename%${ext}}
You can create the new directory without changing your current directory:
So to create a directory outside your current directory use the following command
mkdir ../docking_$filename
And to copy the file in the new directory use cp command
After correction
Your script should look like:
receptors=./Receptors
ext=.pdb
for pdb in $receptors/*.pdb
do
filename=$(basename "$pdb")
filename=${filename%${ext}}
echo "Processing of $filename file"
mkdir ../docking_$filename
cp $pdb ../docking_$filename
done

Adding a status (file integrity)check to a cbr cbz converting bash script

First post, so Hi! Let me start by saying I'm a total noob regarding programming. I understand very basic stuff, but when it comes to checking exit codes or what the adequate term is, I'm at a loss. Apparently my searchfoo is really weak in this area, I guess it's a question of terminology.
Thanks in advance for taking your time to reading this/answering my question!
Description: I found a script that converts/repack .cbr files to .cbz files. These files are basically your average rar and zip files, however renamed to another extension as they are used for (comic)book applications such as comicrack, qcomicbook and what not. Surprisingly enough there no cbr -> cbz converters out there. The advantages of .cbz is besides escaping the proprietary rar file format, that one can store the metadata from Comic Vine with e. g comictagger.
Issue: Sometimes the repackaging of the files doesn't end well and would hopefully be alleviated by a integrity check & another go. I modified said script slightly to use p7zip as it can both pack/unpack 7z, zip-files and some others, i. e great for options. p7zip can test the archive by:
7z t comicfile.cbz tmpworkingdir
I guess it's a matter of using if & else here(?) to check the integrity and then give it another go, if there are any error.
Question/tl;dr: What would be the "best"/adequate approach to add a integrity file check to the script below?
#!/bin/bash
#Source: http://comicrack.cyolito.com/forum/13-scripts/30013-cbr3cbz-rar-to-zip-conversion-for-linux
echo "Converting CBRs to CBZs"
# Set the "field separator" to something other than spaces/newlines" so that spaces
# in the file names don't mess things up. I'm using the pipe symbol ("|") as it is very
# unlikely to appear in a file name.
IFS="|"
# Set working directory where to create the temp dir. The user you are using must have permission
# to write into this directory.
# For performance reasons I'm using ram disk (/dev/shm/) in Ubuntu server.
WORKDIR="/dev/shm/"
# Set name for the temp dir. This directory will be created under WORDDIR
TEMPDIR="cbr2cbz"
# The script should be invoked as "cbr2cbz {directory}", where "{directory}" is the
# top-level directory to be searched. Just to be paranoid, if no directory is specified,
# then default to the current working directory ("."). Let's put the name of the
# directory into a shell variable called SOURCEDIR.
# Note: "$1" = "The first command line argument"
if test -z "$1"; then
SOURCEDIR=`pwd`
else
SOURCEDIR="$1"
fi
echo "Working from directory $SOURCEDIR"
# We need an empty directory to work in, so we'll create a temp directory here
cd "$WORKDIR"
mkdir "$TEMPDIR"
# and step into it
cd "$TEMPDIR"
# Now, execute a loop, based on a "find" command in the specified directory. The
# "-printf "$p|" will cause the file names to be separated by the pipe symbol, rather than
# the default newline. Note the backtics ("`") (the key above the tab key on US
# keyboards).
for CBRFILE in `find "$SOURCEDIR" -name "*.cbr" -printf "%p|while read line; do
# Now for the actual work. First, extract the base file name (without the extension)
# using the "basename" command. Warning: more backtics.
BASENAME=`basename $CBRFILE ".cbr"`
# And the directory path for that file, so we know where to put the finished ".cbz"
# file.
DIRNAME=`dirname $CBRFILE`
# Now, build the "new" file name,
NEWNAME="$BASENAME.cbz"
# We use RAR file's name to create folder for unpacked files
echo "Processing $CBRFILE"
mkdir "$BASENAME"
# and unpack the rar file into it
7z x "$CBRFILE" -O"$BASENAME"
cd "$BASENAME"
# Lets ensure the permissions allow us to pack everything
sudo chmod 777 -R ./*
# Put all the extracted files into new ".cbz" file
7z a -tzip -mx=9 "$NEWNAME" *
# And move it to the directory where we found the original ".cbr" file
mv "$NEWNAME" $DIRNAME/"$NEWNAME"
# Finally, "cd" back to the original working directory, and delete the temp directory
# created earlier.
cd ..
rm -r "$BASENAME"
# Delete the RAR file also
rm "$CBRFILE"
done
# At the end we cleanup by removing the temp folder from ram disk
cd ..
echo "Conversion Done"
rm -r "$TEMPDIR"
Oh the humanity, not posting more than two links before 10 reputation and I linked the crap out of OP.. [edit]ah.. mh-mmm.. there we go..
[edit 2] I removed unrar as an dependency and use p7zip instead, as it can extract rar-files.
You will need two checks:
7z t will test the integrity of the archive
You should also test the integrity of all the image files in the archive. You can use at tools like ImageMagick for this.
A simple test would be identify file but that might read only the header. I'd use convert file -resize 5x5 png:- > /dev/null
This scales the image down to 5x5 pixels, converts it to PNG and then pipes the result to /dev/null (discarding it). For the scaling, the whole image has to be read. If this command fails with an error, something is wrong with the image file.

Resources