I have an array of empty subdirectories and when I loop through all of the subdirectories I want to do something different depending on if it is in the array of empty subdirectories or not.
My issue is that I don't think my script is picking up on if a value is in the empty directories array. When I loop through the arrays and echo their values I can see that some values are in both arrays so I don't know why they aren't being picked up.
Code:
readarray empty_dirs < <(find /local/documents/guests/* -maxdepth 0 -empty)
for f in "${empty_dirs[#]}"
do
echo $f
echo "------------------------"
done
guest_path=/local/documents/guests/guest*
guest_arr=( $guest_path )
for dir in "${guest_arr[#]}"
do
echo "$dir"
if [[ " ${empty_dirs[#]} " =~ " ${dir} " ]];then
echo "--- $dir found ---"
fi
done
Output:
/local/documents/guests/guest10
------------------------
/local/documents/guests/guest12
------------------------
/local/documents/guests/guest15
------------------------
/local/documents/guests/guest18
------------------------
/local/documents/guests/guest20
------------------------
/local/documents/guests/guest1
/local/documents/guests/guest10
/local/documents/guests/guest11
/local/documents/guests/guest12
/local/documents/guests/guest13
/local/documents/guests/guest14
/local/documents/guests/guest15
/local/documents/guests/guest16
/local/documents/guests/guest17
/local/documents/guests/guest18
/local/documents/guests/guest19
/local/documents/guests/guest2
/local/documents/guests/guest20
/local/documents/guests/guest21
/local/documents/guests/guest22
/local/documents/guests/guest23
/local/documents/guests/guest24
/local/documents/guests/guest25
/local/documents/guests/guest3
/local/documents/guests/guest4
/local/documents/guests/guest5
/local/documents/guests/guest6
/local/documents/guests/guest7
/local/documents/guests/guest8
/local/documents/guests/guest9
The problem is ${empty_dirs[#]} contains the names including trailing newlines.
You can stop storing them using the -t option of readarray, or remove them before running the final loop:
empty_dirs=${empty_dirs[#]%$'\n'}
Related
First of all I'm trying to obtain a certain property from a KML file. For now, I tried
ogrinfo C:/test.kml -so -al | findstr "Extent"
which was recommended to me and outputs
Extent: (-100.054053, 33.702234) - (-94.647180, 37.125712)
I would require this in the form
-100.054053,-94.647180,33.702234,37.125712 for which I thought to use regex.
I tried the following just to see what it outputted:
ogrinfo C:/test.kml -so -al | findstr "Extent" | findstr /r /c:"-*[0-9]*\.[0-9]*"
but this still outputs
Extent: (-100.054053, 33.702234) - (-94.647180, 37.125712)
I read somewhere that Windows' FINDSTR only outputs the line where it matched and not the regex matches themselves. Is there some other way of doing it?
If I get that working I would save the matches in different variables somehow in a shell script. I'm no expert in shell scripting but I've been looking around and was thinking of doing something like this
#!/bin/bash
for /f "tokens=*" %%a in ('ogrinfo C:/test.kml -so -al ^| findstr "Extent" ^| findstr /r /c:"-*[0-9]*\.[0-9]*"') do (
echo %%a
#do something
)
done >output
but running this causes the shell to immediately disappears and can't even see the error.
Assumptions
You have a kml file with raw data.
You can extract a single line which starts with "Extent: " to get the values you want
Single line => there is only 1 line with that format in the kml file
The format of that line is:
Extent: (NUMBER1, NUMBER2) - (NUMBER3, NUMBER4)
A number can have the following characters: 0 1 2 3 4 5 6 7 8 9 . -
The output you want is:
NUMBER1,NUMBER3,NUMBER2,NUMBER4
Using Linux tools only, you can do this:
#!/bin/bash
#
datafile="data.kml"
# Ensure the data file exists
if [[ ! -f "$datafile" ]]
then
echo "ERROR: the data file does not exist."
exit 1
fi
# Extract the "Extent:" line
dataline=$(grep "Extent: " "$datafile")
# Make sure the line is of a valid format, and assign the number variables
if [[ $dataline =~ "Extent: ("([0-9.-]+)", "([0-9.-]+)") - ("([0-9.-]+)", "([0-9.-]+)")" ]] && number1="${BASH_REMATCH[1]}" && number2="${BASH_REMATCH[2]}" && number3="${BASH_REMATCH[3]}" && number4="${BASH_REMATCH[4]}"
then
echo "-----DEBUG-----"
echo "line==$dataline"
echo "1==$number1"
echo "2==$number2"
echo "3==$number3"
echo "4==$number4"
echo "-- END DEBUG --"
echo ""
echo "$number1,$number3,$number2,$number4"
else
echo "ERROR: there is no \"Extent: \" line in the data file ($datafile)"
fi
Details:
Everything is done in the if line.
=~ matches the left side with the pattern on the right side.
In the regular expression, you can define sections you want to reuse with ( ).
Ex: abcd(1)efgh(2)ijkl. The sections you can reuse are 1 and 2.
So in the if, each number is surrounded by parentheses.
When the =~ is processed, the BASH_REMATCH array is defined with each section.
The "DEBUG" echo statements can be removed or commented out.
If you have more than one "Extent: ..." in the KML file, you can loop on the lines and process each one at a time.
This question already has answers here:
Extract filename and extension in Bash
(38 answers)
Closed 1 year ago.
I'm trying to create a bash script that will move all files recursively from a source folder to a target folder, and simply rename files if they already exist. Similar to the way M$ Windows does, when a file exists it auto-renames it with "<filemame> (X).<ext>", etc. except for ALL files.
I've create the below, which works fine for almost all scenarios except when a folder has a (.) period in its name and a file within that folder has no extension (no period in its name).
eg a folder-path-file such as: "./oldfolder/this.folder/filenamewithoutextension"
I get (incorrectly):
"./newfolder/this (1).folder/filenamewithoutextension"
if "./newfolder/this.folder/filenamewithoutextension" already exist in the target location (./newfolder),
instead of correctly naming the new file: "./oldfolder/this.folder/filenamewithoutextension (1)"
#!/bin/bash
source=$1 ; target=$2 ;
if [ "$source" != "" ] && [ "$target" != "" ] ; then
#recursive file search
find "$source" -type f -exec bash -c '
#setup variables
oldfile="$1" ; osource='"${source}"' ; otarget='"${target}"' ;
#set new target filename with target path
newfile="${oldfile/${osource}/${otarget}}" ;
#check if file already exists at target
[ -f "${newfile}" ] && {
#get the filename and fileextension for numbering - ISSUE HERE?
filename="${newfile%/}" ; newfileext="${newfile##*.}" ;
#compare filename and file extension for missing extension
if [ "$filename" == "$newfileext" ] ; then
#filename has no ext - perhaps fix the folder with a period issue here?
newfileext="" ;
else
newfileext=".$newfileext" ;
fi
#existing files counter
cnt=1 ; while [ -f "${newfile%.*} (${cnt})${newfileext}" ] ; do ((cnt+=1)); done
#set new filename with counter - New Name created here *** Needs re-work, as folder with a period = fail
newfile="${newfile%.*} (${cnt})${newfileext}";
}
#show mv command
echo "mv \"$oldfile\" \"${newfile}\""
' _ {} \;
else
echo "Requires source and target folders";
fi
I suspect the issue is, how to properly identify the filename and extension, found in this line:
filename="${newfile%/}" ; newfileext="${newfile##*.}" which doesn't identify a filename properly (files are always after the last /).
Any suggestion on how to make it work properly?
UPDATED: Just some completion notes - Issues fixes with:
Initially Splitting each full path filename: path - filename - (optional ext)
Reconstructing the full path filename: path - filename - counter - (optional ext)
fixed the file move to ensure directory structure exists with mkdir -p (mv does not create new folders if they do not exist in the target location).
Maybe you could try this instead?
filename="${newfile##*/}" ; newfileext="${filename#*.}"
The first pattern means: remove the longest prefix (in a greedy way) up to the last /.
The second one: remove the prefix up to the first dot (the greedy mode seems unnecessary here) − and as you already noted, in case the filename contains no dot, you will get newfileext == filename…
Example session:
newfile='./oldfolder/this.folder/filenamewithoutextension'
filename="${newfile##*/}"; newfileext="${filename#*.}"
printf "%s\n" "$filename"
#→ filenamewithoutextension
printf "%s\n" "$newfileext"
#→ filenamewithoutextension
newfile='./oldfolder/this.folder/file.tar.gz'
filename="${newfile##*/}"; newfileext="${filename#*.}"
printf "%s\n" "$filename"
#→ file.tar.gz
printf "%s\n" "$newfileext"
#→ tar.gz
I have a large data file that contains many joint files.
It has an separate index file has that file name, start + end byte of each file within the data file.
I'm needing help in creating a bash script to split the large file into it's 1000's of sub files.
Data File : fileafilebfilec etc
Index File:
filename.png<0>3049
folder\filename2.png<3049>6136.
I guess this needs to loop through each line of the index file, then using dd to extract the relevant bytes into a file. Maybe a fiddly part might be the folder structure bracket being windows style rather than linux style.
Any help much appreciated.
while read p; do
q=${p#*<}
startbyte=${q%>*}
endbyte=${q#*>}
filename=${p%<*}
count=$(($endbyte - $startbyte))
toprint="processing $filename startbyte: $startbyte endbyte: $endbyte count: $c$
echo $toprint
done <indexfile
Worked it out :-) FYI:
while read p; do
#sort out variables
q=${p#*<}
startbyte=${q%>*}
endbyte=${q#*>}
filename=${p%<*}
count=$(($endbyte - $startbyte))
#let it know we're working
toprint="processing $filename startbyte: $startbyte endbyte: $endbyte count: $c$
echo $toprint
if [[ $filename == *"/"* ]]; then
echo "have found /"
directory=${filename%/*}
#if no directory exists, create it
if [ ! -d "$directory" ]; then
# Control will enter here if $directory doesn't exist.
echo "directory not found - creating one"
mkdir ~/etg/$directory
fi
fi
dd skip=$startbyte count=$count if=~/etg/largefile of=~/etg/$filename bs=1
done <indexfile
I have a situation in which i have to list file which is of the type as
databaseName.schemaName#1234sdf2323.lock where _Database_Name and _target_schema_name
_lockFolder are variables
# is a token then it is followed by the random alphanumeric number and the same is end with .lock .
I have acheived this in batch file through the code as
FOR /R %_lockFolder% %%F in (%_Database_Name%.%_target_schema_name%#*.lock) do (
for /f "tokens=1* delims=# " %%G IN ("%%~nF") DO (
SET _no=%%H
)
)
but when i am changing it into bash enviorment so that it can run on unix enviorment
for entry in "${_lockFolder}"/*
do
echo ENTRY "$entry"
name='${_lockFolder}/${_Database_Name}.${_target_schema_name}#*.lock'
ls -l $name > "${lockFolder}"
if [ "$?" -eq "0" ]
then
echo "Do your work here"
else
echo "No files are there for the given pattern"
fi
# exit 21
done
It is not able to recognize the pattern . The files are present in the folder which i have specified
you can simple write a for loop like,
for file in `find ${lockdir} -name "^${_lockFolder}/${_Database_Name}.${_target_schema_name}#*.lock$"`
do
echo $file
:
:
your job
done
example,
[root#giam20 unix]# ls
checksumupdator.sh GIAMMEFProcessor.sh GIAMRoleExtractor.sh
GIAMAccountExtractor.sh GIAMMetaDataLoader.sh GIAMRoleLoader.sh
GIAMAccountLoader.sh GIAMOOPControlledAttrExtractor.sh GIAMRoleMappingLoader.sh
GIAMAccountTransferLoader.sh GIAMOOPControlledAttrsLoader.sh
[root#giam20 unix]# find . -name "GIAM*.sh"
./GIAMAccountTransferLoader.sh
./GIAMIntermediateCodeUpgrader.sh
./GIAMServiceUpdator.sh
./GIAMOOPControlledAttrsUpdator.sh
./GIAMRoleUpdator.sh
./GIAMProvisioningPolicyExtractor.sh
./GIAMCompExemptionExtractor.sh
./GIAMApprovalNotificationLoader.sh
In Unix shell scripting, variable references are not resolved in a string that is enclosed in single quotes. Therefore, in this line
name='${_lockFolder}/${_Database_Name}.${_target_schema_name}#*.lock'
the value will be stored into name literally, including all the ${name} references. It is no surprise then that the pattern is not matched later.
So, just change the single quotes to double quotes:
name="${_lockFolder}/${_Database_Name}.${_target_schema_name}#*.lock"
I have a large library of music files stored as ./"Artist Name"/"Album Name"/"audio files".
I would like to re-organize to ./"Artist name --- Album name"/"audio files"
And be able to put it back to how it was.
This is your first loop
mkdir "ArtistName-Albumname"
cd "ArtistName/Albumname"
for filename in *; do
mv "$filename" "$ArtistName-Albumname/" ;;
done
I think something like this will get you started. It doesn't do anything but just parses your structure and works out what needs doing:
#!/bin/bash
find . -depth 2 -type d | while IFS= read p
do
p=${p:2} # Trim ./ from start
album=${p##*/} # album is everything after /
artist=${p%/*} # artist is everything before /
newloc="${artist} - ${album}"
echo Would move $artist/$album to ${newloc}
done
Sample output:
Would move artist1/album1 to artist1 - album1
Would move artist1/album2 to artist1 - album2
Would move artist1/album3 to artist1 - album3
Would move artist1/album4 to artist1 - album4
Would move artist2/album1 to artist2 - album1
Would move artist2/album2 to artist2 - album2
Would move artist2/album3 to artist2 - album3
Would move artist3/album1 to artist3 - album1
Would move artist3/album2 to artist3 - album2
Would move artist3/album3 to artist3 - album3
Would move artist4/album1 to artist4 - album1
Would move artist4/album2 to artist4 - album2
Would move artist4/album3 to artist4 - album3
Would move artist4/album4 to artist4 - album4
Would move artist4/album5 to artist4 - album5
The reverse operation is tricky as there may be a hyphen naturally occurring in an album name, so it will be hard to differentiate that from the hyphen introduced by the code below.
Here is the script to merge folder from "Artist Name"/"Album Name" to "Artist name - Album name"
#! /usr/bin/env bash
cd /PATH
find . -type f |while read -r line
do
file=${line##*/}
folder=${line%/*}
album=${folder##*/}
folder=${folder%/*}
artist=${folder##*/}
newfolder="$artist - $album"
mkdir -p "$newfolder"
echo mv "$line" "$newfolder"
done
If you understand above script, you should be fine to write a reverse one.
Took some time for an amateur. But here's my final sollution. Thank you so much for your input #mark #BMW.
function flatten() {
echo flattening...
ls -ld --format=single-column */* | while IFS= read albumpath
do
echo Flattening "$albumpath"
artist=${albumpath%/*} # artist is everything before /
echo Artist: "$artist"
echo Album: "$albumpath"
album=${albumpath##*/} # album is everything after /
newfolder="$artist --- $album"
echo Moving "$albumpath" to "$newfolder"
mv "$albumpath" "$newfolder"
done
find . -depth -type d -empty -delete #delete all empty (artist)folders
}
function unflatten() {
ls -ld --format=single-column */ | while IFS= read pathname
do
echo REVERSING "$pathname" ;
artist=${pathname% ---*} # artist is everything before " ---"
echo Artist: "$artist"
if [ ! -d "$artist" ];
then
echo Creating "$artist" folder
mkdir "$artist"
fi
album=${pathname##*--- } # album is everything after "--- "
album=${album%/*} # strip trailing /
echo Album: "$album"
echo Moving "$pathname" "$artist"/"$album"
mv "$pathname" "$artist"/"$album"
done
}