Bash command does not work in script but in console - bash

I have running the two commands in a script where I want to check if all files in a directoy are media:
1 All_LINES=$(ls -1 | wc -l)
2 echo "Number of lines: ${All_LINES}"
3
4 REACHED_LINES=$(ls -1 *(*.JPG|*.jpg|*.PNG|*.png|*.JPEG|*.jpeg) | wc -l)
5 echo "Number of reached lines: ${REACHED_LINES}"
if[...]
Running line 4 and 5 sequentially in a shell it works as expected, counting all files ending with .jpg, .JPG...
Running all together in a script gives the following error though:
Number of lines: 12
/home/andreas/.bash_scripts/rnimgs: command substitution: line 17: syntax error near unexpected token `('
/home/andreas/.bash_scripts/rnimgs: command substitution: line 17: `ls -1 *(*.JPG|*.jpg|*.PNG|*.png|*.JPEG|*.jpeg) | wc -l)'
Number of reached lines:
Could somebody explain this to me, please?
EDIT: This is as far as I got:
#!/bin/bash
# script to rename images, sorted by "type" and "date modified" and named by current folder
#get/set basename for files
CURRENT_BASENAME=$(basename "${PWD}")
echo -e "Current directory/basename is: ${CURRENT_BASENAME}\n"
read -e -p "Please enter basename: " -i "${CURRENT_BASENAME}" BASENAME
echo -e "\nNew basename is: ${BASENAME}\n"
#start
echo -e "START RENAMING"
#get nr of all files in directory
All_LINES=$(ls -1 | wc -l)
echo "Number of lines: ${All_LINES}"
#get nr of media files in directory
REACHED_LINES=$(ls -1 *(*.JPG|*.jpg|*.PNG|*.png|*.JPEG|*.jpeg) | wc -l)
echo "Number of reached lines: ${REACHED_LINES}"
EDIT1: Thanks again guys, this is my result so far. Still room for improvement, but a start and ready to test.
#!/bin/bash
#script to rename media files to a choosable name (default: ${basename} of current directory) and sorted by date modified
#config
media_file_extensions="(*.JPG|*.jpg|*.PNG|*.png|*.JPEG|*.jpeg)"
#enable option extglob (extended globbing): If set, the extended pattern matching features described above under Pathname Expansion are enabled.
#more info: https://askubuntu.com/questions/889744/what-is-the-purpose-of-shopt-s-extglob
#used for regex
shopt -s extglob
#safe and set IFS (The Internal Field Separator): IFS is used for word splitting after expansion and to split lines into words with the read builtin command.
#more info: https://bash.cyberciti.biz/guide/$IFS
#used to get blanks in filenames
SAVEIFS=$IFS;
IFS=$(echo -en "\n\b");
#get and print current directory
basedir=$PWD
echo "Directory:" $basedir
#get and print nr of files in current directory
all_files=( "$basedir"/* )
echo "Number of files in directory: ${#all_files[#]}"
#get and print nr of media files in current directory
media_files=( "$basedir"/*${media_file_extensions} )
echo -e "Number of media files in directory: ${#media_files[#]}\n"
#validation if #all_files = #media_files
if [ ${#all_files[#]} -ne ${#media_files[#]} ]
then
echo "ABORT - YOU DID NOT REACH ALL FILES, PLEASE CHECK YOUR FILE ENDINGS"
exit
fi
#make a copy
backup_dir="backup_95f528fd438ef6fa5dd38808cdb10f"
backup_path="${basedir}/${backup_dir}"
mkdir "${backup_path}"
rsync -r "${basedir}/" "${backup_path}" --exclude "${backup_dir}"
echo "BACKUP MADE"
echo -e "START RENAMING"
#set new basename
basename=$(basename "${PWD}")
read -e -p "Please enter file basename: " -i "$basename" basename
echo -e "New basename is: ${basename}\n"
#variables
counter=1;
new_name="";
file_extension="";
#iterate over files
for f in $(ls -1 -t -r *${media_file_extensions})
do
#catch file name
echo "Current file is: $f"
#catch file extension
file_extension="${f##*.}";
echo "Current file extension is: ${file_extension}"
#create new name
new_name="${basename}_${counter}.${file_extension}"
echo "New name is: ${new_name}";
#rename file
mv $f "${new_name}";
echo -e "Counter is: ${counter}\n"
((counter++))
done
#get and print nr of media files before
echo "Number of media files before: ${#media_files[#]}"
#get and print nr of media files after
media_files=( "$basedir"/*${media_file_extensions} )
echo -e "Number of media files after: ${#media_files[#]}\n"
#delete backup?
while true; do
read -p "Do you wish to keep the result? " yn
case $yn in
[Yy]* ) rm -r ${backup_path}; echo "BACKUP DELETED"; break ;;
[Nn]* ) rm -r !(${backup_dir}); rsync -r "${backup_path}/" "${basedir}"; rm -r ${backup_path}; echo "BACKUP RESTORED THEN DELETED"; break;;
* ) echo "Please answer yes or no.";;
esac
done
#reverse IFS to default
IFS=$SAVEIFS;
echo -e "END RENAMING"

You don't need to and don't want to use ls at all here. See https://mywiki.wooledge.org/ParsingLs
Also, don't use uppercase for your private variables. See Correct Bash and shell script variable capitalization
#!/bin/bash
shopt -s extglob
read -e -p "Please enter basename: " -i "$PWD" basedir
all_files=( "$basedir"/* )
echo "Number of files: ${#all_files[#]}"
media_files=( "$basedir"/*(*.JPG|*.jpg|*.PNG|*.png|*.JPEG|*.jpeg) )
echo "Number of media files: ${#media_files[#]}"

As #chepner already pointed out in a comment, you likely need to explicitly enable extended globbing on your script. c.f. Greg's WIKI
It's also possible to condense that pattern to eliminate some redundancy and add mixed case if you like -
$: ls -1 *.*([Jj][Pp]*([Ee])[Gg]|[Pp][Nn][Gg])
a.jpg
b.JPG
c.jpeg
d.JPEG
mixed.jPeG
mixed.pNg
x.png
y.PNG
You can also accomplish this without ls, which is error-prone. Try this:
$: all_lines=(*)
$: echo ${#all_lines[#]}
55
$: reached_lines=( *.*([Jj][Pp]*([Ee])[Gg]|[Pp][Nn][Gg]) )
$: echo ${#reached_lines[#]}
8
c.f. this breakdown
If all you want is counts, but prefer not to include directories:
all_dirs=( */ )
num_files=$(( ${#all_files[#]} - ${#all_dirs[#]} ))
If there's a chance you will have a directory with a name that matches your jpg/png pattern, then it gets trickier. At that point it's probably easier to just use #markp-fuso's solution.
One last thing - avoid all-caps variable names. Those are generally reserved for system stuff.

Assuming the OP wants to limit the counts to normal files (ie, exclude non-files like directories, pipes, symbolic links, etc), a solution based on find may provide more accurate counts.
Updating OP's original code to use find (ignoring dot files for now):
ALL_LINES=$(find . -maxdepth 1 -type f | wc -l)
echo "Number of lines: ${ALL_LINES}"
REACHED_LINES=$(find . -maxdepth 1 -type f \( -iname '*.jpg' -o -iname '*.png' -o -iname '*.jpeg' \) | wc -l)
echo "Number of reached lines: ${REACHED_LINES}"

Related

Sort files in a directory before renaming them

I am learning Bash and therefore I would like to write a script with runs over my files and names them after the current directory.
E.g. current_folder_1, current_folder_2, current_folder_3...
#!/bin/bash
# script to rename images, sorted by "type" and "date modified" and named by current folder
#get current folder which is also basename of files
basename=$(basename "$PWD");
echo "Current folder is: ${basename}";
echo '';
#set counter for iteration and variables
counter=1;
new_name="";
file_extension="";
#for each file in current folder
for f in *
do
#catch file name
echo "Current file is: ${f}"
#catch file extension
file_extension="${f##*.}";
echo "Current file extension is: ${file_extension}"
#create new name
new_name="${basename}_${counter}.${file_extension}"
echo "New name is: ${new_name}";
#mv $f "${new_name}";
echo "Counter is: ${counter}"
((counter++));
done
One of my two problems is I would like to sort them by first type and then date_modified before running the for-each-loop.
Something like
for f in * | sort -k "type" -k "date_modified"
[...]
I'd appreciate some help.
EDIT1: Solved the sorting by date problem with
for f in $(ls -1 -t -r)
This could be a start. I've commented in the code where I think it's needed but please ask if anything is unclear.
#!/bin/bash
#get current folder which is also basename of files
folder=$(basename "$PWD");
echo "Current folder is: ${folder}";
echo
#set counter for iteration
counter=1;
for f in *
do
file_extension="${f##*.}";
# replace all whitespaces with underscores
sort_extension=${file_extension//[[:space:]]/_}
# get modification time in seconds since epoch
sort_modtime=$(stat --format=%Y "$f")
# output fed to sort
echo $sort_extension $sort_modtime "/$f/"
# sort on extension first and modification time after
done | sort -k1 -k2n | while read -r dummy1 dummy2 file
do
# remove the slashes we added above
file=${file:1:-1}
file_extension="${file##*.}";
new_name="${folder}_${counter}.${file_extension}"
echo "moving \"$file\" to \"$new_name\""
#mv "$file" "$new_name"
(( counter++ ))
done
I decided to do a workaround with two loops, one for images and one for video.
#used for regex
shopt -s extglob
#used to get blanks in filenames
SAVEIFS=$IFS;
IFS=$(echo -en "\n\b");
[...]
image_extensions="(*.JPG|*.jpg|*.PNG|*.png|*.JPEG|*.jpeg)"
video_extensions="(*.mp4|*.gif)"
[...]
for f in $(ls -1 -t -r *${media_file_extensions})
[...]
for f in $(ls -1 -t -r *${video_extensions})
[...]
#reverse IFS to default
IFS=$SAVEIFS;
ls -lt | sort
The -t parameter will sort by date and time, and sort should sort by type of file.

Sequentially numbering of files in different folders while keeping the name after the number

I have a lot of ogg or wave files in different folders that I want to sequentially number while keeping everything that stands behind the prefixed number. The input may look like this
Folder1/01 Insbruck.ogg
02 From Milan to Rome.ogg
03 From Rome to Naples.ogg
Folder2/01 From Naples to Palermo.ogg
02 From Palermo to Syracrus.ogg
03 From Syracrus to Tropea
The output should be:
Folder1/01 Insbruck.ogg
02 From Milan to Rome.ogg
03 From Rome to Naples.ogg
Folder2/04 From Naples to Palermo.ogg
05 From Palermo to Syracrus.ogg
06 From Syracrus to Tropea.ogg
The sequential numbering across folders can be done with this BASH script that I found here:
find . | (i=0; while read f; do
let i+=1; mv "$f" "${f%/*}/$(printf %04d "$i").${f##*.}";
done)
But this script removes the title that I would like to keep.
TL;DR
Like this, using find and perl rename:
rename -n 's#/\d+#sprintf "/%0.2d", ++$::c#e' Folder*/*
Drop -n switch if the output looks good.
With -n, you only see the files that will really be renamed, so only 3 files from Folder2.
Going further
The variable $::c (or $main::c is a package variable) is a little hack to avoid the use of more complex expressions:
rename -n 's#/\d+#sprintf "/%0.2d", ++our $c#e' Folder*/*
or
rename -n '{ no strict; s#/\d+#sprintf "/%0.2d", ++$c#e; }' Folder*/*
or
rename -n '
do {
use 5.012;
state $c = 0;
s#/\d+#sprintf "/%0.2d", ++$c#e
}
' Folder*/*
Thanks go|dfish & Grinnz on freenode
A bash script for this job would be:
#!/bin/bash
argc=$#
width=${#argc}
n=0
for src; do
base=$(basename "$src")
dir=$(dirname "$src")
if ! [[ $base =~ ^[0-9]+\ .*\.(ogg|wav)$ ]]; then
echo "$src: Unexpected file name. Skipping..." >&2
continue
fi
printf -v dest "$dir/%0${width}d ${base#* }" $((++n))
echo "moving '$src' to '$dest'"
# mv -n "$src" "$dest"
done
and could be run as
./renum Folder*/*
assuming the script is saved as renum. It will just print out source and destination file names. To do actual moving, you should drop the # at the beginning of the line # mv -n "$src" "$dest" after making sure it will work as expected. Note that the mv command will not overwrite an existing file due to the -n option. This may or may not be desirable. The script will print out a warning message and skip unexpected file names, that is, the file names not fitting the pattern specified in the question.
The sequential numbering across folders can be done with this BASH script that I found here:
find . | (i=0; while read f; do
let i+=1; mv "$f" "${f%/*}/$(printf %04d "$i").${f##*.}";
done)
But this script removes the title that I would like to keep.
Not as robust as the accepted answer but this is the improved version of your script and just in case rename is not available.
#!/usr/bin/env bash
[[ -n $1 ]] || {
printf >&2 'Needs a directory as an argument!\n'
exit 1
}
n=1
directory=("$#")
while IFS= read -r files; do
if [[ $files =~ ^(.+)?\/([[:digit:]]+[^[:blank:]]+)(.+)$ ]]; then
printf -v int '%02d' "$((n++))"
[[ -e "${BASH_REMATCH[1]}/$int${BASH_REMATCH[3]}" ]] && {
printf '%s is already in sequential order, skipping!\n' "$files"
continue
}
echo mv -v "$files" "${BASH_REMATCH[1]}/$int${BASH_REMATCH[3]}"
fi
done < <(find "${directory[#]}" -type f | sort )
Now run the script with the directory in question as the argument.
./myscript Folder*/
or
./myscript Folder1/
or
./myscript Folder2/
or a . the . is the current directory.
./myscript .
and so on...
Remove the echo if you're satisfied with the output.

counting files, directories and subdirectories in a given path

I am trying to figure how to run a simple script "count.sh" that is called together with a path, e.g.:
count.sh /home/users/documents/myScripts
The script will need to iterate over the directories in this path and print how many files and folders (including hidden) in each level of this path.
For example:
7
8
9
10
(myScripts - 7, documents - 8, users -9, home - 10)
And by the way, can I run this script using count.sh pwd?
More or less something like that:
#!/bin/sh
P="$1"
while [ "/" != "$P" ]; do
echo "$P `find \"$P\" -maxdepth 1 -type f | wc -l`"
P=`dirname "$P"`;
done
echo "$P `find \"$P\" -maxdepth 1 -type f | wc -l`"
You can use it from the current directory with script.sh `pwd`
Another approach is to handle separation or tokenizing the path using an array and controlling word-splitting with the internal field separator (IFS). You can include the root directory if desired (you would need to trim the additional leading '/' in the printout in that case)
#!/bin/bash
[ -z "$1" -o ! -d "$1" ] && {
printf "error: directory argument required.\n"
exit 1
}
p="$1" ## remove two lines to include /
[ "${p:0:1}" = "/" ] && p="${p:1}"
oifs="$IFS" ## save internal field separator value
IFS=$'/' ## set to break on '/'
array=( $p ) ## tokenize given path into array
IFS="$oifs" ## restore original IFS
## print out path level info using wc
for ((i=0; i<${#array[#]}; i++)); do
dirnm="${dirnm}/${array[i]}"
printf "%d. %s -- %d\n" "$((i+1))" "$dirnm" $(($(ls -Al "$dirnm" | wc -l)-1))
done
Example Output
$ bash filedircount.sh /home/david/tmp
1. /home -- 5
2. /home/david -- 132
3. /home/david/tmp -- 113
As an alternative, you could use a for loop to loop through and count the items in the directory at each level instead of using wc if desired.
You could try the following
#!/bin/bash
DIR=$(cd "$1" ; pwd)
PREFIX=
until [ "$DIR" = / ] ; do
echo -n "$PREFIX"$(basename "$DIR")" "$(ls -Ab "$DIR" | wc -l)
DIR=$(dirname "$DIR")
PREFIX=", "
done
echo
(ls -Ab lists all files and folders except . and .. and escapes special characters so that only one line is printed per file even if filenames include newline characters. wc -l counts lines.)
You can invoke the script using
count.sh `pwd`

How can I search a group of files for any part of a string?

All I want to do is to search a directory of files for ANY part of a string. Here is the section of my code that processes this:
(input is set through user input)
input2=$(eval find . -name "$input" -print | sed -n 1p) #Using find to search for the file I need to read
... (Checking the file I 'found' actually exists)
out=$(eval $(shuf -n 1 "$input2")) #Shuffle it randomly
echo "$out" #Echo the output
If I had the file 'date' with the contents date "+%A %d %B %Y" I could type 'date' and get the intended result but 'What is the date' would fail with
find: paths must precede expression: is
Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]
What is the easiest way to fix this problem?
edit
I haven't tried much as I am at a loss of what to do. I have tried wildcards in the search string but they don't work at all. Here is the entire input-to-output part of the source code:
while true; do
echo -n ">"
read input
input=${input,,}
echo "[DEBUG]: Recieved $input"
echo "[IO]: Accessing database"
input2=$(eval find . -name "$input" -print | sed -n 1p) || out="WARNING: Input/Output Error{TYPE=IO.GENERAL, VALUE=$input2}"
echo "[INPUT2]: Recieved $input2"
if [ ! -f "$input2" ]
then
input2="./db/unknowncommand"
fi
#out=$(<"db/$input") || out="WARNING: Input/Output ERROR" #Old 'just print whatever is in the file approach
out=$(eval $(shuf -n 1 "$input2")) || out="WARNING: Input/Output ERROR{TYPE=IO.GENERAL, VALUE=$input}" #Random output
echo "$out"
./JAISpeak "$out" #A small text-to-speech program I made
done
For the "look for any part of the string in filenames" part, I'd do:
input2=$(for partname in ${input}; do find . -name "${partname}" -print ; done)
This will match every occurrence it can...
If you only need the first occurrence (weird, but hey, it's a bit the same with $PATH) :
input2=$(for partname in ${input}; do find . -name "${partname}" -print ; done | head -1)
Note that i use ${input} instead of "${input}" on purpose: I want to separate each words in ${input} and loop over each one individually.
I let you revise the rest of your program ^^ (I need to go ...)

Adding up file sizes in bash shells

I've written a shell script that takes a directory as an arg and prints the file names and sizes, I wanted to find out how to add up the file sizes and store them so that I can print them after the loop. I've tried a few things but haven't gotten anywhere so far, any ideas?
#!/bin/bash
echo "Directory <$1> contains the following files:"
let "x=0"
TEMPFILE=./count.tmp
echo 0 > $TEMPFILE
ls $1 |
while read file
do
if [ -f $1/$file ]
then
echo "file: [$file]"
stat -c%s $file > $TEMPFILE
fi
cat $TEMPFILE
done
echo "number of files:"
cat ./count.tmp
Help would be thoroughly appreciated.
A number of issues in your code:
Don't parse ls
Quote variables in large majority of cases
Don't use temp files when they're not needed
Use already made tools like du for this (see comments)
Assuming you're just wanting to get practice at this and/or want to do something else other than what du already does, you should change syntax to something like
#!/bin/bash
dir="$1"
[[ $dir == *'/' ]] || dir="$dir/"
if [[ -d $dir ]]; then
echo "Directory <$1> contains the following files:"
else
echo "<$1> is not a valid directory, exiting"
exit 1
fi
shopt -s dotglob
for file in "$dir"*; do
if [[ -f $file ]]; then
echo "file: [$file]"
((size+=$(stat -c%s "$file")))
fi
done
echo "$size"
Note:
You don't have to pre-allocate variables in bash, $size is assumed to be 0
You can use (()) for math that doesn't require decimal places.
You can use globs (*) to get all files (including dirs, symlinks, etc...) in a particular directory (and globstar ** for recursive)
shopt -s dotglob Is needed so it includes hidden .whatever files in glob matching.
You can use ls -l to find size of files:
echo "Directory $1 contains the following:"
size=0
for f in "$1"/*; do
if [[ ! -d $f ]]; then
while read _ _ _ _ bytes _; do
if [[ -n $bytes ]]; then
((size+=$bytes))
echo -e "\tFile: ${f/$1\//} Size: $bytes bytes"
fi
done < <(ls -l "$f")
fi
done
echo "$1 Files total size: $size bytes"
Parsing ls results for size is ok here as byte size will always be found in the 5th field.
If you know what the date stamp format for ls is on your system and portability isn't important, you can parse ls to reliably find both the size and file in a single while read loop.
echo "Directory $1 contains the following:"
size=0
while read _ _ _ _ bytes _ _ _ file; do
if [[ -f $1/$file ]]; then
((size+=$bytes))
echo -e "\tFile: $file Size: $bytes bytes"
fi
done < <(ls -l "$1")
echo "$1 Files total size: $size bytes"
Note: These solutions would not include hidden files. Use ls -la for that.
Depending on the need or preference, ls can also print sizes in a number of different formats using options like -h or --block-size=SIZE.
#!/bin/bash
echo "Directory <$1> contains the following files:"
find ${1}/* -prune -type f -ls | \
awk '{print; SIZE+=$7} END {print ""; print "total bytes: " SIZE}'
Use find with -prune (so it does not recurse into subdirectories) and -type f (so it will only list files and no symlinks or directories) and -ls (so it lists the files).
Pipe the output into awk and
for each line print the whole line (print; replace with print $NF to only print the last item of each line, which is the filename including the directory). Also add the value of the 7th field, which is the file size (in my version of find) to the variable SIZE.
After all lines have been processed (END) print the calculated total size.

Resources