bash: Call a function from a while with arguments - bash

I have this working:
$ find . -name 'copy_*.txt' |while read i ; do echo $i; git mv $i ${i%.txt}.cob ;done
I whant to have the main body in a bash function:
$ my_mv () { echo $1; mv $1 ${1%.cob}.toto; }
To then call it with:
$ find . -name 'copy_*.txt' |while read i ;do my_mv $i; done
But I get a silent execution and nothing append:
$ my_mv () { echo $1; mv $1 ${1%.cob}.toto; }
$ find . -name 'copy_*.txt' |while read i ;do my_mv $i; done
$
same with:
$ my_mv () { printf '%s\n' $1; mv $1 ${1%.cob}.toto; }

As said in the comments, it's not working, because you're searching for files copy_*.txt and are trying to move files with suffix .cob. Furthermore
your variables are not quoted and will cause problems with white space in filenames.
Export your function to make my_mv available in find and use -exec to prevent problems with filenames containing newlines:
my_mv () { for i; do echo "$i"; mv "$i" "${i%.cob}.toto"; done; }
export -f my_mv
find . -name 'copy_*.cob' -exec bash -c 'my_mv "$#"' bash {} +
Its often easier to use a small shell script instead of a function:
find . -name 'copy_*.cob' -exec sh -c '
for i; do
echo "$i"
mv "$i" "${i%.cob}.toto"
done
' sh {} +
Or move the code into a shell script mymv.sh
#!/bin/sh
for i; do
echo "$i"
mv "$i" "${i%.cob}.toto"
done
and execute the script in find:
find . -name 'copy_*.cob' -exec ./mymv.sh {} +

Related

Remove YYYY_MM_DD_HH_MM from filename

We have few csv and xml files in following formats
String_YYYY_MM_DD_HH_MM.csv
String_YYYY_MM_DD_HH_MM.xml
String.xml
String.csv
Examples:
Reference_Categories_2021_02_24_17_14.csv
CD_CategoryTree_2021_02_24_17_14.csv
New_Categories.xml
Mobile_Footnote_2021_03_05_16_21.csv
Campaign_Version_2018_09_24_20_00.xml
Campaign_new.csv
Now we have to remove _YYYY_MM_DD_HH_MM from filenames so result will be
Reference_Categories.csv
CD_CategoryTree.csv
New_Categories.xml
Mobile_Footnote.csv
Campaign_Version.xml
Campaign_new.csv
Any idea how to do that in bash?
In pure bash:
pat='_[0-9][0-9][0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]_[0-9][0-9]'
for f in *$pat*.{csv,xml}; do echo mv "$f" "${f/$pat}"; done
Delete the echo if the output looks fine.
With bash Something like:
shopt -s nullglob
for f in *.{xml,csv}; do
ext="${f##*.}"
[[ "${f%%_[0-9]*}" = *.#(xml|csv) ]] && continue
echo mv -v -- "$f" "${f%%_[0-9]*}.$ext"
done
With the =~ operator and BASH_REMATCH
shopt -s nullglob
regexp='^(.{1,})(_[[:digit:]]{4}_[[:digit:]]{2}_[[:digit:]]{2}_[[:digit:]]{2}_[[:digit:]]{2})([.].*)$'
for f in *.{xml,csv}; do
[[ "$f" =~ $regexp ]] &&
echo mv -v -- "$f" "${BASH_REMATCH[1]}${BASH_REMATCH[-1]}"
done
Remove the echo if you're satisfied with the output.
Using bash, find, and awk:
Use find to find files with .csv or .xml suffix in the current directory. Pipe the find output to awk and create the mv commands that are output and passed to bash.
bash < <(find * -type f \( -name '*.csv' -o -name '*.xml' \) | awk '{orig=$0; gsub(/_[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]{2}_[0-9]{2}/,""); print "mv "orig" "$0}')
Directory contents before:
find * -type f
CD_CategoryTree_2021_02_24_17_14.csv
Campaign_Version_2018_09_24_20_00.xml
Campaign_new.csv
Mobile_Footnote_2021_03_05_16_21.csv
New_Categories.xml
Reference_Categories_2021_02_24_17_14.csv
Directory contents after:
find * -type f
CD_CategoryTree.csv
Campaign_Version.xml
Campaign_new.csv
Mobile_Footnote.csv
New_Categories.xml
Reference_Categories.csv

moving files to their respective folders using bash scripting

I have files in this format:
2022-03-5344-REQUEST.jpg
2022-03-5344-IMAGE.jpg
2022-03-5344-00imgtest.jpg
2022-03-5344-anotherone.JPG
2022-03-5343-kdijffj.JPG
2022-03-5343-zslkjfs.jpg
2022-03-5343-myimage-2010.jpg
2022-03-5343-anotherone.png
2022-03-5342-ebee5654.jpeg
2022-03-5342-dec.jpg
2022-03-5341-att.jpg
2022-03-5341-timephoto_december.jpeg
....
about 13k images like these.
I want to create folders like:
2022-03-5344/
2022-03-5343/
2022-03-5342/
2022-03-5341/
....
I started manually moving them like:
mkdir name
mv name-* name/
But of course I'm not gonna repeat this process for 13k files.
So I want to do this using bash scripting, and since I am new to bash, and I am working on a production environment, I want to play it safe, but it doesn't give me my results. This is what I did so far:
#!/bin/bash
name = $1
mkdir "$name"
mv "${name}-*" $name/
and all I can do is: ./move.sh name for every folder, I didn't know how to automate this using loops.
With bash and a regex. I assume that the files are all in the current directory.
for name in *; do
if [[ "$name" =~ (^....-..-....)- ]]; then
dir="${BASH_REMATCH[1]}"; # dir contains 2022-03-5344, e.g.
echo mkdir -p "$dir" || exit 1;
echo mv -v "$name" "$dir";
fi;
done
If output looks okay, remove both echo.
Try this
xargs -i sh -c 'mkdir -p {}; mv {}-* {}' < <(ls *-*-*-*|awk -F- -vOFS=- '{print $1,$2,$3}'|uniq)
Or:
find . -maxdepth 1 -type f -name "*-*-*-*" | \
awk -F- -vOFS=- '{print $1,$2,$3}' | \
sort -u | \
xargs -i sh -c 'mkdir -p {}; mv {}-* {}'
Or find with regex:
find . -maxdepth 1 -type f -regextype posix-extended -regex ".*/[0-9]{4}-[0-9]{2}-[0-9]{4}.*"
You could use awk
$ cat awk.script
/^[[:digit:]-]/ && ! a[$1]++ {
dir=$1
} /^[[:digit:]-]/ {
system("sudo mkdir " dir )
system("sudo mv " $0" "dir"/"$0)
}
To call the script and use for your purposes;
$ awk -F"-([0-9]+)?[[:alpha:]]+.*" -f awk.script <(ls)
You will see some errors such as;
mkdir: cannot create directory ‘2022-03-5341’: File exists
after the initial dir has been created, you can safely ignore these as the dir now exist.
The content of each directory will now have the relevant files
$ ls 2022-03-5344
2022-03-5344-00imgtest.jpg 2022-03-5344-IMAGE.jpg 2022-03-5344-REQUEST.jpg 2022-03-5344-anotherone.JPG

Is there a way to perform echo | tee during find -exec?

I have a problem with a code similar to the following:
function echotee() { echo $1 | tee -a ${FILE}; }
export -f echotee
find . -delete -exec sh -c 'echotee "Deleting: {}"' \;
The function echotee usually works as expected. However, during the -exec it does not. Indeed, it just prints on the terminal, omitting tee.
Hoping the question is not too trivial, thanks in advance.
Why don't you just use this:
find . -delete -exec sh -c 'echo "Deleting: $1" | tee -a "$2"' _ {} "${FILE}" \;
No need to define and call a function.
You mentioned in a comment that you want to use echotee as a central point to print and log information. Have you considered a setup like this instead:
#!/usr/bin/env bash
# Send all script output to console and logfile
LOGFILE="..."
exec > >(tee -ia "${LOGFILE}") 2>&1
find . -delete -printf "Deleting: %f\n"
or this:
#!/usr/bin/env bash
# Set up fd 3 to send output to console and logfile on demand
LOGFILE="..."
exec 3> >(tee -ia "${LOGFILE}")
find . -delete -printf "Deleting: %f\n" 1>&3 2>&1
Use name() instead of function name().
You did not set nor export FILE variable.
sh does not support exporting functions. It's a feature of bash, you have to call bash.
sh -c ' .... "{}"' will break on filenames containing " character. Put it as positional argument and use $1.
$1 and $FILE expansions are not quoted and are subject to word splitting and filename expansion.
echo $1 will break on filenames like -e. Prefer printf.
Check your scripts with shellcheck - it will catch many such mistakes.
I think you meant to:
FILE=/tmp/log.txt
echotee() { printf "%s\n" "$1" | tee -a "$FILE"; }
export -f echotee
export FILE
find . -exec bash -c 'echotee "Deleting: $1"' -- {} \;
But the version from Shawn with -printf "Deleting: %p\n" | tee "$FILE" looks just nicer.
I think spawning tee and pipe will be slower then, I think doing like so could be a bit faster:
echotee() { printf "%s\n" "$1" >> "$FILE"; printf "%s\n" "$1"; }
or like:
exec 10>>"$FILE"
echotee() { printf "%s\n" "$1" >&10; printf "%s\n" "$1"; }
You could remove the pipe either way, just:
echotee() { tee -a "$FILE" <<<"$1"; }

Finding a particular string at a specific line number, anywhere in a directory

I am trying to find instances of the word package in line 171 of any file in a certain directory. What is the best way to do this?
You can do, from the directory you want to check, recursively:
find . -type f -exec bash -c '[[ $(sed -n '171p' "$1") =~ package ]] && echo "$1"' _ {} +
this will show you the filenames that contain package in their 171-th line.
Non-recursively:
find . -maxdepth 1 -type f -exec bash -c '[[ $(sed -n '171p' "$1") =~ package ]] && echo "$1"' _ {} +
Example:
I am looking for bar:
$ cat foo
foo
bar
$ find . -type f -exec bash -c '[[ $(sed -n '2p' "$1") =~ bar ]] && echo "$1"' _ {} +
./foo
#This will search second line of all the files and grep over them.
#Pass search pattern as argument.
pattern=$1
for file in *
do
cat $file |sed -n '2p' |grep $pattern
done

bash scripting challenge

I need to write a bash script that will iterate through the contents of a directory (including subdirectories) and perform the following replacements:
replace 'foo' in any file names with 'bar'
replace 'foo' in the contents of any files with 'bar'
So far all I've got is
find . -name '*' -exec {} \;
:-)
With RH rename:
find -f \( -exec sed -i s/foo/bar/g \; , -name \*foo\* -exec rename foo bar {} \; \)
find "$#" -depth -exec sed -i -e s/foo/bar/g {} \; , -name '*foo*' -print0 |
while read -d '' file; do
base=$(basename "$file")
mv "$file" "$(dirname "$file")/${base//foo/bar}"
done
UPDATED: 1632 EST
Now handles whitespace but 'while read item' never terminates. Better,
but still not right. Will keep
working on this.
aj#mmdev0:~/foo_to_bar$ cat script.sh
#!/bin/bash
dirty=true
while ${dirty}
do
find ./ -name "*" |sed -s 's/ /\ /g'|while read item
do
if [[ ${item} == "./script.sh" ]]
then
continue
fi
echo "working on: ${item}"
if [[ ${item} == *foo* ]]
then
rename 's/foo/bar/' "${item}"
dirty=true
break
fi
if [[ ! -d ${item} ]]
then
cat "${item}" |sed -e 's/foo/bar/g' > "${item}".sed; mv "${item}".sed "${item}"
fi
dirty=false
done
done
#!/bin/bash
function RecurseDirs
{
oldIFS=$IFS
IFS=$'\n'
for f in *
do
if [[ -f "${f}" ]]; then
newf=`echo "${f}" | sed -e 's/foo/bar/g'`
sed -e 's/foo/bar/g' < "${f}" > "${newf}"
fi
if [[ -d "${f}" && "${f}" != '.' && "${f}" != '..' && ! -L "${f}" ]]; then
cd "${f}"
RecurseDirs .
cd ..
fi
done
IFS=$oldIFS
}
RecurseDirs .
bash 4.0
#!/bin/bash
shopt -s globstar
path="/path"
cd $path
for file in **
do
if [ -d "$file" ] && [[ "$file" =~ ".*foo.*" ]];then
echo mv "$file" "${file//foo/bar}"
elif [ -f "$file" ];then
while read -r line
do
case "$line" in
*foo*) line="${line//foo/bar}";;
esac
echo "$line"
done < "$file" > temp
echo mv temp "$file"
fi
done
remove the 'echo' to commit changes
for f in `tree -fi | grep foo`; do sed -i -e 's/foo/bar/g' $f ; done
Yet another find-exec solution:
find . -type f -exec bash -c '
path="{}";
dirName="${path%/*}";
baseName="${path##*/}";
nbaseName="${baseName/foo/bar}";
#nbaseName="${baseName//foo/bar}";
# cf. http://www.bash-hackers.org/wiki/doku.php?id=howto:edit-ed
ed -s "${path}" <<< $'H\ng/foo/s/foo/bar/g\nwq';
#sed -i "" -e 's/foo/bar/g' "${path}"; # alternative for large files
exec mv -iv "{}" "${dirName}/${nbaseName}"
' \;
correction to find-exec approach by gregb (adding quotes):
# compare
bash -c '
echo $'a\nb\nc'
'
bash -c '
echo $'"'a\nb\nc'"'
'
# therefore we need
find . -type f -exec bash -c '
...
ed -s "${path}" <<< $'"'H\ng/foo/s/foo/bar/g\nwq'"';
...
' \;

Resources