Linux/sh: How to listing only files in folder (with witespace) and save into a variable in one line - shell

I try to get the lists of files in that format (with witespaces):
"file1.html" "file 2.php" "file_3.php"
#!/bin/sh
WEB_DIR="/volume1/web"
IFS=$'\n'
for file in $(find "${WEB_DIR}" -mindepth 1 -maxdepth 1 -type f) ; do
printf "\"$file\" "
done
output:
"/volume1/web/.htaccess" "/volume1/web/file.html" "/volume1/web/a b.php"
and the output is perfect but... how to put this output to the variable?
I do this...
IFS=$'\n'
for file in $(find "${WEB_DIR}" -mindepth 1 -maxdepth 1 -type f) ; do
mystring+=$(printf "\"$file\" ")
done
echo ${mystring}
In output I have this:
tmp.sh: line 48: mystring+="/volume1/web/.htaccess" : not found

Note:
The answer below accepts the premise of the question: to build a single string value with a list of double-quoted file paths, such as the one shown in the question (
"/volume1/web/.htaccess" "/volume1/web/file.html" "/volume1/web/a b.php")
However, the OP ultimately wanted to use that string as part of another command, which does not work, because the embedded double quotes are no longer recognized as string delimiters that identify separate arguments when you reference the string variable.
The correct solution is to use find ... -exec ...+ (in this case) or, generally, xargs to pass a list of filenames as operands (arguments) to another command; e.g., to pass the filenames to command foo ({} robustly passes all file paths, whether they contain spaces or not):
find "${WEB_DIR}" -mindepth 1 -maxdepth 1 -type f -exec foo -bar {} +
If the list of filenames doesn't go at the end of the target command line, an intermediate sh -c command is necessary:
find "${WEB_DIR}" -mindepth 1 -maxdepth 1 -type f -exec sh -c 'foo -bar "$#" -baz' - {} +
You've tagged your question bash, but your shebang line targets sh, where Bash extensions to the POSIX shell specification aren't guaranteed to be available.
In your case (it sounds like you're using dash, which act as /bin/sh on Ubuntu):
ANSI C-quoted strings such as $'\n' aren't available - $'\n' expands to literal $\n.
This means that any of these 3 literal chars. - $, \ or n serve as the field separator - in your case that just happened to work, because the file paths happened not to contain these characters.
Operator += isn't recognized - the whole token mystring+="/volume1/web/.htaccess" is treated as a command name, which causes the error you're seeing.
Possible solutions:
If you do want to target Bash, replace #!/bin/sh with #!/bin/bash.
Note that for your bash code to be fully robust, you should turn off globbing (set +f) in addition to setting $IFS.
Your code can be streamlined - see below.
If not (if your code must be portable), you must find POSIX-compliant alternatives - see below.
Here's a portable solution (works with any POSIX-compliant sh):
while IFS= read -r file; do
mystring="$mystring$(printf "\"$file\" ")"
done <<EOF
$(find "${WEB_DIR}" -mindepth 1 -maxdepth 1 -type f)
EOF
echo "$mystring"
A much more efficient variant that uses find ... -exec ... + to produce the output with (typically) a single printf call:
IFS= read -r mystring <<EOF
$(find "${WEB_DIR}" -mindepth 1 -maxdepth 1 -type f -exec printf '"%s" ' {} +)
EOF
echo "$mystring"
The bash equivalent, using a process substitution (<(...)):
IFS= read -r mystring < \
<(find "${WEB_DIR}" -mindepth 1 -maxdepth 1 -type f -exec printf '"%s" ' {} +)
echo "$mystring"
Also note that GNU find has a built-in -printf action that supports a variety of format strings, which makes calling the external printf utility unnecessary:
IFS= read -r mystring < <(find "${WEB_DIR}" -mindepth 1 -maxdepth 1 -type f -printf '"%p" ')
echo "$mystring"

I am not really sure why we are bothering with the intermediate step of using a variable:
find "${WEB_DIR}" -mindepth 1 -maxdepth 1 -type f -print0 | xargs -0 /usr/syno/bin/7z \
a "${BACKUP_DIR}/backup_webfiles_${TIMESTAMP}.7z" -xr!thumbs.db -xr!#eaDir -xr!#tmp \
-xr!#recycle -xr!lost+found -xr!.DS_Store -t7z -m0=lzma2 -ms=off -mhe -mmt -mx9 \
-v${SPLIT_VOLUME} -p"${PASSWORD}"
I am not sure if you have to tell 7z that the files are coming from stdin, so you might have to add a hyphen (-) where "${only_files}" used to be??

Related

Find single line files and move them to a subfolder

I am using the following bash line to find text files in a subfolder with a given a pattern inside it and move them to a subfolder:
find originalFolder/ -maxdepth 1 -type f -exec grep -q 'mySpecificPattern' {} \; -exec mv -i {} destinationFolder/ \;
Now instead of grepping a pattern, I would like to move the files to a subfolder if they consist only of a single line (of text): how can I do that?
You can do it this way:
while IFS= read -r -d '' file; do
[[ $(wc -l < "$file") -eq 1 ]] && echo mv -i "$file" destinationFolder/
done < <(find originalFolder/ -maxdepth 1 -type f -print0)
Note use of echo in front of mv so that you can verify output before actually executing mv. Once you're satisfied with output, remove echo before mv.
Using wc as shown above is the most straightforward way, although it reads the entire file to determine the length. It's also possible to do length checks with awk, and the exit function lets you fit that into a find command.
find . -type f -exec awk 'END { exit (NR==1 ? 0 : 1) } NR==2 { exit 1 }' {} \; -print
The command returns status 0 if there has been only 1 input record at end-of-file, and it also exits immediately with status 1 when line 2 is encountered; this should easily outrun wc if large files are a performance concern.

Count filenumber in directory with blank in its name

If you want a breakdown of how many files are in each dir under your current dir:
for i in $(find . -maxdepth 1 -type d) ; do
echo -n $i": " ;
(find $i -type f | wc -l) ;
done
It does not work when the directory name has a blank in the name. Can anyone here tell me how I must edite this shell script so that such directory names also accepted for counting its file contents?
Thanks
Your code suffers from a common issue described in http://mywiki.wooledge.org/BashPitfalls#for_i_in_.24.28ls_.2A.mp3.29.
In your case you could do this instead:
for i in */; do
echo -n "${i%/}: "
find "$i" -type f | wc -l
done
This will work with all types of file names:
find . -maxdepth 1 -type d -exec sh -c 'printf "%s: %i\n" "$1" "$(find "$1" -type f | wc -l)"' Counter {} \;
How it works
find . -maxdepth 1 -type d
This finds the directories just as you were doing
-exec sh -c 'printf "%s: %i\n" "$1" "$(find "$1" -type f | wc -l)"' Counter {} \;
This feeds each directory name to a shell script which counts the files, similarly to what you were doing.
There are some tricks here: Counter {} are passed as arguments to the shell script. Counter becomes $0 (which is only used if the shell script generates an error. find replaces {} with the name of a directory it found and this will be available to the shell script as $1. This is done is a way that is safe for all types of file names.
Note that, wherever $1 is used in the script, it is inside double-quotes. This protects it for word splitting or other unwanted shell expansions.
I found the solution what I have to consider:
Consider_this
#!/bin/bash
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
for i in $(find . -maxdepth 1 -type d); do
echo -n " $i: ";
(find $i -type f | wc -l) ;
done
IFS=$SAVEIFS

How to put custom function's output into find utility as an option?

I want to send a list of extensions as a parameter. So I wrote a small helper function that parses a string of extensions and formats it in a way that "find" utility expects:
exts="txt log";
track=0;
function ext_parse()
{
for i in $exts; do
if [[ $track -eq 0 ]]
then
varstr="-iname \"*.$i\"";
track=1;
else
varstr="$varstr -o -iname \"*.$i\" ";
fi
done
echo "$varstr";
}
So it returns:
-iname "*.txt" -o -iname "*.log"
If I put this into "find" directly it works well:
find . -type f \( -iname "*.txt" -o -iname "*.log" \) -print
But any attempt to substitute this string with the function above that I've tried fails.
Is there any way to obtain that behavior or it is impossible by design?
I would argue it's cleaner, safer and easier to use arrays:
ext_parse() {
local i
varstr=()
for i; do
((${#varstr[#]}!=0)) && varstr+=( -o )
varstr+=( -iname "*.$i" )
done
}
To use this function, you would first call it with appropriate arguments, e.g., ext_parse txt log; this will set the array varstr; and then you can use it as:
find -type f \( "${varstr[#]}" \)
So your workflow looks like this:
$ ext_parse txt log
$ find -type f \( "${varstr[#]}" \)
RE: your comment: to clarify your worry about find being run once per element in array (which is wrong!), do the following test: save the following script as banana:
#!/bin/bash
for ((i=1;i<=$#;++i)); do
printf 'Argument %d: %s\n' "$i" "${!i}"
done
Then chmod +x banana, and try it:
$ ext_parse txt log
$ ./banana -type f \( "${varstr[#]}" \)
Argument 1: -type
Argument 2: f
Argument 3: (
Argument 4: -iname
Argument 5: *.txt
Argument 6: -o
Argument 7: -iname
Argument 8: *.log
Argument 9: )
So you can see that banana is executed once only, with all the arguments given above: exactly what you want!

copy list of filenames in a textfile in bash

I need to copy a list of filenames in a textfile. Trying by this:
#!/bin/sh
mjdstart=55133
mjdend=56674
datadir=/nfs/m/ir1/ssc/evt
hz="h"
for mjd in $(seq $mjdstart $mjdend); do
find $datadir/ssc"${hz}"_allcl_mjd"${mjd}".evt -maxdepth 1 -type f -printf $datadir'/%f\n' > ssc"${hz}".list
done
I tried also:
find $datadir/ssc"${hz}"_allcl_mjd"${mjd}".evt -maxdepth 1 -type f -printf $datadir'/%f\n' | split -l999 -d - ssc"${hz}".list
Or other combinations, but clearly I am missing something: the textfile is empty. Where is my mistake?
Use >> (append) instead of > (overwrite) otherwise you will have output of last command only:
> ssc"${hz}".list
for mjd in $(seq $mjdstart $mjdend); do
find $datadir/ssc"${hz}"_allcl_mjd"${mjd}".evt -maxdepth 1 -type f -printf $datadir'/%f\n' >> ssc"${hz}".list
done
You don't need to use find here, as you simply have a range of specific file names whose existence you are checking for:
#!/bin/sh
mjdstart=55133
mjdend=56674
datadir=/nfs/m/ir1/ssc/evt
hz="h"
for mjd in $(seq $mjdstart $mjdend); do
fnname="$datadir/ssc${hz}_allcl_mjd${mjd}.evt"
[[ -f $fname ]] && printf "$fname\n"
done > "ssc$hz.list"
You are using find wrong. The first argument is the directory, in which it should search. Also, using > overwrites your list file in every turn. Use >> to concatenate:
find $datadir -maxdepth 1 -type f -name "src${hz}_allcl_mjd${mjd}.evt" >> ssc"${hz}".list

reducing commands with sed

I'm just interested if it's possible to reduce this commands to one line without &&?
find /backup/daily.1/var/www/ -iname "*.jpg" -type f >> ~/backuppath.txt
sed 's|/backup/daily.1||g' ~/backuppath.txt > ~/wwwpath.txt
paste -d " " ~/backuppath.txt ~/wwwpath.txt > ~/files.txt
while read line; do cp $line; done < ~/files.txt
I wouldn't write it on one line, but you can do without the intermediate files:
find /backup/daily.1/var/www/ -iname "*.jpg" -type f |
sed 's%/backup/daily.1\(.*\)%cp & \1%' |
sh -x
The sed command splits the file names into two components, the /backup/daily.1 prefix and 'the rest', and replaces that with the complete copy command copying the original name to the name without the prefix. The output of sed is fed to the shell as a script.
This should work fine unless there's a file name that contains shell metacharacters, spaces or newlines. You can improve the resiliency if there won't be newlines or single quotes in the file names with:
find /backup/daily.1/var/www/ -iname "*.jpg" -type f |
sed "s%/backup/daily.1\(.*\)%cp '&' '\1'%" |
sh -x
This wraps each filename in single quotes.
This does not deal with filenames with spaces. (This is not important, I merely
state this to preempt the inevitable comments.)
find /backup/daily.1/var/www/ -iname "*.jpg" -type f |
while read name; do cp $name ${name#/backup/daily.1}; done
You can also just do:
find /backup/daily.1/var/www/ -iname "*.jpg" \
-type f -exec sh -c 'cp "$0" "${0#/backup/daily.1}"' {} \;
which handles unusual filename well.
find /backup/daily.1/var/www/ -iname "*.jpg" -type f \
| sed 's|^/backup/daily\.1\(.*\)$|\0 \1|' \
| ( while read origin dest; do cp "$origin" "$dest"; done)
In the sed expression :
\0 is replaced by the matched string, which is the whole line is this case
\1 is replaced by the subpattern match \(.*\), that is everything from after /backup/daily.1 up to the end of the line

Resources