Filter Hidden Files with Bash (for Batch Image Resize Script) - bash

I'm writing a script to batch resize images. Originally I was applying an operation for file in $(ls $1), but I would like to be able to use globbing, so I'm looking at something more like for file in $(echo $1). The problem is that dotglob may or may not be enabled, so echo * could return hidden files (notably, .DS_Store), which cause convert to throw an error and stop the script. I would like the default behavior of the command to be that if I cd into a directory full of images and execute resize * 400x400 jpg, all of the images will be resized excluding hidden files, regardless of whether dotglob is enabled.
So, in pseudo code, I'm looking for:
for file in $(echo $1 | [filter-hidden-files])
Here is my script with the older behavior. Will update with new behavior when I find a solution:
# !/bin/bash
# resize [folder] [sizeXxsizeY] [outputformat]
# if [outputformat] is omitted, the input file format is assumed
for file in $(ls $1)
do
IMGNAME=$(echo "$file" | cut -d'.' -f1)
if test -z $3
then
EXTENSION=$(echo "$file" | cut -d'.' -f2)
convert $1/$file -resize $2 -quality 100 $1/$IMGNAME-$2.$EXTENSION
echo "$file => $IMGNAME-$2.$EXTENSION"
else
convert $1/$file -resize $2 -quality 100 $1/$IMGNAME-$2.$3
echo "$file => $IMGNAME-$2.$3"
fi
done
Here is the current script:
# !/bin/bash
# resize [pattern] [sizeXxsizeY] [outputformat]
# if [outputformat] is omitted, the input file format is assumed
for file in $(echo $1)
do
IMGNAME=$(echo "$file" | cut -d'.' -f1)
if test -z $3 && if test -f $3
then
EXTENSION=$(echo "$file" | cut -d'.' -f2)
convert $file -resize $2 -quality 100 $IMGNAME-$2.$EXTENSION
echo "$file => $IMGNAME-$2.$EXTENSION"
else
convert $file -resize $2 -quality 100 $IMGNAME-$2.$3
echo "$file => $IMGNAME-$2.$3"
fi
done
Given the command resize * 400x400, convert throws an error as it cannot process .DS_Store (a hidden file residing in every file on an OSX system). As I will never be processing hidden images, I would like to automatically filter them. I've been trying to do this with grep or find, but I haven't figured it out yet.
New script goes here:
for file in $(echo $1)
do

I would suggest changing the commandline of your script from resize * 400x400 to resize 400x400 *, the script would be more like the standard unix tools that way (grep, sed, ln, etc.). You would not have to unset the dotglob option in your script (which is better since it's up to the user of the script if he wants hidden files globbed or not).
Your script would look something like this:
#!/bin/bash
OUTPUTFORMAT=$1
# Remove original $1 from the list of arguments
shift
for i in "$#"
do
# Use $OUTPUTFORMAT here
etc....
If you do not want to change the commandline for your script. You could try setting GLOBIGNORE
export GLOBIGNORE=".*"
Or if extglob is set you could try file globbing like so:
echo !(.*)

There is a dotglob shell option that decides if files starting with . are included when globbing. You can check if this is the case with
shopt dotglob
You also can explicitly disable it in your script:
#!/bin/bash
shopt -u dotglob
for file in $1/*; do
...
done

there's no need to use ls with a for loop, most of the time its useless. also the for loop with * doesn't return hidden files, unless you specifically specify it. To show hidden files,
for files in .*
do
echo $files
done

as you are getting a list of files in $* you can check them one by one
for i in $*
do
expr $i : '^\..*' > /dev/null && continue
# process file
done

Related

Associative array, file names refering to the path, for dmenu

And I started playing with dmenu and it seems such an automation for almost every thing. Unfortunately I'm not familiar with bash and it should be on my list.
I have a folder for my markdowns with subfolders containing my files. I'm trying to have a script to show them in dmenu while using an alias.
If the path to a file is
/home/user/docs/markdown/practice01/rmd/network.rmd
I would like to have
network
as an option in my dmenu. So when I choose
network -----> /home/user/docs/markdown/practice01/rmd/network.rmd
Here is my broken script. There are a few things I'm missing.
This way I get full path on my dmenu which i don't need. I tried to read about associative arrays but I can't figure it out in bash.
This script works but in case I decide to ESC and exit, still it opens up an empty vim in my directory. Hence, I should know if statements huh!
#!/bin/bash
DMenu=("dmenu -l 10 -i -nb "#eaeaea" -sb "#E53935" -nf "#474747"")
cd ~/docs/markdown/
target=$(find -type f -name '*.rmd' | $DMenu)
st vim "$target"
I made a little example. But the problem is that it is a manual work to add each file, which definitely we don't wanna do right!
#!/bin/bash
declare -A dotfiles
dotfiles[i3]="/home/user/dotfiles/i3/.config/i3/config"
dotfiles[vimrc]="/home/user/dotfiles/vim/.vimrc"
list=("i3\nvimrc")
target=$(echo -e $list | dmenu -i -nb "#eaeaea" -sb "#E53935" -nf "#474747")
st vim "${dotfiles["$target"]}"
Thank you
Associative arrays can be weird... but returning output to a variable makes it easier to manipulate as any other string in bash, as shown in the example below:
prefix="$HOME/git/notes"
suffix=".md"
shopt -s nullglob globstar
item=( "$prefix"/**/*${suffix}) # Search *.md in all dirs/subdirs
item=( "${item[#]#"$prefix"/}" )
item=( "${item[#]%${suffix}}" ) # Removes '.md' string from item name
result=$(printf '%s\n' "${item[#]}" | dmenu)
[[ -n $result ]] || exit # exit if nothing is found
gedit "${prefix}/${result}.md" # Open file by adding again '.md'
When the percent sign (%) is used in the pattern ${variable%substring}, it will return content of the variable with the shortest occurrence of substring deleted from the back of the variable.
Listed below for reference are 2 examples I wrote, one in Bash and the other in Python, for managing pass and markdown notes with dmenu:
dmenu-pass.sh
dmenu-launch.py
Also, listed below are a couple nice articles that might help you out:
The weird, wondrous world of Bash arrays
Advanced Bash-Scripting Guide: Manipulating Strings
Instead of putting some code in an array, use a function!
my_dmenu() {
dmenu -l 10 -i -nb "#eaeaea" -sb "#e53935" -nf "#474747"
}
If your markdown files are all in the same folder (and not in subfolders), you certainly don't need find: use a glob instead! and if your files are in subfolders, use a glob instead (with the globstar shell option).
All in all:
#!/bin/bash
my_dmenu() {
dmenu -l 10 -i -nb "#eaeaea" -sb "#e53935" -nf "#474747"
}
base_dir=~/docs/markdown
# Also, check the return code of cd!
cd "$base_dir" || { echo >&2 "Can't cd to $base_dir. Exiting"; exit 1; }
# Using a glob: use the shell option nullglob
shopt -s nullglob
files=( *.rmd )
# Check that there are some files found:
if (( ${#files[#]} == 0 )); then
echo "No files found. Exiting."
exit 1
fi
# Now we're ready to send the files to dmenu:
chosen_file=$(printf '%s\n' "${files[#]}" | my_dmenu)
# If dmenu returns nothing: don't launch vim!
if [[ ! $chosen_file ]]; then
echo "No files selected. Exiting."
exit 1
fi
# Now you can launch vim!
st vim "$chosen_file"
If you also want to find the *.rmd files in subfolders: use instead:
shopt -s nullglob globstar
files=( **/*.rmd )
Edit to address the requirement in your comment (and the edit of your question):
If you want to strip the .rmd suffix to show in dmenu, use:
chosen_file=$(printf '%s\n' "${files[#]%.rmd}" | my_dmenu)
# ...
st vim "$chosen_file.rmd"
The expansion ${files[#]%.rmd} will strip the suffix .rmd from each field of the array files. Don't forget to add this suffix back when you edit the file (as shown in the last line).
dmenuoptions="-l 10 -i -nb '#eaeaea' -sb '#E53935' -nf '#474747'"
st -e vim $(find ~/docs/markdown -type f -name '*.rmd' | dmenu $dmenuoptions)

zgrep tar.gz file with file location results & match [duplicate]

Am trying to grep pattern from dozen files .tar.gz but its very slow
am using
tar -ztf file.tar.gz | while read FILENAME
do
if tar -zxf file.tar.gz "$FILENAME" -O | grep "string" > /dev/null
then
echo "$FILENAME contains string"
fi
done
If you have zgrep you can use
zgrep -a string file.tar.gz
You can use the --to-command option to pipe files to an arbitrary script. Using this you can process the archive in a single pass (and without a temporary file). See also this question, and the manual.
Armed with the above information, you could try something like:
$ tar xf file.tar.gz --to-command "awk '/bar/ { print ENVIRON[\"TAR_FILENAME\"]; exit }'"
bfe2/.bferc
bfe2/CHANGELOG
bfe2/README.bferc
I know this question is 4 years old, but I have a couple different options:
Option 1: Using tar --to-command grep
The following line will look in example.tgz for PATTERN. This is similar to #Jester's example, but I couldn't get his pattern matching to work.
tar xzf example.tgz --to-command 'grep --label="$TAR_FILENAME" -H PATTERN ; true'
Option 2: Using tar -tzf
The second option is using tar -tzf to list the files, then go through them with grep. You can create a function to use it over and over:
targrep () {
for i in $(tar -tzf "$1"); do
results=$(tar -Oxzf "$1" "$i" | grep --label="$i" -H "$2")
echo "$results"
done
}
Usage:
targrep example.tar.gz "pattern"
Both the below options work well.
$ zgrep -ai 'CDF_FEED' FeedService.log.1.05-31-2019-150003.tar.gz | more
2019-05-30 19:20:14.568 ERROR 281 --- [http-nio-8007-exec-360] DrupalFeedService : CDF_FEED_SERVICE::CLASSIFICATION_ERROR:408: Classification failed even after maximum retries for url : abcd.html
$ zcat FeedService.log.1.05-31-2019-150003.tar.gz | grep -ai 'CDF_FEED'
2019-05-30 19:20:14.568 ERROR 281 --- [http-nio-8007-exec-360] DrupalFeedService : CDF_FEED_SERVICE::CLASSIFICATION_ERROR:408: Classification failed even after maximum retries for url : abcd.html
If this is really slow, I suspect you're dealing with a large archive file. It's going to uncompress it once to extract the file list, and then uncompress it N times--where N is the number of files in the archive--for the grep. In addition to all the uncompressing, it's going to have to scan a fair bit into the archive each time to extract each file. One of tar's biggest drawbacks is that there is no table of contents at the beginning. There's no efficient way to get information about all the files in the archive and only read that portion of the file. It essentially has to read all of the file up to the thing you're extracting every time; it can't just jump to a filename's location right away.
The easiest thing you can do to speed this up would be to uncompress the file first (gunzip file.tar.gz) and then work on the .tar file. That might help enough by itself. It's still going to loop through the entire archive N times, though.
If you really want this to be efficient, your only option is to completely extract everything in the archive before processing it. Since your problem is speed, I suspect this is a giant file that you don't want to extract first, but if you can, this will speed things up a lot:
tar zxf file.tar.gz
for f in hopefullySomeSubdir/*; do
grep -l "string" $f
done
Note that grep -l prints the name of any matching file, quits after the first match, and is silent if there's no match. That alone will speed up the grepping portion of your command, so even if you don't have the space to extract the entire archive, grep -l will help. If the files are huge, it will help a lot.
For starters, you could start more than one process:
tar -ztf file.tar.gz | while read FILENAME
do
(if tar -zxf file.tar.gz "$FILENAME" -O | grep -l "string"
then
echo "$FILENAME contains string"
fi) &
done
The ( ... ) & creates a new detached (read: the parent shell does not wait for the child)
process.
After that, you should optimize the extracting of your archive. The read is no problem,
as the OS should have cached the file access already. However, tar needs to unpack
the archive every time the loop runs, which can be slow. Unpacking the archive once
and iterating over the result may help here:
local tempPath=`tempfile`
mkdir $tempPath && tar -zxf file.tar.gz -C $tempPath &&
find $tempPath -type f | while read FILENAME
do
(if grep -l "string" "$FILENAME"
then
echo "$FILENAME contains string"
fi) &
done && rm -r $tempPath
find is used here, to get a list of files in the target directory of tar, which we're iterating over, for each file searching for a string.
Edit: Use grep -l to speed up things, as Jim pointed out. From man grep:
-l, --files-with-matches
Suppress normal output; instead print the name of each input file from which output would
normally have been printed. The scanning will stop on the first match. (-l is specified
by POSIX.)
Am trying to grep pattern from dozen files .tar.gz but its very slow
tar -ztf file.tar.gz | while read FILENAME
do
if tar -zxf file.tar.gz "$FILENAME" -O | grep "string" > /dev/null
then
echo "$FILENAME contains string"
fi
done
That's actually very easy with ugrep option -z:
-z, --decompress
Decompress files to search, when compressed. Archives (.cpio,
.pax, .tar, and .zip) and compressed archives (e.g. .taz, .tgz,
.tpz, .tbz, .tbz2, .tb2, .tz2, .tlz, and .txz) are searched and
matching pathnames of files in archives are output in braces. If
-g, -O, -M, or -t is specified, searches files within archives
whose name matches globs, matches file name extensions, matches
file signature magic bytes, or matches file types, respectively.
Supported compression formats: gzip (.gz), compress (.Z), zip,
bzip2 (requires suffix .bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2),
lzma and xz (requires suffix .lzma, .tlz, .xz, .txz).
Which requires just one command to search file.tar.gz as follows:
ugrep -z "string" file.tar.gz
This greps each of the archived files to display matches. Archived filenames are shown in braces to distinguish them from ordinary filenames. For example:
$ ugrep -z "Hello" archive.tgz
{Hello.bat}:echo "Hello World!"
Binary file archive.tgz{Hello.class} matches
{Hello.java}:public class Hello // prints a Hello World! greeting
{Hello.java}: { System.out.println("Hello World!");
{Hello.pdf}:(Hello)
{Hello.sh}:echo "Hello World!"
{Hello.txt}:Hello
If you just want the file names, use option -l (--files-with-matches) and customize the filename output with option --format="%z%~" to get rid of the braces:
$ ugrep -z Hello -l --format="%z%~" archive.tgz
Hello.bat
Hello.class
Hello.java
Hello.pdf
Hello.sh
Hello.txt
All of the code above was really helpful, but none of it quite answered my own need: grep all *.tar.gz files in the current directory to find a pattern that is specified as an argument in a reusable script to output:
The name of both the archive file and the extracted file
The line number where the pattern was found
The contents of the matching line
It's what I was really hoping that zgrep could do for me and it just can't.
Here's my solution:
pattern=$1
for f in *.tar.gz; do
echo "$f:"
tar -xzf "$f" --to-command 'grep --label="`basename $TAR_FILENAME`" -Hin '"$pattern ; true";
done
You can also replace the tar line with the following if you'd like to test that all variables are expanding properly with a basic echo statement:
tar -xzf "$f" --to-command 'echo "f:`basename $TAR_FILENAME` s:'"$pattern\""
Let me explain what's going on. Hopefully, the for loop and the echo of the archive filename in question is obvious.
tar -xzf: x extract, z filter through gzip, f based on the following archive file...
"$f": The archive file provided by the for loop (such as what you'd get by doing an ls) in double-quotes to allow the variable to expand and ensure that the script is not broken by any file names with spaces, etc.
--to-command: Pass the output of the tar command to another command rather than actually extracting files to the filesystem. Everything after this specifies what the command is (grep) and what arguments we're passing to that command.
Let's break that part down by itself, since it's the "secret sauce" here.
'grep --label="`basename $TAR_FILENAME`" -Hin '"$pattern ; true"
First, we use a single-quote to start this chunk so that the executed sub-command (basename $TAR_FILENAME) is not immediately expanded/resolved. More on that in a moment.
grep: The command to be run on the (not actually) extracted files
--label=: The label to prepend the results, the value of which is enclosed in double-quotes because we do want to have the grep command resolve the $TAR_FILENAME environment variable passed in by the tar command.
basename $TAR_FILENAME: Runs as a command (surrounded by backticks) and removes directory path and outputs only the name of the file
-Hin: H Display filename (provided by the label), i Case insensitive search, n Display line number of match
Then we "end" the first part of the command string with a single quote and start up the next part with a double quote so that the $pattern, passed in as the first argument, can be resolved.
Realizing which quotes I needed to use where was the part that tripped me up the longest. Hopefully, this all makes sense to you and helps someone else out. Also, I hope I can find this in a year when I need it again (and I've forgotten about the script I made for it already!)
And it's been a bit a couple of weeks since I wrote the above and it's still super useful... but it wasn't quite good enough as files have piled up and searching for things has gotten more messy. I needed a way to limit what I looked at by the date of the file (only looking at more recent files). So here's that code. Hopefully it's fairly self-explanatory.
if [ -z "$1" ]; then
echo "Look within all tar.gz files for a string pattern, optionally only in recent files"
echo "Usage: targrep <string to search for> [start date]"
fi
pattern=$1
startdatein=$2
startdate=$(date -d "$startdatein" +%s)
for f in *.tar.gz; do
filedate=$(date -r "$f" +%s)
if [[ -z "$startdatein" ]] || [[ $filedate -ge $startdate ]]; then
echo "$f:"
tar -xzf "$f" --to-command 'grep --label="`basename $TAR_FILENAME`" -Hin '"$pattern ; true"
fi
done
And I can't stop tweaking this thing. I added an argument to filter by the name of the output files in the tar file. Wildcards work, too.
Usage:
targrep.sh [-d <start date>] [-f <filename to include>] <string to search for>
Example:
targrep.sh -d "1/1/2019" -f "*vehicle_models.csv" ford
while getopts "d:f:" opt; do
case $opt in
d) startdatein=$OPTARG;;
f) targetfile=$OPTARG;;
esac
done
shift "$((OPTIND-1))" # Discard options and bring forward remaining arguments
pattern=$1
echo "Searching for: $pattern"
if [[ -n $targetfile ]]; then
echo "in filenames: $targetfile"
fi
startdate=$(date -d "$startdatein" +%s)
for f in *.tar.gz; do
filedate=$(date -r "$f" +%s)
if [[ -z "$startdatein" ]] || [[ $filedate -ge $startdate ]]; then
echo "$f:"
if [[ -z "$targetfile" ]]; then
tar -xzf "$f" --to-command 'grep --label="`basename $TAR_FILENAME`" -Hin '"$pattern ; true"
else
tar -xzf "$f" --no-anchored "$targetfile" --to-command 'grep --label="`basename $TAR_FILENAME`" -Hin '"$pattern ; true"
fi
fi
done
zgrep works fine for me, only if all files inside is plain text.
it looks nothing works if the tgz file contains gzip files.
You can mount the TAR archive with ratarmount and then simply search for the pattern in the mounted view:
pip install --user ratarmount
ratarmount large-archive.tar mountpoint
grep -r '<pattern>' mountpoint/
This is much faster than iterating over each file and piping it to grep separately, especially for compressed TARs. Here are benchmark results in seconds for a 55 MiB uncompressed and 42 MiB compressed TAR archive containing 40 files:
Compression
Ratarmount
Bash Loop over tar -O
none
0.31 +- 0.01
0.55 +- 0.02
gzip
1.1 +- 0.1
13.5 +- 0.1
bzip2
1.2 +- 0.1
97.8 +- 0.2
Of course, these results are highly dependent on the archive size and how many files the archive contains. These test examples are pretty small because I didn't want to wait too long. But, they already exemplify the problem well enough. The more files there are, the longer it takes for tar -O to jump to the correct file. And for compressed archives, it will be quadratically slower the larger the archive size is because everything before the requested file has to be decompressed and each file is requested separately. Both of these problems are solved by ratarmount.
This is the code for benchmarking:
function checkFilesWithRatarmount()
{
local pattern=$1
local archive=$2
ratarmount "$archive" "$archive.mountpoint"
'grep' -r -l "$pattern" "$archive.mountpoint/"
}
function checkEachFileViaStdOut()
{
local pattern=$1
local archive=$2
tar --list --file "$archive" | while read -r file; do
if tar -x --file "$archive" -O -- "$file" | grep -q "$pattern"; then
echo "Found pattern in: $file"
fi
done
}
function createSampleTar()
{
for i in $( seq 40 ); do
head -c $(( 1024 * 1024 )) /dev/urandom | base64 > $i.dat
done
tar -czf "$1" [0-9]*.dat
}
createSampleTar myarchive.tar.gz
time checkEachFileViaStdOut ABCD myarchive.tar.gz
time checkFilesWithRatarmount ABCD myarchive.tar.gz
sleep 0.5s
fusermount -u myarchive.tar.gz.mountpoint
In my case the tarballs have a lot of tiny files and I want to know what archived file inside the tarball matches. zgrep is fast (less than one second) but doesn't provide the info I want, and tar --to-command grep is much, much slower (many minutes)1.
So I went the other direction and had zgrep tell me the byte offsets of the matches in the tarball and put that together with the list of offsets in the tarball of all archived files to find the matching archived files.
#!/bin/bash
set -e
set -o pipefail
function tar_offsets() {
# Get the byte offsets of all the files in a given tarball
# based on https://stackoverflow.com/a/49865044/60422
[ $# -eq 1 ]
tar -tvf "$1" -R | awk '
BEGIN{
getline;
f=$8;
s=$5;
}
{
offset = int($2) * 512 - and((s+511), compl(512)+1)
print offset,s,f;
f=$8;
s=$5;
}'
}
function tar_byte_offsets_to_files() {
[ $# -eq 1 ]
# Convert the search results of a tarball with byte offsets
# to search results with archived file name and offset, using
# the provided tar_offsets output (single pass, suitable for
# process substitution)
offsets_file="$1"
prev_offset=0
prev_offset_filename=""
IFS=' ' read -r last_offset last_len last_offset_filename < "$offsets_file"
while IFS=':' read -r search_result_offset match_text
do
while [ $last_offset -lt $search_result_offset ]; do
prev_offset=$last_offset
prev_offset_filename="$last_offset_filename"
IFS=' ' read -r last_offset last_len last_offset_filename < "$offsets_file"
# offsets increasing safeguard
[ $prev_offset -le $last_offset ]
done
# now last offset is the first file strictly after search result offset so prev offset is
# the one at or before it, and must be the one it is in
result_file_offset=$(( $search_result_offset - $prev_offset ))
echo "$prev_offset_filename:$result_file_offset:$match_text"
done
}
# Putting it together e.g.
zgrep -a --byte-offset "your search here" some.tgz | tar_byte_offsets_to_files <(tar_offsets some.tgz)
1 I'm running this in Git for Windows' minimal MSYS2 fork unixy environment, so it's possible that the launch overhead of grep is much much higher than on any kind of real Unix machine and would make `tar --to-command grep` good enough there; benchmark solutions for your own needs and platform situation before selecting.

bash call script with variable

What I want to achieve is the following :
I want the subtitles for my TV Show downloaded automatically.
The script "getSubtitle.sh" is ran as soon as the show is downloaded, but it can happen that no subtitle are released yet.
So what I am doing to counter this :
Creating a file each time "getSubtitle.sh" is ran. It contain the location of the script with its arguments, for example :
/Users/theo/logSubtitle/getSubtitle.sh "The Walking Dead - 5x10 - Them.mp4" "The.Walking.Dead.S05E10.480p.HDTV.H264.mp4" "/Volumes/Window HD/Série/The Walking Dead"
If a subtitle has been found, this file will contain only this line, if no subtitle has been found, this file will have 2 lines (the first one being "no subtitle downloaded", and the second one being the path to the script as explained above)
Now, once I get this, I'm planning to run a cron everyday that will do the following :
Remove all file that have only 1 line (Subtitle found), and execute the script again for the remaining file. Here is the full script :
cd ~/logSubtitle/waiting/
for f in *
do nbligne=$(wc -l $f | cut -c 8)
if [ "$nbligne" = "1" ]
then
rm $f
else
command=$(sed -n "2 p" $f)
sh $command 3>&1 1>&2 2>&3 | grep down > $f ; echo $command >> $f
fi
done
This is unfortunately not working, I have the feeling that the script is not called.
When I replace $command by the line in the text file, it is working.
I am sure that $command match the line because of the "echo $command >> $f" at the end of my script.
So I really don't get what I am missing here, any ideas ?
Thanks.
I'm not sure what you're trying to achieve with the cut -c 8 part in wc -l $f | cut -c 8. cut -c 8 will select the 8th character of the output of wc -l.
A suggestion: to check whether your file contains 1 or two lines (and since you'll need the content of the second line, if any, anyway), use mapfile. This will slurp the file in an array, one line per field. You can use the option -n 2 to read at most 2 lines. This will be much more efficient, safe and nice than your solution:
mapfile -t -n 2 ary < file
Then:
if ((${#ary[#]}==1)); then
printf 'File contains one line only: %s\n' "${ary[0]}"
elif ((${#ary[#]==2)); then
printf 'File contains (at least) two lines:\n'
printf ' %s\n' "${ary[#]}"
else
printf >&2 'Error, no lines found in file\n'
fi
Another suggestion: use more quotes!
With this, a better way to write your script:
#!/bin/bash
dir=$HOME/logSubtitle/waiting/
shopt -s nullglob
for f in "$dir"/*; do
mapfile -t -n 2 ary < "$f"
if ((${#ary[#]}==1)); then
rm -- "$f" || printf >&2 "Error, can't remove file %s\n" "$f"
elif ((${#ary[#]}==2)); then
{ sh -c "${ary[1]}" 3>&1 1>&2 2>&3 | grep down; echo "${ary[1]}"; } > "$f"
else
printf >&2 'Error, file %s contains no lines\n' "$f"
fi
done
After the done keyword you can even add the redirection 2>> logfile to a log file if you wish. Make sure the cron job is run with your user: check crontab -l and, if needed, edit it with crontab -e.
Use eval instead of sh. The reason it works with eval and not sh is due to the number of passes to evaluate variables. sh will treat the sed command as its command to execute while eval will evaluate the sed command first and then execute the result.
Briefly explained.

Need help writing bash script to move folders around

What i need to do is replace the folder amtlib.framework into each Adobe app on my mac
if i do:
cd /Applications; ls | grep Adobe, this gives me all the folders which i need
here's some pseudo code:
apps = ls | grep Adobe
for each x in apps
if (x/x.app/contents/frameworks/amtlib.framwork) //if this folder exists
add .bak extension //amtlib.framework.bak
copy ~/Downloads/.../amtlib.framwork to x/x.app/contents/frameworks/
how would i implement this as a bash script?
Something like
for x in $( ls | grep Adobe) ; do
if [[ -d "${x}"/"${x}".app/contents/frameworks/amtlib.framwork ]] ; then
# add .bak extension # //amtlib.framework.bak
#? mkdir "${x}"/"${x}".app/contents/frameworks/amtlib.framwork.bak
#? /bin/mv "${x}"/"${x}".app/contents/frameworks/amtlib.framwork {x}/${x}.app/contents/frameworks/amtlib.framwork.bak
/bin/cp ~/Downloads/.../amtlib.framwork to "${x}"/"${x}".app/contents/frameworks/
else
: # ??? what do you want to do if there's not
fi
done # loop
If you're likely to have spaces in your dirnames, (not sure if OSX support -print0), but try
find . -name 'Adobe' -print0 \
| while read x ; do
if ....
As an FYI, assignments in bash are done like (without spaces around the =):
apps=$(ls | grep Adobe)
Depending on the situation then, you'll want to use "$apps", or just plain $apps, which leaves each word in the list as a separate token. (If there are spaces in your filename or path, 1 path/file is now 2 words, and will cause issues). There are also array notations to use, apps=( $(ls | grep Adobe) ), and using those vars like ${#apps[#]} (number of elems), ${apps[#]} (all elems), ${apps[1]}, (first elem) is possible.
Also, it's not clear what your intent with add .bak extension is for. My best guess is my 2nd option, /bin/mv ... .bak.
IHTH.
First of all there's a typo error in the original post that's made its way throughout the examples given. The folder you are looking to rename/replace is amtlib.framework, not framwork.
Second, for some reason, the test for existence of the .bak directory is not working for me, even when I split this out to a separate if-then statement it doesn't work:
cd /Applications
for x in *Adobe* ; do
printf "$x \n"
printf "%s" " "
if [ -d "$x/$x.app/contents/frameworks/amtlib.framework.bak" ]; then
printf "removing old bak... "
fi
if [ -d "$x/$x.app/contents/frameworks/amtlib.framework" ]; then
printf "moving... "
printf "copying... "
printf "%s\n" "done!"
else
printf "%s\n" "nothing to do here!"
fi
done
cd ~
Finally, understanding the goal you will fail to update a couple of apps that have an additional folder level (e.g., Acrobat Pro and Illustrator).

How can I check the size of a file using Bash?

I've got a script that checks for 0-size, but I thought there must be an easier way to check for file sizes instead. I.e. file.txt is normally 100 kB; how can I make a script check if it is less than 90 kB (including 0), and make it Wget a new copy because the file is corrupt in this case?
What I'm currently using...
if [ -n file.txt ]
then
echo "everything is good"
else
mail -s "file.txt size is zero, please fix. " myemail#gmail.com < /dev/null
# Grab wget as a fallback
wget -c https://www.server.org/file.txt -P /root/tmp --output-document=/root/tmp/file.txt
mv -f /root/tmp/file.txt /var/www/file.txt
fi
[ -n file.txt ] doesn't check its size. It checks that the string file.txt is non-zero length, so it will always succeed.
If you want to say "size is non-zero", you need [ -s file.txt ].
To get a file's size, you can use wc -c to get the size (file length) in bytes:
file=file.txt
minimumsize=90000
actualsize=$(wc -c <"$file")
if [ $actualsize -ge $minimumsize ]; then
echo size is over $minimumsize bytes
else
echo size is under $minimumsize bytes
fi
In this case, it sounds like that's what you want.
But FYI, if you want to know how much disk space the file is using, you could use du -k to get the size (disk space used) in kilobytes:
file=file.txt
minimumsize=90
actualsize=$(du -k "$file" | cut -f 1)
if [ $actualsize -ge $minimumsize ]; then
echo size is over $minimumsize kilobytes
else
echo size is under $minimumsize kilobytes
fi
If you need more control over the output format, you can also look at stat. On Linux, you'd start with something like stat -c '%s' file.txt, and on BSD and Mac OS X, something like stat -f '%z' file.txt.
stat can also check the file size. Some methods are definitely better: using -s to find out whether the file is empty or not is easier than anything else if that's all you want. And if you want to find files of a size, then find is certainly the way to go.
I also like du a lot to get file size in kb, but, for bytes, I'd use stat:
size=$(stat -f%z $filename) # BSD stat
size=$(stat -c%s $filename) # GNU stat?
An alternative solution with AWK and double parenthesis:
FILENAME=file.txt
SIZE=$(du -sb $FILENAME | awk '{ print $1 }')
if ((SIZE<90000)) ; then
echo "less";
else
echo "not less";
fi
If your find handles this syntax, you can use it:
find -maxdepth 1 -name "file.txt" -size -90k
This will output file.txt to stdout if and only if the size of file.txt is less than 90k. To execute a script script if file.txt has a size less than 90k:
find -maxdepth 1 -name "file.txt" -size -90k -exec script \;
If you are looking for just the size of a file:
cat $file | wc -c
Sample output:
203233
This works in both Linux and macOS:
function filesize
{
local file=$1
size=`stat -c%s $file 2>/dev/null` # Linux
if [ $? -eq 0 ]
then
echo $size
return 0
fi
eval $(stat -s $file) # macOS
if [ $? -eq 0 ]
then
echo $st_size
return 0
fi
return -1
}
Use:
python -c 'import os; print (os.path.getsize("... filename ..."))'
It is portable, for all flavours of Python, and it avoids variation in stat dialects.
For getting the file size in both Linux and Mac OS X (and presumably other BSD systems), there are not many options, and most of the ones suggested here will only work on one system.
Given f=/path/to/your/file,
what does work in both Linux and Mac's Bash:
size=$( perl -e 'print -s shift' "$f" )
or
size=$( wc -c "$f" | awk '{print $1}' )
The other answers work fine in Linux, but not in Mac:
du doesn't have a -b option in Mac, and the BLOCKSIZE=1 trick doesn't work ("minimum blocksize is 512", which leads to a wrong result)
cut -d' ' -f1 doesn't work because on Mac, the number may be right-aligned, padded with spaces in front.
So if you need something flexible, it's either perl's -s operator , or wc -c piped to awk '{print $1}' (awk will ignore the leading white space).
And of course, regarding the rest of your original question, use the -lt (or -gt) operator:
if [ $size -lt $your_wanted_size ]; then, etc.
Based on gniourf_gniourf’s answer,
find "file.txt" -size -90k
will write file.txt to stdout if and only if the size of file.txt is less than 90K, and
find "file.txt" -size -90k -exec command \;
will execute the command command if file.txt has a size less than 90K. 
I have tested this on Linux. 
From find(1),
…  Command-line arguments following (the -H, -L and -P options) are taken to be names of files or directories to be examined, up to the first argument that begins with ‘-’, …
(emphasis added).
ls -l $file | awk '{print $6}'
assuming that ls command reports filesize at column #6
I would use du's --threshold for this. Not sure if this option is available in all versions of du but it is implemented in GNU's version.
Quoting from du(1)'s manual:
-t, --threshold=SIZE
exclude entries smaller than SIZE if positive, or entries greater
than SIZE if negative
Here's my solution, using du --threshold= for OP's use case:
THRESHOLD=90k
if [[ -z "$(du --threshold=${THRESHOLD} file.txt)" ]]; then
mail -s "file.txt size is below ${THRESHOLD}, please fix. " myemail#gmail.com < /dev/null
mv -f /root/tmp/file.txt /var/www/file.txt
fi
The advantage of that, is that du can accept an argument to that option in a known format - either human as in 10K, 10MiB or what ever you feel comfortable with - you don't need to manually convert between formats / units since du handles that.
For reference, here's the explanation on this SIZE argument from the man page:
The SIZE argument is an integer and optional unit (example: 10K is
10*1024). Units are K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (powers
of 1000). Binary prefixes can be used, too: KiB=K, MiB=M, and so on.
Okay, if you're on a Mac, do this:
stat -f %z "/Users/Example/config.log"
That's it!

Resources