I am running the following command:
OLDIFS=$IFS
IFS=$'\n'
for i in $(find $HOME/test -maxdepth 1 -type f); do
if [ $? -eq 0 ]; then
telegram-upload --to 12345 --directories recursive --large-files split --caption '{file_name}' $i &&
rm $i
fi
done
IFS=$OLDIFS
If the telegram upload command exits with a non zero code I intend to do the following:
rm somefile && wget someurl && rerun the whole command
How do I go about doing something like this?
I believe you can capture the exit code as follows:
exit_code=$(telegram-upload --to 12345 ...)
From here, you can use the variable $exit_code as a regular variable,
like if [ $exit_code -eq 0 ]; then).
It's straightforward: Do an endless loop, which you break out once you succeed:
for i in $(find $HOME/test -maxdepth 1 -type f)
do
while true
do
if telegram-upload --to 12345 --directories recursive --large-files split --caption '{file_name}' $i
then
break
else
rm somefile && wget someurl
fi
done
done
UPDATE : For completeness, I added the loop over the files as well. Note that your approach of looping over the files will fail, if you have files where the name contains white space. However this is already a problem in your own approach and not part of the question, so I don't discuss this here.
Related
I am writing a shell script to check two things at one time. The first condition is to check for the existence of a specific file and the second condition is to confirm that there is only one file in that directory.
I am using the following code:
conf_file=ls -1 /opt/files/conf.json 2>/dev/null | wc -l
total_file=ls -1 /opt/files/* 2>/dev/null| wc -l
if [ $conf_file -eq 1 ] && [ $total_file -eq 1 ]
then
echo "done"
else
echo "Not Done"
fi
It is returning the following error
0
0
./ifexist.sh: 4: [: -eq: unexpected operator
Not Done
I am probably doing a very silly mistake. Can anyone help me a little bit?
One of the reasons you should normally not parse ls is that you can get strange results when you have files with newlines. In your case that won't be an issue, because any file different from json.conf should make the test fail. However you should make the code counting the files be future-proof. You can use find for this.
Your code can be changed into
jsonfile="/opt/files/conf.json"
countfiles=$(find /opt/files -maxdepth 1 -type f -exec printf '.\n' \; | wc -l)
if [[ -f "${jsonfile}" ]] && (( "${countfiles}" == 1)); then
echo "Done"
else
echo "Not Done"
fi
When you say this:
conf_file=ls -1 /opt/files/conf.json 2>/dev/null | wc -l
That assigns the value "ls" to the variable conf_file, and then tries to run a command called "-1" and pipe the result to wc If you want to run a pipe sequence, you have to enclose it in $( ):
conf_file=$(ls -1 /opt/files/conf.json 2./dev/null | wc -l)
Next, when combining clauses in the test command ([), do it inside the command:
if [ $conf_file -eq 1 -a $total_file -eq 1 ]
However, there are better ways to do this. You can check if a file exists with "-f", and you can just check whether the output of ls matches what you expect, without creating variables or running other commands:
if [ -f /opt/files/conf.json -a "$(ls /opt/files/conf.*)" -eq "/opt/files/conf.json" ]
However, it is not a friendly practice to prohibit other files. In many cases, people might want to leave backup or test copies (conf.json.bak or conf.json.test), and there's no reason for you to block that.
I'm writing unit-tests to test file-IO functions.
There's no formalized test-framework in my target language, so my idea is to run a little test program that somehow manipulates files in a test-directory, and after that checks the results in a little shell script.
To evaluate the output, I want to check a given directory whether all expected files are there and no other files have been created during the test.
My first attempt goes like this:
set -e
test -e "${test_dir}/a.txt"
test -e "${test_dir}/b.txt"
test -d "${test_dir}/dir"
find "${test_dir}" -mindepth 1 \
-not -wholename "${test_dir}/a.txt" \
-not -wholename "${test_dir}/b.txt" \
-not -wholename "${test_dir}/dir" \
| grep . && exit 1 || true
This properly detects whether there are two files a.txt and b.txt, and a subdirectory dir/ in the ${test_dir}.
If there happens to be a file c.txt, the test should and will fail.
However, this doesn't scale well.
There are dozens of unit-tests and each has a different set of files/directories, so I find myself repeating lines very similar to the above again and again.
So I'd rather wrap the above into a function call like so:
if checkdirectory "${test_dir}" a.txt b.txt dir/ dir/subdir/ dir/.hidden.txt; then
echo "ok"
else
echo "ko"
fi
Unfortunately I have no clue how to implement checkdirectory (esp. the find invocation with multiple -not -wholename ... stanzas gives me headache).
To add a bit of fun, the constraints are:
support both (and differentiate between) files and directories
must (EDITed from should) run on Linux, macOS & MSYS2/MinGW, therefore:
POSIX if possible (in reality it will be bash, but probably bash<<4! so no fancy features)
EDIT
some more constraints (these didn't make it into original my late-night question; so just consider them "extra constraints for bonus points")
the test-directory may contain subdirectories and files in subdirectories (up to an arbitrary depth), so any check needs to operate on more than just the top-level directory
ideally, the paths may contain weirdo characters like spaces, linebreaks,... (this is really unit-testing. we do want to test for such cases)
the testdir is more often than not some randomly generated directory using mktemp -d, so it would be nice if we could avoid hardcoding it in the tests
no assumptions about the underlying filesystem can be made.
Assuming we have a directory tree as an example:
$test_dir/a.txt
$test_dir/b.txt
$test_dir/dir/c.txt
$test_dir/dir/"d e".txt
$test_dir/dir/subdir/
then would you please try:
#!/bin/sh
checkdirectory() {
local i
local count
local testdir=$1
shift
for i in "$#"; do
case "$i" in
*/) [ -d "$testdir/$i" ] || return 1 ;; # check if the directory exists
*) [ -f "$testdir/$i" ] || return 1 ;; # check if the file exists
esac
done
# convert each filename to just a newline, then count the lines
count=`find "$testdir" -mindepth 1 -printf "\n" | wc -l`
[ "$count" -eq "$#" ] || return 1
return 0
}
if checkdirectory "$test_dir" a.txt b.txt dir/ dir/c.txt "dir/d e.txt" dir/subdir/; then
echo "ok"
else
echo "ko"
fi
One easy fast way would be to compare the output of find with a reference string:
Lets start with an expected directory and files structure:
d/FolderA/filexx.csv
d/FolderA/filexx.doc
d/FolderA/Sub1
d/FolderA/Sub2
testassert
#!/usr/bin/env bash
assertDirContent() {
read -r -d '' s < <(find "$1" -printf '%y %p\n')
[ "$2" = "$s" ]
}
testref='d d/FolderA/
f d/FolderA/filexx.csv
f d/FolderA/filexx.doc
d d/FolderA/Sub1
d d/FolderA/Sub2'
if assertDirContent 'd/FolderA/' "$testref"; then
echo 'ok'
else
echo 'Directory content assertion failed'
fi
Testing it:
$ ./testassert
ok
$ touch d/FolderA/unwantedfile
$ ./testassert
Directory content assertion failed
$ rm d/FolderA/unwantedfile
$ ./testassert
ok
$ rmdir d/FolderA/Sub1
$ ./testassert
Directory content assertion failed
$ mkdir d/FolderA/Sub1
$ ./testassert
ok
$ rmdir d/FolderA/Sub2
# Replace with a file instead of a directory
touch d/FolderA/Sub2
$ ./testassert
Directory content assertion failed
Now if you add timestamps and other info like permissions, owner, group to the find -printf output, you can also check all these matches the asserted string output.
I don't know what you mean by differentiating between files and directories since your last if statement is somehow binary. Here's what worked for me:
#! /bin/bash
function checkdirectory()
{
test_dir="${1}"
shift
content="$#"
for file in ${content}
do
[[ -z "${test_dir}/${file}" ]] && return 1
done
# -I is meant to be appended to "ls" to ignore the files in order to check if other files exist.
matched=" -I ${content// / -I } -I ${test_dir}"
[[ -e `ls $matched` ]] && return 1
return 0
}
if checkdirectory /some/directory a.txt b.txt dir; then
echo "ok"
else
echo "ko"
fi
Here's a possible solution i dreamed up during the night.
It destroys the test-data, so might not be usable in many cases (though it might just work for paths generated on-the-fly during unit tests):
checkdirectory() {
local i
local testdir=$1
shift
# try to remove all the listed files
for i in "$#"; do
if [ "x${i}" = "x${i%/}" ]; then
rm "${testdir}/${i}" || return 1
fi
done
# the directories should now be empty,
# so try to remove those dirs that are listed
for i in "$#"; do
if [ "x${i}" != "x${i%/}" ]; then
rmdir "${testdir}/${i}" || return 1
fi
done
# finally ensure that no files are left
if find "${testdir}" -mindepth 1 | grep . >/dev/null ; then
return 1
fi
return 0
}
When invoking the checkdirectory function, deeper directories must come first (that is checkdirectory foo/bar/ foo/ rather than checkdirectory foo/ foo/bar/).
I am trying to write a script that will check if a directory contains only
a specific kind of file (and/or folder) and will return 1 for false, 0 for true.
IE: I want to check if /my/dir/ contains only *.gz files and nothing else.
This is what i have so far, but it doesn't seem to be working as intended:
# Basic vars
readonly THIS_JOB=${0##*/}
readonly ARGS_NBR=1
declare dir_in=$1
dir_in=$1"/*.gz"
#echo $dir_in
files=$(shopt -s nullglob dotglob; echo ! $dir_in)
echo $files
if (( ${#files} ))
then
echo "Success: Directory contains files."
exit 0
else
echo "Failure: Directory is empty (or does not exist or is a file)"
exit 1
fi
I want to check if /my/dir/ contains only *.gz files and nothing else.
Use find instead of globulation. It's really easier to use find and to parse find output. Globulation are simple for simple scripts, but once you want to parse "all files in a directory" and do some filtration and such, it's way easier (and safer) to use find:
find "$1" -mindepth 1 -maxdepth 1 \! -name '*.gz' -o \! -type f | wc -l | xargs test 0 -eq
This finds all "things" that are not named *.gz inside the directory or are not files (so mkdir a.gz is accounted for), counts them, and then tests if they're count is equal to 0. If the count is equal to 0, xargs test 0 -eq will return 0, if not, it will return status between 1 - 125. You can handle the nonzero return status with a simple || return 1 if you wish.
You can remove xargs with a simple bash substitution and use the method from this thread for a little speedup and get test return value, which is 0 or 1:
[ 0 -eq "$(find "$1" -mindepth 1 -maxdepth 1 \! -name '*.gz' -o \! -type f -print '.' | wc -c)" ]
Remember that the exit status of a script is the exit status of the last command executed. So you don't need anything else in your script if you wish, only a shebang and this oneliner will suffice.
Using Bash's extglob, !(*.gz) and grep:
$ if grep -qs . path/!(*.gz) ; then echo yes ; else echo nope ; fi
man grep:
-q, --quiet, --silent
Quiet; do not write anything to standard output. Exit
immediately with zero status if any match is found, even if an
error was detected. Also see the -s or --no-messages option.
-s, --no-messages
Suppress error messages about nonexistent or unreadable files.
Since you are using bash, there is another setting you can use: GLOBIGNORE
#!/bin/bash
containsonly(){
dir="$1"
glob="$2"
if [ ! -d "$dir" ]; then
echo 1>&2 "Failure: directory does not exist"
return 2
fi
local res=$(
cd "$dir"
GLOBIGNORE=$glob"
shopt -s nullglob dotglob
echo *
)
if [ ${#res} = 0 ]; then
echo 1>&2 "Success: directory contains no extra files"
return 0
else
echo 1>&2 "Failure: directory contains extra files"
return 1
fi
}
# ...
containsonly myfolder '*.gz'
Some have suggested to count all files which do not match the globbing pattern *.gz. This might be quite inefficient depending on the the number of files. For you job it is sufficient to find just one file, which does not match your globbing pattern. Use the -quite action of find to exit after the first match:
if [ -z "$(find /usr/share/man/man1/* -not -name '*.gz' -print -quit)" ]
then echo only gz
fi
As an example, I have 7 directories each containing 4 files. The 4 files follow the following naming convention name_S#_L001_R1_001.fastq.gz. The sed command is to partially keep the unique file name.
I have a nested for loop in order to enter a directory and perform a command, exit the directory and proceed to the next directory. Everything seems to be working beautifully, however the code gets stuck on the last directory looping 4 times.
for f in /completepath/*
do
[ -d $f ] && cd "$f" && echo Entering into $f
for y in `ls *.fastq.gz | sed 's/_L00[1234]_R1_001.fastq.gz//g' | sort -u`
do
echo ${y}
done
done
Example output-
Entering into /completepath/m_i_cast_avpv_1
iavpvcast1_S6
Entering into /completepath/m_i_cast_avpv_2
iavpvcast2_S6
Entering into /completepath/m_i_int_avpv_1
iavpvint1_S5
Entering into /completepath/m_i_int_avpv_2
iavpvint2_S5
Entering into /completepath/m_p_cast_avpv_1
pavpvcast1_S8
Entering into /completepathd/m_p_int_avpv_1
pavpvint1_S7
Entering into /completepath/m_p_int_avpv_2
pavpvint2_S7
pavpvint2_S7
pavpvint2_S7
pavpvint2_S7
Any recommendations of how to correctly exit the inner loop?
It looks like /completepath/ contains some entries that are not directories. When the loop over /completepath/* sees something that's not a directory, it doesn't enter it, thanks to the [ -d $f ] check.
But it still continues to run the next for y in ... loop.
At that point the script is still in the previous directory it has seen.
One way to solve that is to skip the rest of the loop when $f is not a directory:
if [ -d $f ]; then
cd "$f" && echo Entering into $f
else
continue
fi
There's an even better way. By writing /completepath/*/ only directory entries will be matched, so you can simplify your loop to this:
for f in /completepath/*/
do
cd "$f" && echo "Entering into $f" || { echo "Error: could not enter into $f"; continue; }
for y in $(ls *.fastq.gz | sed 's/_L00[1234]_R1_001.fastq.gz//g' | sort -u)
do
echo ${y}
done
done
I am trying to parallel the task of rpw_gen_features in the following bash script:
#!/bin/bash
maxjobs=8
jobcounter=0
MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
DIR=$1
FILES=`find $MYDIR/${DIR}/${DIR}\_*.hpl -name *.hpl -type f -printf "%f\n" | sort -n -t _ -k 2`
for f in $FILES; do
fileToProcess=$MYDIR/${DIR}/$f
# construct .pfl file name
filebasename="${f%.*}"
fileToCheck=$MYDIR/${DIR}/$filebasename.pfl
# check if the .pfl file is already generated
if [ ! -f $fileToCheck ];
then
echo ../bin/rpw_gen_features -r $fileToProcess &
jobcounter=jobcounter+1
fi
if [jobcounter -eq maxjobs]
wait
jobcounter=0
fi
done
but it generates some error at runtime:
line 20: syntax error near unexpected token `fi'
I'm not an expert in bash programming, so please feel free to comment on the whole code.
I am curious why you don't just use GNU Parallel:
MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
DIR=$1
find $MYDIR/${DIR}/${DIR}\_*.hpl -name *.hpl -type f |
parallel '[ ! -f {.}.pfl ] && echo ../bin/rpw_gen_features -r {}'
Or even:
MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
parallel '[ ! -f {.}.pfl ] && echo ../bin/rpw_gen_features -r {}' ::: $MYDIR/$1/$1\_*.hpl
It seems to be way more readable, and it will automatically scale when you move from an 8-core to a 64-core machine.
Watch the intro video for a quick introduction:
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). You command line
with love you for it.
You are missing a then, spaces and ${} around the variables:
if [jobcounter -eq maxjobs]
wait
jobcounter=0
fi
Should be
if [ ${jobcounter} -eq ${maxjobs} ]; then
wait
jobcounter=0
fi
Further, you need to double check your script as I can see many missing ${} for example:
jobcounter=jobcounter+1
Even if you use the variables correctly this still will not work:
jobcounter=${jobcounter}+1
Will yield:
1
1+1
1+1+1
And not what you expect. You need to use:
jobcounter=`expr $jobcounter + 1`
With never versions of BASH you should be able to do:
(( jobcounter++ ))