I have a makefile as below . It checks if the gzip file is empty , if not it must called the test_recipe else print out a message . But in the below case , it calls the test_recipe anyway. Not sure what im doing wrong or what needs to be done to check the gzip content .
SHELL := /bin/bash
current_month = $(shell date +"%m")
current_year = $(shell date +"%Y")
current_date = $(shell date +"%Y-%m-%d")
check_file_dump_empty :
# [ -s test_data.txt.gz ] && $(MAKE) test_recipe || echo "file is empty"
test_recipe :
#bash sqoop.sh ${current_date} ${current_year}
Thanks in advance .
A gz file which exists but is empty will not be zero bytes in size. Your check is wrong. You'll want to decompress it and check whether there is any output.
Here's a refactoring which also avoids overriding the SHELL variable for no good reason.
current_month = $(shell date +"%m")
current_year = $(shell date +"%Y")
current_date = ${current_year}-${current_month}
check_file_dump_empty:
gzip -dc test_data.txt.gz | grep -q '^' \
&& $(MAKE) test_recipe \
|| echo "file is empty"
test_recipe:
bash sqoop.sh "${current_date}" "${current_year}"
Instead of explicitly calling bash, making the script file executable and ensuring that it has a correct shebang on the first line is usually preferable.
Related
I have the following code from a make file, I know this creates bin folder in Home if that doesn't exist... but I couldn't understand what $$HOME/bin mean...
I googled and found $$ is to get the processid of the bash... but couldn't understand what $$HOME/bin mean... can someone please explain ?
.PHONY: home_bin
home_bin: ## Create home bin if not created
# if [[ ! -d "$$HOME/bin" ]]; then \
echo "Creating $$HOME/bin"; \
mkdir $$HOME/bin; \
echo "✔︎ $$HOME/bin created"; \
else \
echo "✔︎ $$HOME/bin already created"; \
fi
Thank you.
make itself performs expansion of $-prefixed characters; the $$ is expanded to a single literal $ to pass to the shell.
Consider a simple Makefile:
x=f
all:
xoo=3 && echo $xoo
which will output foo, because
make expands $x to the single character f.
make passes the string xoo=3 && echo foo to the shell for execution
Compare with
x=f
all:
xoo=3 && echo $$xoo
which outputs 3, because
make expands $$ to $
make passes the string xoo=3 && echo $xoo to the shell for execution
On Ubuntu 16.10, I'm trying to express the following statement in a makefile:
if file exists and is younger than 5 minutes do X else nothing
Currently, I have part of the solution, I believe.
compile_service:
ifeq ($(wildcard $(FILES)/$(CLASS).java)),"") #If file does not exist then $(wildcard file) will evaluate to an empty string.
else
ifeq ???
CLASSPATH=$(RT) javac $(FILES)/$(CLASS).java
$(CMD) $(FILES)/$(CLASS).class
endif
endif
However, I have issues with checking the age and also not sure if this is a good way to do it.
The purpose is that I don't recompile a Java file if it hasn't been modified in X amount of time.
Edit:
I feel like I am almost there. I'm trying something like this:
compile_service:
ifneq (echo $(($(date +%s)/60)), echo $(($(stat -c %Y $(FILES)/$(CLASS).class)/60)))
CLASSPATH=$(RT) javac $(FILES)/$(CLASS).java
$(CMD) $(FILES)/$(CLASS).class
else
$(info Makefile: Recent file exists, using that instead)
endif
But this always fails (the message is printed).
Any clue what I'm doing wrong? Executing those commands in the terminal prints a good time that is accurate to the minute.
I would solve your problem from different approach. I think the test should be part of shell-command instead of part of make-specific commands. For example:
compile_service:
find YOURFILE -type f -Btime +5m -maxdepth 0 2>/dev/null && YOUR_DESIRED_COMMAND
Firstly, you could make a "5 minute file" and use it as a target for the
file, but this solution seems simpler:
FILE = myfile
NOW = $$( date +%s )
FILE_MTIME = $$( date -r $(FILE) +%s 2>/dev/null || echo 0 )
# FILE_AGE = $$( echo $$(( ($(NOW) - $(FILE_MTIME))/60 )) )# # age in minutes
FILE_AGE = $$(( ($(NOW) - $(FILE_MTIME))/60 ))# # age in minutes
THRESHOLD = 5# # cutoff age in minutes
all:
#echo -n "| $(FILE) "
#test -f $(FILE) && echo "is $(FILE_AGE)m old" || echo "NOT FOUND"
#test $(FILE_AGE) -ge $(THRESHOLD) && echo do nothing || make X
X:
#echo .. do X recipe ...
Testing:
$ make all
| myfile NOT FOUND
do nothing
$ touch --date='4 min ago 55 sec ago' myfile
$ make all
| myfile is 4m old
.. do X recipe ...
$ make all
| myfile is 5m old
do nothing
in C-shell I need to check if a file exists or if it is older than another file (or in this example older than 5 seconds from the beginning of unix time). if the file does not exist or is old, some stuff should be executed.
In my example "bla.txt" does not exist, so the first condition is true
if ( ! -f bla.txt || `stat -c "%Y" bla.txt` > 5 ) echo 1
stat: cannot stat `bla.txt': No such file or directory
1
Problem is, if I combine these conditions in an if statement, the second one (age of file) is executed although the first one is already true and gives an error because the file is not there.
in bash, everything works as it should
if [ ! -f bla.txt ] || [ `stat -c "%Y" bla.txt` > 5 ]; then echo 1; fi
1
any ideas on how to achieve this behaviour in csh WITHOUT an else if? I don't want to have the commands to execute twice in my code.
thanks!
CSH has a parser which, to be honest, doesn't deserve the name.
The issue in this particular instance is that it doesn't evaluate the left side of the || construct first before starting stat (as you've seen). As you're depending on the standard output of stat you can't redirect output via >& /dev/null either, and redirection of just stderr is a bit of a nuisance (see Redirecting stderr in csh).
If you want clean csh code that is still understandable but do not want to code the actual code call twice, I think the cleanest solution is to use an intermediate variable. Something like this:
#!/bin/csh
set f=$1
set do=0
if ( ! -f $f ) then
set do=1
else
if ( `stat -c "%Y" $f >& /dev/null ` < 5 ) set do=1
endif
if ( $do ) echo "File $f does not exist or is older than 5s after epoch"
(Note that your original code also had the age test reversed from your prose.)
You can move the -f test inside the shell command from which you are redirecting the output of stat. Here is a script to illustrate:
#!/bin/csh
set verbose
set XX=/tmp/foo
set YY=2
rm -f $XX
foreach ZZ ( 0 1 )
if ( ` stat -c "%Y" $XX` > $YY ) echo 1
if ( ` test -f $XX && stat -c "%Y" $XX` > $YY ) echo 1
if ( $ZZ == 0 ) touch $XX
stat -c "%Y" $XX
sleep $YY
end
Basically my HDD crashed, I was able to recover all the files, but, all the files have retained their meta & some have retained their names, I have 274000 images, which I need to more or less, sort into folders by date.
So let's say it starts with the first files, it would get the date from the file, create a sub folder, and until the date changes, keep moving that file into the created folder, once the date changes, it would create a new folder and keep doing the same thing.
I'm sure this is possible, I really didn't want to have to do this manually as it would take weeks...
Lets say I have a target folder /target/
Target contains, 274000 files, in no sub folders at all.
The folders structure should be /target/YY/DD_MM/filenames
I would like to create a bash script for this, but I'm not really sure where to proceed from here.
I've found this:
#!/bin/bash
DIR=/home/data
target=$DIR
cd "$DIR"
for file in *; do
dname="$( date -d "${file%-*}" "+$target/%Y/%b_%m" )"
mkdir -vp "${dname%/*}"
mv -vt "$dname" "$file"
done
Would creating a folder without checking if it exists delete files inside that folder?
I'm also not quite sure what adding an asterix to the dir pathname would do?
I'm not quite familiar with bash, but I'd love to get this working if someone could please explain to me a little more what's going on?
Thankyou!
I seemed to have found an answer that suited me, this worked on OSX just fine on three files, before I run it on the massive folder, can you guys just check that this isn't going to fail somewhere?
#!/bin/bash
DIR=/Users/limeworks/Downloads/target
target=$DIR
cd "$DIR"
for file in *; do
# Top tear folder name
year=$(stat -f "%Sm" -t "%Y" $file)
# Secondary folder name
subfolderName=$(stat -f "%Sm" -t "%d-%m-%Y" $file)
if [ ! -d "$target/$year" ]; then
mkdir "$target/$year"
echo "starting new year: $year"
fi
if [ ! -d "$target/$year/$subfolderName" ]; then
mkdir "$target/$year/$subfolderName"
echo "starting new day & month folder: $subfolderName"
fi
echo "moving file $file"
mv "$file" "$target/$year/$subfolderName"
done
I've had issues with the performance of the other solutions since my filesystem is remotely mounted and access times are big.
I've worked on some improved solutions in bash and python:
Bash version:
record # cat test.sh
for each in *.mkv
do
date=$(date +%Y-%d-%m -r "$each");
_DATES+=($date);
FILES+=($each);
done
DATES=$(printf "%s\n" "${_DATES[#]}" | sort -u);
for date in ${DATES[#]}; do
if [ ! -d "$date" ]; then
mkdir "$date"
fi
done
for i in ${FILES[#]}; do
dest=$(date +%Y-%d-%m -r "$i")
mv $i $dest/$i
done
record # time bash test.sh
real 0m3.785s
record #
Python version:
import os, datetime, errno, argparse, sys
def create_file_list(CWD):
""" takes string as path, returns tuple(files,date) """
files_with_mtime = []
for filename in [f for f in os.listdir(CWD) if os.path.splitext(f)[1] in ext]:
files_with_mtime.append((filename,datetime.datetime.fromtimestamp(os.stat(filename).st_mtime).strftime('%Y-%m-%d')))
return files_with_mtime
def create_directories(files):
""" takes tuple(file,date) from create_file_list() """
m = []
for i in files:
m.append(i[1])
for i in set(m):
try:
os.makedirs(os.path.join(CWD,i))
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
def move_files_to_folders(files):
""" gets tuple(file,date) from create_file_list() """
for i in files:
try:
os.rename(os.path.join(CWD,i[0]), os.path.join(CWD,(i[1] + '/' + i[0])))
except Exception as e:
raise
return len(files)
if __name__ == '__main__':
parser = argparse.ArgumentParser(prog=sys.argv[0], usage='%(prog)s [options]')
parser.add_argument("-e","--extension",action='append',help="File extensions to match",required=True)
args = parser.parse_args()
ext = ['.' + e for e in args.extension]
print "Moving files with extensions:", ext
CWD = os.getcwd()
files = create_file_list(CWD)
create_directories(files)
print "Moved %i files" % move_files_to_folders(files)
record # time python sort.py -e mkv
Moving files with extensions: ['.mkv']
Moved 319 files
real 0m1.543s
record #
Both scripts are tested upon 319 mkv files modified in the last 3 days.
I worked on a little script and tested it.Hope this helps.
#!/bin/bash
pwd=`pwd`
#list all files,cut date, remove duplicate, already sorted by ls.
dates=`ls -l --time-style=long-iso|grep -e '^-.*'|awk '{print $6}'|uniq`
#for loop to find all files modified on each unique date and copy them to your pwd
for date in $dates; do
if [ ! -d "$date" ]; then
mkdir "$date"
fi
#find command will find all files modified at particular dates and ignore hidden files.
forward_date=`date -d "$date + 1 day" +%F`
find "$pwd" -maxdepth 1 -not -path '*/\.*' -type f -newermt "$date" ! -newermt "$forward_date" -exec cp -f {} "$pwd/$date" \;
done
You must be in your working directory where your files to be copied according to date are present.
I want to create a multiline file with Make, having exact content:
#!/bin/bash
if [ "$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$( cat <<EOF
-Dmapred.job.tracker=$JT
EOF
)
${HADOOP_HOME}/bin/hadoop $1 $HADOOP_CONFIG_VARS ${*:2} 2>&1 | grep -v SLF4J
How can I tell make to output a file with this exact content somewhere?
I tried this:
define SCRIPT_CONTENT
#!/bin/bash
if [ "$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$( cat <<EOF
-Dmapred.job.tracker=$JT
EOF
)
${HADOOP_HOME}/bin/hadoop $1 $HADOOP_CONFIG_VARS ${*:2} 2>&1 | grep -v SLF4J
endef
export SCRIPT_CONTENT
bin/script:
#echo "$$SCRIPT_CONTENT" > bin/script
This paricular solution 1) wipes $ and first char after $-es and 2) is ugly because the definition should happen outside of the particular target where it's needed :(
I also tried this:
bin/script:
#echo '
#!/bin/bash
if [ "$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$( cat <<EOF
-Dmapred.job.tracker=$JT
EOF
)
${HADOOP_HOME}/bin/hadoop $1 $HADOOP_CONFIG_VARS ${*:2} 2>&1 | grep -v SLF4J
' > bin/script
This returns error when in make, works outside of make...
Any suggestion is very welcome!
Make wants any $ characters that should be reproduced literally to be escaped by inserting another $ in front of them.
More broadly, though, it seems like you're trying to use Make as a shell-script replacement. The more idomatic way to do this would be to put that content in a source file that you can copy to the destination, or to put it in a script that will write it into a specified destination. The Makefile then just has to invoke the copy command or the script.
With the help from this magnificent answer, I cooked up the following.
# From https://stackoverflow.com/a/8316519/874188
define \n
endef
define SCRIPT_CONTENT
#!/bin/bash
if [ "$$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$$( cat <<EOF
-Dmapred.job.tracker=$$JT
EOF
)
$${HADOOP_HOME}/bin/hadoop $$1 $$HADOOP_CONFIG_VARS $${*:2} 2>&1 | grep -v SLF4J
endef
bin/script:
echo '$(subst $(\n),\n,$(SCRIPT_CONTENT))' >$#
When testing, I found that I needed to have a semicolon at the end of the echo line when it didn't have any redirection. I can speculate that there is a built-in echo which gets invoked when there are no shell metacharacters in the command line?
Also, notice that the definition cannot contain any single quotes, and that all dollar signs have to be doubled. Maybe one or the other of these restrictions could be removed; I was unsuccessful, but then I didn't spend too much time or effort.
You cannot do this in make. Beyond what Novelocrat says regarding $, there's the fact that make is line-oriented and does not have any ability to generate a command that contains a newline character. All newlines that appear unescaped (without a backslash before them) are parsed by make as ending that recipe line, and each recipe line is sent to a different invocation of the shell. If you want the entire command to be sent as a single string to the same shell, then you must escape the newlines.
However, make will remove all backslash/newline pairs before it runs the command.
The only possible way to do this completely within make is to generate the file one line at a time, like this:
bin/script:
#echo '#!/bin/bash' > $#
#echo 'if [ "$$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi' >> $#
#echo 'export CONFIG_VARS=$$( cat <<EOF' >> $#
#echo ' -Dmapred.job.tracker=$$JT' >> $#
#echo 'EOF' >> $#
#echo ')' >> $#
#echo '$${HADOOP_HOME}/bin/hadoop $$1 $$HADOOP_CONFIG_VARS $${*:2} 2>&1 | grep -v SLF4J' >> $#
As Novelocrat says, the typical way this is done is to have the script file as a separate file and copy it where you want it, rather than generating it.