I have been struggling like for two hours to figure out the issue regarding this script of mine. When I used it statically without any variable it fetches the grep results, but when I put them with those variables, i keep receiving error and no results. I believe there is something wrong with the special character escape which I can not handle.
I have the file FLAGS_IN with this structure :
automotive_susan_s dataset1 -funsafe-math-optimizations -fno-guess-branch-probability -fno-ivopts -fno-inline-functions -fno-omit-frame-pointer -fselective-scheduling -fno-inline-small-functions -fno-tree-pre -ftracer -fno-move-loop-invariants
that have the flags for i in AppName and the dataset$j as structured above. Could anyone help me figure out what is wrong with this part of my sh script?
GCC_OPT="-O3"
OPT_FLAGS=$("grep $i\ dataset$j\ $FLAGS_IN|sed\ s/$i\ dataset$j//g")
echo $GCC_OPT
echo $OPT_FLAGS
echo "found the validated flags, they are \n $GCC_OPT $OPT_FLAGS"
make -f Makefile.gcc -j4 CCC_OPTS="$GCC_OPT\ $OPT_FLAG"
You're a little overzealous with your quoting. Also, it's a little easier to use cut here than sed.
OPT_FLAGS=$(grep "$i dataset$j" FLAGS_IN | cut -d " " -f3-)
and
make -f Makefile.gcc -j4 CCC_OPTS="$GCC_OPT $OPT_FLAG"
Is this what you're trying to do:
$ cat file
foo
automotive_susan_s dataset1 -funsafe-math-optimizations ...
bar
$ i=automotive_susan_s
$ j=1
$ sed -n "s/$i dataset$j//p" file
-funsafe-math-optimizations ...
Related
I had a makefile I used to create jobs in one step then submit them to a cluster in another, then finally merge them. This worked well and I had no issues with it, I was using a python script to break csv files down into smaller ones in the create-jobs step however, and it took longer than I'd like so I decided to add a step to do this with bash code instead, thinking it would be faster.
I laid the new step out in the Makefile in the exact same way as the others, and created a small bash script that would accept the name of the csv, how many parts to split it into and eventually I was going to add a variable for the location to place the output csves but I tried testing it with it just sticking them in the same directory as itself first. I first defined the variables that would later be passed on in the bash file so that I could check it worked by itself, it does. Those variables are commented out now but this is the bash file in question:
#!/bin/bash
#INPUT_DATA="/scratch/cb27g11/Looping_Bash/missing.csv"
#nJobs=2
tail -n +2 ${INPUT_DATA} | split -l ${nJobs} - split_
for file in split_*
do
head -n 1 ${INPUT_DATA} > tmp_file
cat $file >> tmp_file
mv -f tmp_file ../$file
done
Nothing super complicated.
The Makefile looks like this:
########################
### --- Split jobs - ###
########################
SPLIT_JOB_NAME = "Name_for_job"
SPLIT_JOB_INPUT_DATA = "/scratch/cb27g11/Looping_Bash/missing.csv"
SPLIT_JOB_HEADER = "header/default.header"
SPLIT_JOB_nJobs = 2
#######################
DECLARE_ROOT_DIR = ROOT_DIR="${THDM_T3PS_SCANNER_DIR}/job_submission/MadGraph/"
VAR_CREATE_JOB = $(shell echo '$(.VARIABLES)' | awk -v RS=' ' '/CREATE_JOB_/' | sed 's/CREATE_JOB_//g' )
EXPORT_CREATE_JOB = $(foreach v,$(VAR_CREATE_JOB),$(v)="$(CREATE_JOB_$(v))") $(DECLARE_ROOT_DIR)
VAR_SUBMIT_JOB = $(shell echo '$(.VARIABLES)' | awk -v RS=' ' '/SUBMIT_JOB_/' | sed 's/SUBMIT_JOB_//g' )
EXPORT_SUBMIT_JOB = $(foreach v,$(VAR_SUBMIT_JOB),$(v)="$(SUBMIT_JOB_$(v))") $(DECLARE_ROOT_DIR)
VAR_MERGE_JOB = $(shell echo '$(.VARIABLES)' | awk -v RS=' ' '/MERGE_JOB_/' | sed 's/MERGE_JOB_//g' )
EXPORT_MERGE_JOB = $(foreach v,$(VAR_MERGE_JOB),$(v)="$(MERGE_JOB_$(v))") $(DECLARE_ROOT_DIR)
VAR_SPLIT_JOB = $(shell echo '$(.VARIABLES)' | awk -v RS=' ' '/SPLIT_JOB_/' | sed 's/SPLIT_JOB_//g' )
EXPORT_SPLIT_JOB = $(foreach v,$(VAR_SPLIT_JOB),$(v)="$(SPLIT_JOB_$(v))") $(DECLARE_ROOT_DIR)
#########################################################
create-jobs:
#$(EXPORT_CREATE_JOB) ./utils/new_create-jobs.sh
submit-jobs:
#$(EXPORT_SUBMIT_JOB) ./utils/new_submit-jobs.sh
merge-jobs:
#$(EXPORT_MERGE_JOB) ./utils/merge-jobs.sh
split-jobs:
#$(EXPORT_SPLIT_JOB) ./utils/split-jobs.sh
I'm not currently using all of the variables set-up for split-jobs here but surely that doesn't matter? The other parts, create-jobs, submit-jobs and merge-jobs work just fine and I can't see that I've made a mistake in how I've set up split-jobs compared to them. So I guess there's something I'm missing about the bash file itself?
When I try to run make split-jobs I get this:
(THDM) [cb27g11#cyan53 MadGraph]$ make split-jobs
/bin/sh: ./utils/split-jobs.sh: Permission denied
make: *** [split-jobs] Error 126
Since this is on a shared cluster I don't have sudo rights or anything like that, could this be related to that somehow? Am I calling something wrong? Any help or suggestions would be much appreciated!
I can't really understand what all the shell commands are trying to do, but I'm completely confident that whatever they are doing there's a much simpler and more straightforward way to do it.
However, that doesn't have anything to do with your error, which is:
/bin/sh: ./utils/split-jobs.sh: Permission denied
Did you remember to give your shell script execute permissions:
chmod +x ./utils/split-jobs.sh
? Also, please remove the # from your makefile lines until AFTER your makefile is completely working, and show us the command line that make invoked (maybe once you see it, you will be able to figure out the problem for yourself). Using # is like trying to debug your program while sending all the output to /dev/null.
So my organization is upgrading our Oracle database from 11g to 19c.
Previously, in my makefile, I had been setting ORACLE_HOME like this:
ORACLE_HOME=/opt/app/oracle/product/11.2.0.4/db_1
However, Oracle 19c has a fun feature that whenever they run a patch on it, the db_1 changes incrementally, becoming db_2, then db_3, with each patch, etc.
So obviously I can't hardcode the ORACLE_HOME path anymore.
In a bunch of my scripts, I'm pulling the current value from the ortab file, like this:
setenv ORACLE_SID DATABASE1
setenv ORACLE_HOME `cat /var/opt/oracle/oratab | sed 's/#.*//g' | grep -w $ORACLE_SID | awk -F: '{print $2;}'`
And this is working just fine, pulling the correct ORACLE_HOME path from the ortab file.
However, when I tried to do this in a makefile, like so:
ORACLE_SID=DATABASE1
ORACLE_HOME=`cat /var/opt/oracle/oratab | sed 's/#.*//g' | grep -w $ORACLE_SID | awk -F: '{print $2;}'`
I get this error when I try to run make:
$ make
`cat /var/opt/oracle/oratab | sed 's//bin/proc sys_include=/usr/include lines=yes iname=file1.pc oname=file1.c include=/path/to/include
First RE may not be null
*** Error code 2
make: Fatal error: Command failed for target `file1.o'
So obviously the command isn't working the way I'm expecting, but I am unsure how to fix it.
How do I fix the command to work inside makefile? I'm running Solaris 11.
This is not GNU make, this is just the default make that comes with Solaris 11.
Adding more information:
My ortab file looks like this:
$cat /var/opt/oracle/oratab
DATABASE_TEST:/opt/app/oracle/product/11.2.0.4/db_7:Y
DATABASE1:/opt/app/oracle/product/19.0.0.0/db_3:N
DATABASE2:/opt/app/oracle/product/11.2.0.4/db_13:Y
DATABASE3:/opt/app/oracle/product/11.2.0.4/db_1:Y
DATABASE_PROD:/opt/app/oracle/product/11.2.0.4/db_2:Y
So, what I need to do, is using the ORACLE_SID of DATABASE1, pull out the /opt/app/oracle/product/19.0.0.0/db_3 part, to use as my ORACLE_HOME directory in the makefile.
Update:
Based on an answer below from MadScientist , this is now my makefile:
ORACLE_SID=DATABASE1
#ORACLE_HOME = /opt/app/oracle/product/19.0.0/db_3
ORACLE_HOME = `cat /var/opt/oracle/oratab | sed 's/\#.*//g' | grep -w ${ORACLE_SID} | awk -F: '{print $$2;}'`
PROC=${ORACLE_HOME}/bin/proc
E_INCLUDE=/path/to/include
print-% : ; #echo $* = $($*)
file1.o: file1.pc
${PROC} sys_include=/usr/include lines=yes iname=$*.pc oname=$*.c include=${E_INCLUDE}
When I hardcode ORACLE_HOME, everything works correctly.
When I try to use the dynamically created ORACLE_HOME, I get this error:
$ make
`cat /var/opt/oracle/oratab | sed 's/\#.*//g' | grep -w DATABASE1 | awk -F: '{print $2;}'`/bin/proc sys_include=/usr/include lines=yes iname=file1.pc oname=file1.c include=/path/to/include
make: Fatal error: Command failed for target `file1.o'
So it looks like it's setting ORACLE_HOME as the command itself, rather than as the result of the command.
Weirdly, when I run make print-ORACLE_HOME, I get the expected result /opt/app/oracle/product/19.0.0/db_3
Well, certainly this:
setenv ORACLE_HOME /opt/app/oracle/product/11.2.0.4/db_1
could not have been in your makefile before because this is not valid makefile syntax. Also, it surprises me that anyone is still using csh for anything, and especially scripting, in 2020. But anyway.
The problem you're having is that makefiles are not shell scripts and the rules of syntax are different. Of course a makefile contains shell scripts inside of it, but only in recipes: here you're setting a makefile variable. So just plopping a shell statement down into a variable assignment very well might not work.
Here you have three problems: first, variable reference in makefiles are of the form $(FOO) or ${FOO} but not $FOO. Second, a # is considered a comment character in a makefile and must be escaped. And finally, if you do want an actual $ not a variable reference you have to escape it, as: $$. Fixing those, this should work but note that there are likely simpler ways to do this:
ORACLE_SID = DATABASE1
ORACLE_HOME = `cat /var/opt/oracle/oratab | sed 's/\#.*//g' | grep -w $(ORACLE_SID) | awk -F: '{print $$2;}'`
You say that after this, this rule:
PROC=${ORACLE_HOME}/bin/proc
file1.o: file1.pc
${PROC} sys_include=/usr/include lines=yes iname=$*.pc oname=$*.c include=${E_INCLUDE}
Gives this output:
$ make
`cat /var/opt/oracle/oratab | sed 's/\#.*//g' | grep -w DATABASE1 | awk -F: '{print $2;}'`/bin/proc sys_include=/usr/include lines=yes iname=file1.pc oname=file1.c include=/path/to/include
make: Fatal error: Command failed for target `file1.o'
That error message is not very helpful and means nothing. It's a shame this doesn't give a better message.
I recommend you change the rule to this:
file1.o: file1.pc
echo PROC=\'${PROC}\'; ${PROC} sys_include=/usr/include lines=yes iname=$*.pc oname=$*.c include=${E_INCLUDE}
then, you should see something like this in the output:
$ make
echo PROC=\'`cat /var/opt/oracle/oratab...lots of stuff...
PROC='/...'
make: Fatal error: Command failed for target `file1.o'
What you want to look at is the second line of the output, PROC='/...' and examine that path /..., whatever it is, to make sure it looks right. Also it should not contain any whitespace or other special characters, etc.
If that value that is printed looks wrong, you'll have to fix your script to make it right. If it looks right, then I have no idea what's going on and it must be something particular about the version of make you're using.
Here's a simplified example to start. This initial version uses awk to search, drops the sed absent any # comments
ORACLE_SID := DATABASE1
ORACLE_HOME := $(shell awk -F: "/^$(ORACLE_SID)/ { print \$$2; }" /var/opt/oracle/oratab)
(Update) possible Solaris version from documents, unverified:
ORACLE_HOME:sh = awk -F: '/^DATABASE1/ { print $$2; }' /var/opt/oracle/oratab
I finally got it to work.
Using #Milag 's answer regarding the :sh command substitution, I was able to find the documentation regarding that for Solaris (not GNU) make: link to documentation
THIS was the answer:
ORACLE_HOME :sh =cat /var/opt/oracle/oratab | sed 's/\#.*//g' | grep -w DATABASE1 | awk -F: '{print $2;}'
The key? NOT putting a double dollar sign, as the documentation said:
"In contrast to commands in rules, the command is not subject for macro substitution; therefore, a dollar sign ($) need not be replaced with a double dollar sign ($$)."
Because of this I also had to hardcode DATABASE1 instead of using the ${ORACLE_SID} variable, but since that value will never change, I can live with that.
Imagine following string:
<tr><td>12,3</td><td>deg</td><td>23,4</td><td>humi</td><td>34,5</td><td>press</td></tr>
In bash, how do I extract 23.4, based on the condition that it is followed by humi?
grep -o works well for this sort of thing. I'm sure performance would be better with a single sed command than two greps but that's rarely a serious concern.
X='<tr><td>12,3</td><td>deg</td><td>23,4</td><td>humi</td><td>34,5</td><td>press</td></tr>'
echo $X | grep -o '[0-9,.]*</td><td>humi' | grep -o '[0-9,.]*'
# Result: 23,4
You can additionally pipe through tr , . to get English number format.
I am running a script in mac that was inherited by my team and it has some replacement code which doesnt seem to be running properly. I tried a variety of subsitute characters like %,\ etc but nothing seems to be working. Can someone give me an idea of what else should I change.
The script is run like this
./test.sh /Users/Apple/Test_Folder/ 425323 C6C7CB 425363 425363 425363 666666 666666 425363 425363 666666 425363 425363 a0a9b1 687582 a0a9b2
The issue inside the script(test.sh) is this for loop
for f in "$1"/*.svg;
do
sed -i '' 's\#000002\#'"$2"'/g;s\#000003\#'"$3"'/g;s\#000004\#'"$4"'/g;s\#000005\#'"$5"'/g;s\#000006\#'"$6"'/g;s\#000007\#'"$7"'/g;s\#000008\#'"$8"'/g;s\#000009\#'"$9"'/g;s\#000010\#'"$10"'/g;s\#000011\#'"$11"'/g;s\#000012\#'"$12"'/g;s\#000013\#'"$13"'/g;s\#000014\#'"$14"'/g;s\#000015\#'"$15"'/g;s\#000016\#'"$16"'/g' $f
done
The error i get is
sed: 1: "s%#000002%#425323/g;s%# ...": bad flag in substitute command: '#'
Any pointers?
Thanks,
Ram
Reformat so it's easier to read, find, and edit.
Use double-quotes as script delimiters so you don't need so many workarounds.
for f in "$1"/*.svg;
do
sed -i "
s/#000002/#$2/g;
s/#000003/#$3/g;
s/#000004/#$4/g;
s/#000005/#$5/g;
s/#000006/#$6/g;
s/#000007/#$7/g;
s/#000008/#$8/g;
s/#000009/#$9/g;
s/#000010/#$10/g;
s/#000011/#$11/g;
s/#000012/#$12/g;
s/#000013/#$13/g;
s/#000014/#$14/g;
s/#000015/#$15/g;
s/#000016/#$16/g;
" $f
done
If you're comfortable with heavy abstraction you can try something like this, but make sure you test it heavily...
for s in $(seq 2 16)
do n="$(printf "%06d" $s)"
script="$script s/#$n/#\${$s}/g; "
done
for f in "$1"/*.svg;
do eval " sed -i \" $script \" \"$f\" "
done
I want to extract a string from a text file MODIS_list.txt:
wget https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD09GA/2018/062/ -O MODIS_list.txt
then to extract the name of MODIS file:
less MODIS_list.txt | grep -o -P '(?<=hdf">).*(?<=(MOD09GA.A2018062.h18v04.006)).*(?=</a>)'
which gives as output
MOD09GA.A2018062.h18v04.006.2018064030133.hdf
Let's say I would like to loop over more file changing, for example the date or the product.
prod_var=MOD09GA
prod_date=2018062
how can insert these two variables in the grep command!??
I tried in the following syntax but it does not work:
less MODIS_list.txt | grep -o -P '(?<=hdf">).*(?<=($prod_var.A$prod_date.h18v04.006)).*(?=</a>)'
Nevertheless, instead of using a monster regex, I suggest you to convert your html file into an xml file and to select the node you want by an xpath selection as follows:
tidy -q -f /dev/null -asxml --numeric-entities yes MODIS_list.txt | /usr/bin/xpath -q -e "//a[contains(#href,'$prod_var.A$prod_date.h18v04.006.2018064030133.hdf')]/text()"
The command you want to execute is:
grep -o -P "(?<=hdf\\\">).*(?<=($prod_var.A$prod_date.h18v04.006)).*(?=</a>)" MODIS_list.txt
As wolfrevokcats says (but you need to know what they are speaking about), you have to change the single quotes into double quotes. The problem is that you have a quote after the string hdf which has to be escaped twice: once for the shell, and once for grep, but again you need to know what I am speaking about. Another solution that avoids the problem of escaping the quotes at the right side of 'hdf' is to use a '.' as follows:
grep -o -P "(?<=hdf.>).*(?<=($prod_var.A$prod_date.h18v04.006)).*(?=</a>)" MODIS_list.txt
While grepping, you may concatenate constant string and variable.
Example:
Dumpy:~ admin$ cat /tmp/file.txt
user is john
user is pol
user is bob
user is mark
user is mike
Dumpy:~ admin$ export usrname='john'
Dumpy:~ admin$ grep --color 'user is '$usrname /tmp/file.txt
user is john