I have been using a pretty simple makefile for my code but I'm trying to set it up so that it is more user-friendly, i.e. so that they do not have to edit the makefile themselves. Instead a bash script will read in variables from their input then write those to a makefile 'template' that is copied each time (which then just gets saved as 'Makefile') before running the make commands for them.
I have an issue when it comes to the section for submitting the jobs to a cluster. I have defined the necessary variables for either a Slurm or Torque based system, for personal use I just commented out whichever wasn't needed but this doesn't work with the "user-friendly" approach I'm trying to go for. I thought to use if statements with a variable "cluster_option" that gets set to one of those options, then have the variables inside if statements so that they're only defined for the appropriate option.
cluster_option = "Cluster_"
if [ "${cluster_option}" == 'slurm' ]; then\
SUBMIT_JOB_LIST = "all.jobs"
SUBMIT_JOB_NAME = "job_name_"
SUBMIT_JOB_NODES = "--nodes=node_no_"
SUBMIT_JOB_PPN = "--ntasks-per-node=node_tasks_"
SUBMIT_JOB_TIME = "--time=time_"
SUBMIT_JOB_TASK = "job_task/job_slurm.sh"
SUBMIT_JOB_CLUSTERTYPE = "SLURM"
elif [ "${cluster_option}" == 'torque' ]; then\
SUBMIT_JOB_LIST = "all.jobs"
SUBMIT_JOB_TAG = "job_name_"
SUBMIT_JOB_RESOURCES = "nodes=node_no_:ppn=node_tasks_,walltime=time_"
SUBMIT_JOB_TASK = "job_task/job.sh"
SUBMIT_JOB_CLUSTERTYPE = "TORQUE"
fi
I'm not really sure if makefiles work this way and I'm not sure if I need to add ';' to the end of each line inside or not? Is there a better way of setting this up? I'm not sure if it's relevant, but this just calls multiple bash scripts to run and then submits one (or more) as a job to the cluster.
Related
Suppose I have:
# ./Makefile
CLUSTER=dev
include Makefile.cluster.mk
CLUSTER=local
include Makefile.cluster.mk
And in:
# ./Makefile.cluster.mk
${CLUSTER}.cmd:
cmd ${CLUSTER}
So now I can call:
make dev.cmd
make local.cmd
Great! Except the variable is evaluated too late. Running:
$ make local.cmd # cmd local
$ make dev.cmd # Also cmd local !
Make sense: according to: https://www.gnu.org/software/make/manual/html_node/Reading-Makefiles.html
rule steps are deferred evaluation (vs. immediate/on file load).
immediate : immediate ; deferred
deferred
Is there a better/other way to compose a set of make commands without maintaining multiple copies of the same file?
There are lots of ways to do it, even beyond the options above; you could use static pattern rules:
CLUSTERS := dev local
$(CLUSTERS:%=%.cmd) : %.cmd :
cmd $*
If you really want to have stuff in a separate makefile you can use target-specific variables; change your Makefile.cluster.mk to do this:
# ./Makefile.cluster.mk
${CLUSTER}.cmd: CLUSTER := $(CLUSTER)
${CLUSTER}.cmd:
cmd ${CLUSTER}
Is there a better/other way to compose a set of make commands without maintaining multiple copies of the same file?
Often it's pattern rules. In the case of your particular example, you might do
Makefile
%.cmd:
cmd '$*'
However, that particular version will enable any make foo.cmd, which might not be what you want.
Sometimes it's to make better use of the tools available to you. For example,
Makefile.cluster.mk
${CLUSTER}.cmd:
arg='$#'; cmd "$${arg%.cmd}"
That extracts the wanted cluster name from the name of the target.
Occasionally it is $(eval).
(See the manual for an example.)
And from time to time, it's "don't do that." For example,
Makefile
CLUSTERS = dev local
CMDS = $(patsubst %,%.cmd,$(CLUSTERS))
$(CMDS):
arg='$#'; cmd "$${arg%.cmd}"
That defines only dev.cmd and local.cmd targets, and avoids duplicating the recipe.
Summary:
I am writing a script to check if a directory exists on a remote machine. I need a solution which can allow me to check for that directory and return the result in a usable way. This whole process is automated within a much larger script so I need a functional way to tell the parent script the directory exists or not.
Restrictions:
The tools that I have are limited. $REMOTE_2 can only be accessed through $REMOTE_1. Also, $REMOTE_2 can only be connected via telnet (no ssh available).
Current goal:
I am trying to set a local variable to be then read back to chose a return code. I am open to other options, but this is the closest I've come to a working solution so far.
I realize that $found will take from the parent process, but this is not my desired result and am not sure what syntax I need to return true or false when trying to echo the $found variable.
/usr/bin/expect<<EOF
spawn ssh $USER#$REMOTE_1
expect "*$USER*"
send -- "telnet $REMOTE_2\r"
expect "*login:*"
send -- "root\r"
expect "*$*"
# Everything prior to this can't be changed. Everything after it can be.
send "if \[ -d $DIRECTORY_LOCATION \] ; then found=true; else found=false ; fi\r"
send -- "echo **$found**\r"
expect {
"*true*" {
exit 0
}
"*false*" {
exit 1
}
}
EOF
I believe this type of solution can work, but I am not sure how to use the remote variable that I store within the if statement, later on to allow me to choose which return code to use.
I have recipe file and my SRC_URI section looks something as follows:
SRC_URI += "file://file1.patch \
file://file2.patch \
file://file4.patch \
"
I want to include a file5.patch under the SRC_URI only if a certain environment variable is set. Is there a way to insert a if condition with the SRC_URI that looks something like this:
SRC_URI += "file://file1.patch \
file://file2.patch \
file://file4.patch \
**if $ENVIRONMENT_VARIABLE:
file://file5.patch**
"
Is there any other way I can achieve the same thing?
Well, the short answer is: yes, you can do this, but it's messy and there's probably a Better Way(TM). So let's answer the question first. If you really want to change the behavior of a recipe using an environment variable, the first challenge is to set the environment variable, and then let bitbake know that your new environment variable is safe and allowable. When you source the oe-init-build-env script to setup your project or subsequently to setup your new shell to continue working on the project, it sets an env variable called BB_ENV_EXTRAWHITE. You must include your new env variable in this list like this:
$ export MYENV_VAR=file5.patch
$ export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE MYENV_VAR"
Once this is done, then bitbake won't scrub the environment of your new environment variable.
In your recipe, use a python snippet to conditionally add your patch as follows:
SRC_URI += "${#os.getenv('MYENV_VAR', '')}"
As you can see, it's a bit messy. Of course, you could get a little more complex and test the value of the variable in your recipe, instead of putting the name of the patch file in your environment variable, but this example was the simplest way to demonstrate the concept.
Perhaps a better way is to use an override, and not rely on environment variables. If you are building a bsp with multiple variants, you could use your bsp name as the override, something like this.
SRC_URI_append_mybsp = "file://file5.patch"
This is a much cleaner way to accomplish the same thing. Of course, I'm speculating about your use case. The yocto project reference manual explains overrides. One more suggestion, join #yocto or the yocto project mailing list and you will have access to many smart people to help you.
Hope this helps. ;)
The proper way to accomplish this would be as follows,
1. local.conf
# comment the following line to remove file5.patch
ENV_VAR = "1"
NOTE: Don't forget to include the double quotes, otherwise Yocto will throw error.
2. recipe.bbappend
SRC_URI += "${#bb.utils.contains('ENV_VAR', '1', 'file://file5.patch', '', d)}"
Instead of local.conf you're free to use any .conf file. It's taken from Yocto mailing list
I'm working with a big software project with many build targets. When typing make <tab> <tab> it shows over 1000 possible make targets.
What I want is a bash script that filters those targets by certain rules. Therefore I would like to have this list of make targets in a bash variable.
make_targets=$(???)
[do something with make_targets]
make $make_targets
It would be best if I wouldn't have to change anything with my project.
How can I get such a List?
#yuyichao created a function to get autocomplete output:
comp() {
COMP_LINE="$*"
COMP_WORDS=("$#")
COMP_CWORD=${#COMP_WORDS[#]}
((COMP_CWORD--))
COMP_POINT=${#COMP_LINE}
COMP_WORDBREAKS='"'"'><=;|&(:"
# Don't really thing any real autocompletion script will rely on
# the following 2 vars, but on principle they could ~~~ LOL.
COMP_TYPE=9
COMP_KEY=9
_command_offset 0
echo ${COMPREPLY[#]}
}
Just run comp make '' to get the results, and you can manipulate that. Example:
$ comp make ''
test foo clean
You would need to overwrite / modify the completion function for make. On Ubuntu it is located at:
/usr/share/bash-completion/completions/make
(Other distributions may store the file at /etc/bash_completion.d/make)
If you don't want to change the completion behavior for the whole system, you might write a small wrapper script like build-project, which calls make. Then write a completion function for that mapper which is derived from make's one.
I'm trying to work out the best way to set some environment variables with puppet.
I could use exec and just do export VAR=blah. However, that would only last for the current session. I also thought about just adding it onto the end of a file such as bashrc. However then I don't think there is a reliable method to check if it is all ready there; so it would end up getting added with every run of puppet.
I would take a look at this related question.
*.sh scripts in /etc/profile.d are read at user-login time (as the post says, at the same time /etc/profile is sourced)
Variables export-ed in any script placed in /etc/profile.d will therefore be available to your users.
You can then use a file resource to ensure this action is idempotent. For example:
file { "/etc/profile.d/my_test.sh":
content => 'export MYVAR="123"'
}
Or an alternate means to an indempotent result:
Example
if [[ ! grep PINTO_HOME /root/.bashrc | wc -l > 0 ]] ; then
echo "export PINTO_HOME=/opt/local/pinto" >> /root/.bashrc ;
fi
This option permits this environmental variable to be set when the presence of the
pinto application makes it warrented rather than having to compose a user's
.bash_profile regardless of what applications may wind up on the box.
If you add it to your bashrc you can check that it's in the ENV hash by doing
ENV[VAR]
Which will return => "blah"
If you take a look at Github's Boxen they source a script (/opt/boxen/env.sh) from ~/.profile. This script runs a bunch of stuff including:
for f in $BOXEN_HOME/env.d/*.sh ; do
if [ -f $f ] ; then
source $f
fi
done
These scripts, in turn, set environment variables for their respective modules.
If you want the variables to affect all users /etc/profile.d is the way to go.
However, if you want them for a specific user, something like .bashrc makes more sense.
In response to "I don't think there is a reliable method to check if it is all ready there; so it would end up getting added with every run of puppet," there is now a file_line resource available from the puppetlabs stdlib module:
"Ensures that a given line is contained within a file. The implementation matches the full line, including whitespace at the beginning and end. If the line is not contained in the given file, Puppet appends the line to the end of the file to ensure the desired state. Multiple resources can be declared to manage multiple lines in the same file."
Example:
file_line { 'sudo_rule':
path => '/etc/sudoers',
line => '%sudo ALL=(ALL) ALL',
}
file_line { 'sudo_rule_nopw':
path => '/etc/sudoers',
line => '%sudonopw ALL=(ALL) NOPASSWD: ALL',
}