From the docs:
https://snakemake.readthedocs.io/en/stable/executable.html
Currently, this mode requires that the Snakemake workflow is stored in a git repository. Snakemake uses git to query necessary source files (the Snakefile, scripts, config, …) for workflow execution and encodes them into the kubernetes job.
However, what is not clear to me is how to specify source files as stated. srcdir returns the absolute path with the specified prefix prepended.
Could someone please point me to the relevant area of the docs please!
Thanks
I haven't worked with kubernetics and Snakemake, but I am assuming using the 'default' structure should work. You can easily generate this structure with the snakemake cookiecutter repo:
https://github.com/snakemake-workflows/cookiecutter-snakemake-workflow
Just make sure cookiecutter is installed and you can run this on the command-line:
cookiecutter https://github.com/snakemake-workflows/cookiecutter-snakemake-workflow
Or you can always just take a look at where files are stored in that repo and copy the structure.
Related
I have a mono-repository ("monorepo") that contains the C++ source code for my application, as well as a tree of OE/Yocto files and directories that form the bitbake recipes required to build my final image.
I wish to build and install my application into the image, but as far as I can tell, the Yocto philosophy is that source code is "fetched" (e.g. via git), from an external place, before it is built. But in my case the source code resides in the same repository. It doesn't seem to make sense to me that the entire repository is downloaded again, by bitbake, just to fetch the source for this application, so I'm looking for a better way.
I'm familiar with the idea of putting all the source in the files/ subdirectory, alongside the recipe itself. The issue I have with that is that I don't want to keep the source in the Yocto layer's recipe tree. It can be built with the SDK, for example, or even with other completely unrelated toolchains, so it should not be buried within a Yocto layer. It has its own life, outside Yocto.
I'm also familiar with the EXTERNALSRC directive, that can be used to "point" to the source code with a relative path from the build directory. For example:
EXTERNALSRC = "${TOPDIR}/../../src/myproject"
However, this is frequently not recommended as a "production" mechanism due to path issues, and it also disables devtool:
ERROR: externalsrc is currently enabled for the myproject recipe. This prevents the normal do_patch task from working. You will need to disable this first.
So I'm looking for a recommendation on how to handle compiling an application that resides in the same repository as the recipe, without putting it in files/.
EDIT: I tried something along these lines:
SRC_URI = "file://${TOPDIR}/../../src/myproject/main.c \
file://${TOPDIR}/../../src/myproject/Makefile \
"
This did not compile, but with devtool modify myproject I was able to see that it has in fact copied the source into the build directory. The problem is that it's replicated the entire directory structure from the root all the way up to the original source directory, so my source is now sitting in a location like this:
/home/david/monorepo/yocto/build/workspace/sources/myproject/home/david/monorepo/src/myproject
do_compile will need to determine and set that working directory before it will compile.
This means that the path will change depending on the user and the location of where they've checked out the monorepo. This almost works, but doesn't seem usable as-is. Is there a way to modify where bitbake's "file" fetcher puts the source when given an absolute path?
EDIT 2:
I may have found a way that works with bitbake and devtool:
FILESEXTRAPATHS_prepend := "${TOPDIR}/../../src/myproject:"
SRC_URI = "file://main.c \
file://Makefile \
"
This seems to set up the devtool directory in a much saner way (no replication of the directory tree, just symlinks to the files in oe-local-files/* directory), and the bitbake recipe also builds and installs correctly.
Is this the right way to do it?
EDIT 3: Perhaps not, as FILESEXTRAPATHS is only intended to be modified by .bbappend recipes, not base .bb recipes - any comment on that?
Best practices dictate that you accomplish this by using FILESEXTRAPATHS from within a .bbappend file [source].
EDIT 4: PierreOlivier suggests using a relative symlink in the files/ directory to the application's source directory, so that SRC_URI can find the source as if it was actually present in files/. From what I can tell with my own experiments, this does seem to work, and devtool works with this also.
Are there any implications of this approach that I should be aware of?
Currently I am working on an API which uses Serverless Framework with Go.
I'm using the Serverless-offline plugin for local testing.
This API depends on a few other repositories (which I also maintain), which I import using the go.mod file.
However I am having a hard time refining my developer workflow.
Currently, if I want to make changes in a repository which this API depends upon, I have to alter the projects go.mod to include replace directives for the purpose of testing, but then I'm having to manually change it back for deployment to production.
Basically I'm looking for a way to include replace directives, which only get applied during local development. How has everyone else dealt with this problem?
Bonus question: Is there any way to run Serverless offline in docker? I'm finding that serverless-offline running on the bare metal is causing inconsistencies between different developers environments.
You can run go commands with an alternate go.mod file with the -modfile option:
From Build commands:
The -modfile=file.mod flag instructs the go command to read (and
possibly write) an alternate file instead of go.mod in the module root
directory. The file’s name must end with .mod. A file named go.mod
must still be present in order to determine the module root directory,
but it is not accessed. When -modfile is specified, an alternate
go.sum file is also used: its path is derived from the -modfile flag
by trimming the .mod extension and appending .sum.
Create a local.go.mod file with the necessary replace directive for development and build, for example, with:
go build -modfile=local.go.mod ./...
I have seen the other 2 appends and added sonar.languages=COBOL to my sonar-project.properties file to no avail.
I think it's basically more a case of not understanding the documentation (or the latter never having been "idiot tested"").
My problem is as follows. My properties file contains the following entries:-
# sonar.sources=../../mfuser/seb/source/
sonar.sources=C:/mfuser/SEB/Source
sonar.cobol.file.suffixes=cbl,cpy
sonar.cobol.copy.suffixes=cpy
#
sonar.languages=COBOL
When I run the sonar-scanner bat file, I'm seeing output like this:-
WARN File C:\mfuser\seb\source\vvira20.cbl is ignored. It is not located in module basedir c:\sonar-scanner\bin
To me this gives the impression that I can point to the directory containing my COBOL code, but at the same time the actual code HAS to be in c:\sonar-scanner\bin. Surely, that can't be the case? If it is, what is the point with the sonar.sources entry in the properties file?
In addition, onece I get this working, is there some way of specifying a parm/wild card so as to analyze SPECIFIC files. I tried sonar.sources=C:/mfuser/SEB/Source/vno* but that didn't seem to make any difference.
Thanks
You ran sonar-scanner from its own directory instead of running from the project directory. As the documentation says:
Run the following command from the project base directory
sonar-scanner
This same documentation recommends to create the sonar-project.properties file at the root of the project directory and to set the "sonar.sources" property to a path which is relative to the sonar-project.properties file.
Alternatively, you may want to use the "sonar.projectBaseDir" property. See the documentation for analysis parameters.
having a Bitbucket repository, there is a project in a subdirectory that should be built. I'd like to have the wercker.yml in this subdirectory as well.
I tried to add an environment variable WERCKER_ROOT to the pipeline containing the subdirectory: /here-is-werckerfile, according to wercker devcenter which seemed to have no effect.
Any ideas?
As far as I know you can't change this environment. It's not used by wrecker, it's only meant to provide a read only information in case you need to know where your code is located.
To answer your question: once again as far as I know you can't use a wercker.yml from a subdirectory as there's no way to set it's location for wercker - it'll already search for it in your repository root.
edit however you can change the source dir, as mentioned here: http://old-devcenter.wercker.com/articles/steps/variables.html - with the WERCKER_SOURCE_DIR environment, so you can put your wercker.yml in the repo root but use the sub directory for building.
I want to build a file with GNU make on a machine which I have write permission only to the tmp directory.
When I try to build I get a permission error because MAKE is trying to put the object file in the build directory which I have no write permissions for.
is it possible to provide make a specific directory where to put the object files ?
Thanks,
Itay
is it possible to provide make a specific directory where to put the object files ?
If the makefile does not allow specifying a build directory you can copy (or symlink if possible) the sources into the destination directory and build there.
Autoconf-based projects normally allow this kind of usage.
It depends on where your make rules are coming from.
Usually, rules are written to be location independent: they are written such that whether the make command will work regardless of where the source files are. This is true for the built-in rules, but also for the rules in most Makefiles, if you got any.
In that case, all you need to do is copy or move everything into /tmp and run make there.
However, if you generated your Makefiles with a tool (e.g. a ./configure script, as used by GNU autoconf), the generation process may have introduced absolute paths, so you may need to redo the generation step(s) after copying everything to /tmp.