Let's say I have a makefile like this:
build:
# output will be something like dist/foo-1.2.3-py3-none-any.whl
poetry build
upload: build
aws s3 cp ${WHEEL_NAME} s3://some-bucket/
The name of the wheel is obviously always changing and so it will somehow need to be recorded at the end of build and read at upload. What is the correct way of implementing this?
Here's one simple approach. Instead of dist/, use a directory whose sole purpose is to contain the latest wheel:
WHEELDIR := latest-wheel
build:
#rm -fr $(WHEELDIR)
#mkdir $(WHEELDIR)
# output will be something like $(WHEELDIR)/foo-1.2.3-py3-none-any.whl
# I don't know what "poetry build" does
upload: build
aws s3 cp $(WHEELDIR)/* s3://some-bucket/
If you like, you can add a line or two to store a copy of the wheel in dist/, or some other archive.
Related
I'm trying to create a temporary file in my pipeline, then use that file in another rule.
For example, I have two rules in a .smk file:
#Unzip adapter trimmed fastq file
rule unzip_fastq:
input:
'{sample}.adapterTrim.round2.fastq.gz',
output:
temp('{sample}.adapterTrim.round2.fastq')
conda:
'../envs/rep_element.yaml'
shell:
'gunzip -c {input[0]} > {output[0]}'
#Run bowtie2 to align to rep elements and parse output
rule parse_bowtie2_output_realtime:
input:
'{sample}.adapterTrim.round2.fastq'
output:
'rep_element_pipeline/{sample}.fastq.gz.mapped_vs_' + config["ref"]["bt2_index"] + '.sam'
params:
bt2=config["ref"]["bt2_index_path"], eid=config["ref"]["enst2id"]
conda:
'../envs/rep_element.yaml'
shell:
'perl ../scripts/parse_bowtie2_output_realtime_includemultifamily.pl '
'{input[0]} {params.bt2} {output[0]} {params.eid}'
{sample}.adapterTrim.round2.fastq is used once and should ultimately be deleted upon completion. However, I'm finding that this file is uploaded to Amazon S3, even with the addition of temp(). I'm also finding that this file is removed locally, but still persists on S3.
Am I doing this correctly? '{sample}.adapterTrim.round2.fastq' is not currently written in the rule-all of the Snakefile.
We ultimately need to prevent this file from being uploaded to S3, so if there is a way to specify not to upload this file in the rule, that would be useful.
It seems that the snippet in the question is not consistent with actual use, since for S3 files one would need to wrap file names in remote.
However, as a general solution, documentation contains the following:
The remote() wrapper is mutually-exclusive with the temp() and protected() wrappers.
Hence, if you intend to use a temp file, make sure it's not wrapped in remote, or explicitly wrap the file in local.
I've got a file containing a list of paths that I need to copy by Dockerfile's COPY command on docker build.
My use case is such: I've got a python requirements.txt file, when inside I'm calling multiple other requirements files inside the project, with -r PATH.
Now, I want to docker COPY all the requirements files alone, run pip install, and then copy the rest of the project (for cache and such). So far i haven't managed to do so with docker COPY command.
No need of help on fetching the paths from the file - I've managed that - just if it is possible to be done - how?
thanks!
Not possible in the sense that the COPY directive allows it out of the box, however if you know the extensions you can use a wildcard for the path such as COPY folder*something*name somewhere/.
For simple requirements.txt fetching that could be:
# but you need to distinguish it somehow
# otherwise it'll overwrite the files and keep the last one
# e.g. rename package/requirements.txt to package-requirements.txt
# and it won't be an issue
COPY */requirements.txt ./
RUN for item in $(ls requirement*);do pip install -r $item;done
But if it gets a bit more complex (as in collecting only specific files, by some custom pattern etc), then, no. However for that case simply use templating either by a simple F-string, format() function or switch to Jinja, create a Dockerfile.tmpl (or whatever you'd want to name a temporary file), then collect the paths, insert into the templated Dockerfile and once ready dump to a file and execute afterwards with docker build.
Example:
# Dockerfile.tmpl
FROM alpine
{{replace}}
# organize files into coherent structures so you don't have too many COPY directives
files = {
"pattern1": [...],
"pattern2": [...],
...
}
with open("Dockerfile.tmpl", "r") as file:
text = file.read()
insert = "\n".join([
f"COPY {' '.join(values)} destination/{key}/"
for key, values in files.items()
])
with open("Dockerfile", "w") as file:
file.write(text.replace("{{replace}}", insert))
You might want to do this for example:
FROM ...
ARG files
COPY files
and run with
docker build -build-args items=`${cat list_of_files_to_copy.txt}`
I have a custom target:
add_custom_target(
create-po
COMMAND ${MSGINIT} --no-translator -i "${PROJECT_SOURCE_DIR}/data/${PACKAGE}.pot" - "${PROJECT_SOURCE_DIR}/po/es.po" -l es_MX.utf8
)
so, is invoked like this:
# make create-po
my idea is to change it to something like this:
# make create-po "es"
so, any user can create a custom localed po file. I don't know the word exactly for this, but I'd like to add a parameter in the target name..is it posible with cmake? Thanks
After so long time I found this question for the same reason: Can I use CMake to initialize a .po file if I want to add a new translation? I expect to use it only once in a while for my project, so make the build system do it seems more comfortable to me than find out all the required options and paths every time.
I ended up with the following CMake snippet:
set(INIT_LANG CACHE STRING "give a locale here to create a target which initializes a related .po file")
IF(INIT_LANG)
add_custom_target(
create-po-${INIT_LANG}
... # integrate INIT_LANG in your command
)
ENDIF(INIT_LANG)
Then, if you want to initialize a new translation file, call (assuming your build dir in under the project root):
# cmake -DINIT_LANG=es_MX.utf8 ..
... and you should get a corresponding make target:
# make create-po-es_MX.utf8
Yes, it's not as straight-forward as the OP's idea/expectation (and mine as well), but users can create new .po files by themselves (of course, this will be documented properly for them in the project ;) ).
i have the source-code of an application that supports adding python plugins. i have written a python script and want to build a custom rpm that by-default includes my script. So that i do not have to additionally add it after the rpm installation.
Now as far as i understand, there are two parts to this-
Adding the file to the source code.
Listing that file in the .spec file.
How do i know where to put the file in the source? How do i specify the path where i want my script to be copied? The spec file contains text like-
%if %{with_python}
%files python
%{_mandir}/man5/collectd-python*
%{_libdir}/%{name}/python.so
//Something like this?
// %{_libdir}/%{name}/gearman.py
// %{_libdir}/%{name}/redis.py
%endif
You need to know where to place your script file on the target installation (e.g. /usr/lib/myApp/plugins/myNiceScript.py)
In the spec-File (section %install) you have to copy your script under %{buildroot} into the target directory (which has to be created first.
%install
...
# in case the dir does not exist:
mkdir -p %{buildroot}/usr/lib/myApp/plugins
cp whereitis/myNiceScript.py %{buildroot}/usr/lib/myApp/plugins
At the end you have to define the file flags in the %files section. E.g. if your file has to have 644 under root:
%files
...
%defattr(644,root,root)
/usr/lib/myApp/plugins/myNiceScript.py
If your plugins directory is to be created during installation you need to define these flags too:
%defattr(755,root,root)
%dir /usr/lib/myApp/plugins
I'm new to CMake and I have a problem that I can not figure out a solution to. I'm using CMake to compile a project with a bunch of optional sub-dirs and it builds shared library files as expected. That part seems to be working fine. Each of these sub-dirs contains a sql file. I need to concat all the selected sql files to one sql header file and install the result. So one file like:
sql_header.sql
sub_dir_A.sql
sub_dir_C.sql
sub_dir_D.sql
If I did this directly in a make file I might do something like the following only smarter to deal with only the selected sub-dirs:
cat sql_header.sql > "${INSTALL_PATH}/somefile.sql"
cat sub_dir_A.sql >> "${INSTALL_PATH}/somefile.sql"
cat sub_dir_C.sql >> "${INSTALL_PATH}/somefile.sql"
cat sub_dir_D.sql >> "${INSTALL_PATH}/somefile.sql"
I have sort of figured out pieces of this, like I can use:
LIST(APPEND PACKAGE_SQL_FILES "some_file.sql")
which I assume I can place in each of the sub-dirs CMakeLists.txt files to collect the file names. And I can create a macro like:
CAT(IN "${PACKAGE_SQL_FILES}" OUT "${INSTALL_PATH}/somefile.sql")
But I am lost between when the CMake initially runs and when it runs from the make install. Maybe there is a better way to do this. I need this to work on both Windows and Linux.
I would be happy with some hints to point me in the right direction.
You can create the concatenated file mainly using CMake's file and function commands.
First, create a cat function:
function(cat IN_FILE OUT_FILE)
file(READ ${IN_FILE} CONTENTS)
file(APPEND ${OUT_FILE} "${CONTENTS}")
endfunction()
Assuming you have the list of input files in the variable PACKAGE_SQL_FILES, you can use the function like this:
# Prepare a temporary file to "cat" to:
file(WRITE somefile.sql.in "")
# Call the "cat" function for each input file
foreach(PACKAGE_SQL_FILE ${PACKAGE_SQL_FILES})
cat(${PACKAGE_SQL_FILE} somefile.sql.in)
endforeach()
# Copy the temporary file to the final location
configure_file(somefile.sql.in somefile.sql COPYONLY)
The reason for writing to a temporary is so the real target file only gets updated if its content has changed. See this answer for why this is a good thing.
You should note that if you're including the subdirectories via the add_subdirectory command, the subdirs all have their own scope as far as CMake variables are concerned. In the subdirs, using list will only affect variables in the scope of that subdir.
If you want to create a list available in the parent scope, you'll need to use set(... PARENT_SCOPE), e.g.
set(PACKAGE_SQL_FILES
${PACKAGE_SQL_FILES}
${CMAKE_CURRENT_SOURCE_DIR}/some_file.sql
PARENT_SCOPE)
All this so far has simply created the concatenated file in the root of your build tree. To install it, you probably want to use the install(FILES ...) command:
install(FILES ${CMAKE_BINARY_DIR}/somefile.sql
DESTINATION ${INSTALL_PATH})
So, whenever CMake runs (either because you manually invoke it or because it detects changes when you do "make"), it will update the concatenated file in the build tree. Only once you run "make install" will the file finally be copied from the build root to the install location.
As of CMake 3.18, the CMake command line tool can concatenate files using cat. So, assuming a variable PACKAGE_SQL_FILES containing the list of files, you can run the cat command using execute_process:
# Concatenate the sql files into a variable 'FINAL_FILE'.
execute_process(COMMAND ${CMAKE_COMMAND} -E cat ${PACKAGE_SQL_FILES}
OUTPUT_VARIABLE FINAL_FILE
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
# Write out the concatenated contents to 'final.sql.in'.
file(WRITE final.sql.in ${FINAL_FILE})
The rest of the solution is similar to Fraser's response. You can use configure_file so the resultant file is only updated when necessary.
configure_file(final.sql.in final.sql COPYONLY)
You can still use install in the same way to install the file:
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/final.sql
DESTINATION ${INSTALL_PATH})