When using conda env export it is possible to remove the build information with --no-build:
...
dependencies:
- _libgcc_mutex=0.1=main
- attrs=19.3.0=py_0
- backcall=0.1.0=py37_0
- beautifulsoup4=4.8.2=py37_0
- biopython=1.76=py37h7b6447c_0
- blas=1.0=mkl
- bleach=3.1.0=py37_0
...
and with --no-build
dependencies:
- _libgcc_mutex=0.1
- attrs=19.3.0
- backcall=0.1.0
- beautifulsoup4=4.8.2
- biopython=1.76
- blas=1.0
- bleach=3.1.0
- bzip2=1.0.8
- ca-certificates=2020.1.1
Could you explain in detail what exactly is this build information? The compiler and its version? What else is in there?
The build information is a hash of the variant keys in the recipe. Quoting the docs:
The takeaway message is that hashes will appear when binary compatibility matters, but not when it doesn't.
and
As of conda-build 3.1.0, this hashing scheme has been simplified. A hash will be added if all of these are true for any dependency:
Package is an explicit dependency in build, host, or run deps.
Package has a matching entry in conda_build_config.yaml which is a pin to a specific version, not a lower bound.
That package is not ignored by ignore_version.
OR
Package uses {{ compiler() }} Jinja2 function.
The documentation is here: https://docs.conda.io/projects/conda-build/en/latest/resources/variants.html#differentiating-packages-built-with-different-variants There's also a blog post (that I can't find now) with some more information.
Related
I am trying to install Dirac on a Macbook Air M1 running Monterey 12.5.
I am following the documentation which indicates:
curl -s https://raw.githubusercontent.com/binaryage/dirac/master/install > /tmp/dirac.install && sudo bash /tmp/dirac.install
The installation seems to have worked out since it exists on binaries:
Pedros-MacBook-Air:bin pedro$ pwd
/usr/local/bin
Pedros-MacBook-Air:bin pedro$ ls -1
dirac
lein
Unfortunately, when I go to the 2nd step on manual which is running dirac, the error below is retrieved:
Pedros-MacBook-Air:bin pedro$ dirac
WARNING: update-vals already refers to: #'clojure.core/update-vals in namespace: clojure.tools.analyzer.utils, being replaced by: #'clojure.tools.analyzer.utils/update-vals
WARNING: update-keys already refers to: #'clojure.core/update-keys in namespace: clojure.tools.analyzer.utils, being replaced by: #'clojure.tools.analyzer.utils/update-keys
WARNING: update-vals already refers to: #'clojure.core/update-vals in namespace: clojure.tools.analyzer, being replaced by: #'clojure.tools.analyzer.utils/update-vals
WARNING: update-keys already refers to: #'clojure.core/update-keys in namespace: clojure.tools.analyzer, being replaced by: #'clojure.tools.analyzer.utils/update-keys
WARNING: update-vals already refers to: #'clojure.core/update-vals in namespace: clojure.tools.analyzer.passes, being replaced by: #'clojure.tools.analyzer.utils/update-vals
WARNING: update-vals already refers to: #'clojure.core/update-vals in namespace: clojure.tools.analyzer.passes.uniquify, being replaced by: #'clojure.tools.analyzer.utils/update-vals
Execution error (FileNotFoundException) at dirac.nrepl.compilation.direct/eval15283$loading (direct.clj:1).
Could not locate clojure/data/json__init.class, clojure/data/json.clj or clojure/data/json.cljc on classpath.
Full report at:
/var/folders/fh/7d_1drwd2ps3zw03j9ftkqs80000gn/T/clojure-1010884132545121253.edn
Initially, I tried running it at home folder. I thought it would work out fine.
Then, I decided to run it inside a project folder that had the file clojure/data/json.clj
included in the the project.clj file. Unfortunately, even this approach did not work out.
Below you see a simplified version of the project.clj file:
(defproject abcd ; Should be copied from shadow-cljs.edn
:description "xxxx"
:url "xxxx"
:dependencies [[org.clojure/clojure "1.9.0"]
[org.clojure/clojurescript "1.10.339"]
[org.clojure/tools.nrepl "RELEASE"]
[binaryage/devtools ~devtools-version]
[binaryage/dirac ~dirac-version]
[org.clojure/data.json "2.4.0"]
[figwheel ~figwheel-version]
;; Selmer for templating of .json config files
[selmer "1.12.50"]]
... (omitted content because it seems irrelevant) ...
:profiles {:repl
{:repl-options {:port 8230
:nrepl-middleware [dirac.nrepl/middleware]
:init (do
(require 'dirac.agent)
(dirac.agent/boot!))
}}})
1 - I thought Dirac would work out even outside of the project folder. Is it necessary to be in the project folder?
2 - Why is Dirac complaining about the Json library even though it is listed as a dependency?
Is there any difference between clojure/data/json.clj (error message retrieved) and org.clojure/data.json (my dependency)?
You've received a workaround on dirac/issues/98: adding {:deps {org.clojure/data.json {:mvn/version "2.4.0"}}} to ~/.clojure/deps.edn.
The local launcher script has a dependency on the release version of ClojureScript, that no longer includes clojure/data.json. A better workaround is changing the /usr/local/bin/dirac file line 119 by adding clojure.data.json there as well (or pinning to a ClojureScript version before 1.11.51):
DEPS="{:deps {\
$DIRAC_CLI_DEPS_COORDINATE \
org.clojure/clojurescript {:mvn/version \"RELEASE\"} \
org.clojure/data.json {:mvn/version \"2.4.0\"} \
clj-logging-config/clj-logging-config {:mvn/version \"1.9.12\"} \
}}"
Explanation
The dependency org.clojure/data.json was removed from ClojureScript in version 1.11.51 of May 13th 2022. See release notes:
Vendorization of tools.reader, data.json, and transit-clj (...) After conferring with the Clojure Team, we decided to vendorize all these dependencies. This way we can AOT everything and be confident that we won’t create a conflict that can’t easily be fixed via normal dependency management. (...) The dependance on data.json has been removed.
I am deciding to use https://go-boringcrypto.storage.googleapis.com/go1.14.15b4.linux-amd64.tar.gz.
I want to know which exact NIST CMVP certificate number will be applicable for the BoringCrypto module this version has?
From quick search on NIST I can find these certs -
a) 3753
b) 3678
c) 3318
d) 2964
Is there a way to find this? This will be helpful if later we decide to upgrade and use some other version.
TLDR;
Answer is option (a) - 3753.
Here is the process to find the CMVP for your go+boringcrypto binary -
Download the binary that you want to use using this link - https://go-boringcrypto.storage.googleapis.com/go1.14.15b4.linux-amd64.tar.gz. (Note: in my project I am using 1.14.15b4, this could be different number in your case)
Unzip it and find file on this path - go/src/crypto/internal/boring/build/build.sh
look for these lines -
# Following http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140sp/140sp2964.pdf page 18.
if [ ! -e $chroot/boringssl-24e5886c0edfc409c8083d10f9f1120111efd6f5.tar.xz ]; then
wget -O $chroot/boringssl-24e5886c0edfc409c8083d10f9f1120111efd6f5.tar.xz https://commondatastorage.googleapis.com/chromium-boringssl-docs/fips/boringssl-24e5886c0edfc409c8083d10f9f1120111efd6f5.tar.xz
Link for nist cert is - http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140sp/140sp2964.pdf and version of boringssl is - 24e5886c0edfc409c8083d10f9f1120111efd6f5 , which is matching with the version mentioned in the cert.
In case these lines are not present in your go/src/crypto/internal/boring/build/build.sh file, it will be present in Dockerfile on this path go/src/crypto/internal/boring/build/build.sh.
I want to use a github action matrix for different build types, but there's one case of the matrix that I'm not interested in supporting. How do I stop this case from running but still get the build to marked successfully.
In this particular case I want to build Windows and Ubuntu, 32bit and 64bit but I'm not interested in supporting 32bit on Ubuntu. So my matrix would be:
strategy:
fail-fast: false
matrix:
os: [windows-latest, ubuntu-latest]
platform: ['x64', 'x86']
My current solution is to stop each action running by adding an if expression:
- name: Build Native
if: ${{ ! (matrix.os == 'ubuntu-18.04' && matrix.platform == 'x86') }}
While this works okay, I feel there ought to be a more elegant way of solving this. Can anyone help make my yaml script more beautiful?
Perhaps the strategy.matrix.exclude directive is suitable?
From the documentation:
You can remove a specific configurations defined in the build matrix
using the exclude option. Using exclude removes a job defined by the
build matrix.
So in your case, probably something like this:
strategy:
matrix:
os: [windows-latest, ubuntu-latest]
platform: ['x64', 'x86']
exclude:
- os: ubuntu-latest
platform: x86
There are situations where one wants to include or exclude specific matrix coordinates so as not to run some of them, yet (stretching the question a bit) also still want the job to run for a couple of these coordinates, so as to track the evolution of it across commits, while not blocking the whole process.
In that situation, continue-on-error at the job level, combined with matrix include and exclude is very useful:
Prevents a workflow run from failing when a job fails. Set to true to allow a workflow run to pass when this job fails.
This is similar to GitLab CI's allow_failure, although at time of writing GitHub Actions UI only has two states (red failed and green passed) whereas GitLab introduces a third one (orange warning) in that case.
Here is a real-life workflow example:
jobs:
linux:
continue-on-error: ${{ matrix.experimental }}
strategy:
fail-fast: false
matrix:
os:
- ubuntu-20.04
container:
- 'ruby:2.0'
- 'ruby:2.1'
- 'ruby:2.2'
- 'ruby:2.3'
- 'ruby:2.4'
- 'ruby:2.5'
- 'ruby:2.6'
- 'ruby:2.7'
- 'ruby:3.0'
- 'ruby:2.1-alpine'
- 'ruby:2.2-alpine'
- 'ruby:2.3-alpine'
- 'ruby:2.4-alpine'
- 'ruby:2.5-alpine'
- 'ruby:2.6-alpine'
- 'ruby:2.7-alpine'
- 'jruby:9.2-jdk'
experimental:
- false
include:
- os: ubuntu-20.04
container: 'ruby:3.0.0-preview2'
experimental: true
- os: ubuntu-20.04
container: 'ruby:3.0.0-preview2-alpine'
experimental: true
I am trying to build a python 3.5 environment that supports an old hddm library. Standard approaches fail due to my/anaconda's apparent inability in ignore (or downgrade) the 10.1 cuda library in favor of an older one that works with hddm.
There is a yml file available that describes a successful environment. But the advertised command
conda env create -file hddm_py35.yml
fails with an error listing all of the packages "not found." Here are the errors.
(base) PS C:\Users\Peter\anaconda3_Sep2020> conda env create --file .\hddm_py35.yml
Collecting package metadata (repodata.json): done
Solving environment: failed
ResolvePackageNotFound:
odo==0.5.0=py35_1
cffi==1.7.0=py35_0
dill==0.2.5=py35_0
singledispatch==3.4.0.3=py35_0
nb_conda_kernels==2.0.0=py35_0
requests==2.14.2=py35_0
scikit-learn==0.17.1=np111py35_1
wheel==0.29.0=py35_0
jedi==0.9.0=py35_1
widgetsnbextension==1.2.6=py35_0
bitarray==0.8.1=py35_1
theano==1.0.2=py35_0
pytz==2016.6.1=py35_0
pylint==1.5.4=py35_1
ruamel_yaml==0.11.14=py35_0
partd==0.3.6=py35_0
llvmlite==0.13.0=py35_0
multipledispatch==0.4.8=py35_0
pyparsing==2.1.4=py35_0
console_shortcut==0.1.1=py35_1
ipython_genutils==0.1.0=py35_0
patsy==0.4.1=py35_0
pytest==2.9.2=py35_0
heapdict==1.0.0=py35_1
ipywidgets==5.2.2=py35_0
bokeh==0.12.2=py35_0
hdf5==1.8.15.1=2
networkx==1.11=py35_0
backports==1.0=py35_0
pyasn1==0.1.9=py35_0
pyqt==5.6.0=py35h6538335_6
zlib==1.2.11=hbb18732_2
et_xmlfile==1.0.1=py35_0
traitlets==4.3.0=py35_0
colorama==0.3.7=py35_0
argcomplete==1.0.0=py35_1
pywin32==220=py35_1
astropy==1.2.1=np111py35_0
nose==1.3.7=py35_1
freetype==2.8=h0224ed4_1
pkginfo==1.3.2=py35_0
cloudpickle==0.2.1=py35_0
sqlalchemy==1.0.13=py35_0
lazy-object-proxy==1.2.1=py35_0
markupsafe==0.23=py35_2
prompt_toolkit==1.0.3=py35_0
pickleshare==0.7.4=py35_0
itsdangerous==0.24=py35_0
babel==2.3.4=py35_0
click==6.6=py35_0
six==1.10.0=py35_0
libdynd==0.7.2=0
jdcal==1.2=py35_1
pymc==2.3.6=np111py35_2
pathlib2==2.1.0=py35_0
astroid==1.4.7=py35_0
numba==0.28.1=np111py35_0
qtconsole==4.2.1=py35_2
wrapt==1.10.6=py35_0
idna==2.1=py35_0
pytables==3.2.2=np111py35_4
_nb_ext_conf==0.3.0=py35_0
dynd-python==0.7.2=py35_0
numexpr==2.6.1=np111py35_0
werkzeug==0.11.11=py35_0
rope==0.9.4=py35_1
jupyter_client==4.4.0=py35_0
pyzmq==15.4.0=py35_0
python-dateutil==2.5.3=py35_0
beautifulsoup4==4.5.1=py35_0
blaze==0.10.1=py35_0
nbformat==4.1.0=py35_0
nbpresent==3.0.2=py35_0
sip==4.18=py35_0
chest==0.2.3=py35_0
glob2==0.5=py35_0
locket==0.2.0=py35_1
mistune==0.7.3=py35_0
alabaster==0.7.9=py35_0
setuptools==27.2.0=py35_1
win_unicode_console==0.5=py35_0
filelock==2.0.6=py35_0
_license==1.1=py35_1
ipykernel==4.5.0=py35_0
qt==5.6.2=vc14h6f76a7e_12
pep8==1.7.0=py35_0
xlwings==0.10.0=py35_0
spyder==3.0.0=py35_0
xlrd==1.0.0=py35_0
scipy==0.18.1=np111py35_0
dask==0.11.0=py35_0
nbconvert==4.2.0=py35_0
pip==8.1.2=py35_0
mkl==11.3.3=1
nb_anacondacloud==1.2.0=py35_0
cython==0.24.1=py35_0
flask-cors==2.1.2=py35_0
ipython==5.1.0=py35_0
cycler==0.10.0=py35_0
jpeg==9b=he27b436_2
menuinst==1.4.1=py35_0
anaconda==4.2.0=np111py35_0
configobj==5.0.6=py35_0
boto==2.42.0=py35_0
unicodecsv==0.14.1=py35_0
scikit-image==0.12.3=np111py35_1
contextlib2==0.5.3=py35_0
conda-build==3.0.19=py35h15d37ab_0
jinja2==2.8=py35_1
conda-verify==2.0.0=py35_0
get_terminal_size==1.0.0=py35_0
qtpy==1.1.2=py35_0
anaconda-client==1.5.1=py35_0
decorator==4.0.10=py35_0
ply==3.9=py35_0
openpyxl==2.3.2=py35_0
sockjs-tornado==1.0.3=py35_0
pyyaml==3.12=py35_0
snowballstemmer==1.2.1=py35_0
toolz==0.8.0=py35_0
py==1.4.31=py35_0
xlwt==1.1.2=py35_0
clyent==1.2.2=py35_0
bottleneck==1.1.0=np111py35_0
jupyter==1.0.0=py35_3
mkl-service==1.1.2=py35_2
simplegeneric==0.8.1=py35_1
wcwidth==0.1.7=py35_0
h5py==2.6.0=np111py35_2
gevent==1.1.2=py35_0
pycrypto==2.6.1=py35_4
datashape==0.5.2=py35_0
psutil==4.3.1=py35_0
nltk==3.2.1=py35_0
jsonschema==2.5.1=py35_0
notebook==4.2.3=py35_0
pycparser==2.14=py35_1
xlsxwriter==0.9.3=py35_0
jupyter_core==4.2.0=py35_0
qtawesome==0.3.3=py35_0
fastcache==1.0.2=py35_1
jupyter_console==5.0.0=py35_0
tornado==4.4.1=py35_0
path.py==8.2.1=py35_0
pyflakes==1.3.0=py35_0
sympy==1.0=py35_0
pandas==0.20.1=np111py35_0
pygments==2.1.3=py35_0
anaconda-clean==1.0.0=py35_0
mpmath==0.19=py35_1
comtypes==1.1.2=py35_0
cryptography==1.5=py35_0
chardet==3.0.4=py35_0
entrypoints==0.2.2=py35_0
sphinx==1.4.6=py35_0
greenlet==0.4.10=py35_0
anaconda-navigator==1.3.1=py35_0
flask==0.11.1=py35_0
pyopenssl==16.2.0=py35_0
lxml==3.6.4=py35_0
icu==58.2=h3fcc66b_1
docutils==0.12=py35_2
statsmodels==0.6.1=np111py35_1
nb_conda==2.0.0=py35_0
imagesize==0.7.1=py35_0
(base) PS C:\Users\Peter\anaconda3_Sep2020>
The failure occurred within seconds. I get the feeling that conda didn't even try to look for these packages!?!?
Am I supposed to download these packages, put them somewhere, and then tell conda to find them on my hard drive?
Is there a flag that tells conda to do its usually find-and-load for all "missing" packages -- but only in the environment I'm describing? In my base environment (3.8) I don't wish to downgrade.
Should make a new 3.5 environment and then work through the list one-by-one and uninstall/remove/downgrade each package by hand?
Meta question: This must be a FAQ, and yet I'm not able to google for the answer. That usually means googling for "conda install environment from yaml file" doesn't contain the appropriate vocabulary for, well, trying to induce conda to install an environment from a yaml file. What question should I have asked?
1) Am I supposed to download these packages, put them somewhere, and then
tell conda to find them on my hard drive?
Not necessary. But searching for the versions on anaconda.org helps identify channels for one-by-one manual download.
2) Is there a flag that tells conda to do its usually find-and-load for all
"missing" packages -- but only in the environment I'm describing? In my base
environment (3.8) I don't wish to downgrade.
There is no evidence that conda will automatically download files listed in a yaml file that are missing in the present environment.
3) Should make a new 3.5 environment and then work through the list one-by-
one and uninstall/remove/downgrade each package by hand?
Yes.
4) Meta question: This must be a FAQ, and yet I'm not able to google for the
answer. That usually means googling for "conda install environment from yaml
file" doesn't contain the appropriate vocabulary for, well, trying to induce
conda to install an environment from a yaml file. What question should I have
asked?
There is no evidence that yaml files are anything other than lists of version of packages in an environment. They cannot be used to make new environments (unless all of the components are already present in the host environment, maybe) so their value is largely annotative. Evidently.
For the case of making an environment for hddm in 2020, well, don't try. Cuda support will work against you. There is a hddm host at https://colab.research.google.com/ that is properly configured (without cuda disruption) so that you can use it to kick tires, etc. Getting hddm to work in any other context probably requires dedicated hardware so that the cuda driver can be manipulated for this application only and not break any other applications in the process.
I've got a project structured thusly:
- proj/
- subproj1/
- stack.yaml
- subproj1.cabal
- ...
- subproj2/
- stack.yaml
- pkg1/
- ...
- pkg2/
- ...
- ...
And my subproj1/stack.yaml file contains this:
packages:
- .
- location: ../subproj2
subdirs:
- pkg1
- pkg2
extra-dep: true
I'm noticing inconsistent build behavior, when running "stack build" from within the subproj1/ directory. And I'm wondering if I have set up my project structure in an inherently unstable way. Would it, for instance, be more stable to use a single stack.yaml file, located in the proj/ directory?
Yes, unfortunately there are some known issues with this setup - https://github.com/commercialhaskell/stack/issues/3130 . Hopefully will be fixed at some point! One way to work around this is to set your STACK_YAML environment variable, so that stack invocations will ignore current directory.