I am accustomed to seeing requirements.txt that has libraries optionally with an associated [minimum] version.
pandas==1.0.4
In an existing project I am now working on instead we have library entries like this:
argcomplete==1.12.3 \
--hash=sha256:291f0beca7fd49ce285d2f10e4c1c77e9460cf823eef2de54df0c0fec88b0d81 \
--hash=sha256:2c7dbffd8c045ea534921e63b0be6fe65e88599990d8dc408ac8c542b72a5445
asgiref==3.4.1; python_version >= "3.6" \
--hash=sha256:ffc141aa908e6f175673e7b1b3b7af4fdb0ecb738fc5c8b88f69f055c2415214 \
--hash=sha256:4ef1ab46b484e3c706329cedeff284a5d40824200638503f5768edb6de7d58e9
certifi==2021.5.30; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6" \
--hash=sha256:50b1e4f8446b06f41be7dd6338db18e0990601dce795c2b1686458aa7e8fa7d8 \
--hash=sha256:2bbf76fd432960138b3ef6dda3dde0544f27cbf8546c458e60baf371917ba9ee
cffi==1.14.6; python_version >= "3.6" \
--hash=sha256:22b9c3c320171c108e903d61a3723b51e37aaa8c81255b5e7ce102775bd01e2c \
--hash=sha256:f0c5d1acbfca6ebdd6b1e3eded8d261affb6ddcf2186205518f1428b8569bb99 \
--hash=sha256:99f27fefe34c37ba9875f224a8f36e31d744d8083e00f520f133cab79ad5e819 \
--hash=sha256:55af55e32ae468e9946f741a5d51f9896da6b9bf0bbdd326843fec05c730eb20 \
The project does use poetry - which might have been the generator of the file? How did it generate this file and how can I add my new dependency - which is pytest 7.0.0 ?
First add the library with with Poetry CLI's add command:
poetry add 'pandas==1.0.4'
or any other version constraints than ==1.0.4 that you might want.
Then generate the requirements file with its export command:
poetry export --format requirements.txt --output requirements.txt
Related
I'm trying to install Singularity after installing Go. I've confirmed that Go has been installed successfully:
$ go version
go version go1.17.6 linux/amd64
After that, I run the following commands:
$ export VERSION=3.9.4
$ wget https://github.com/sylabs/singularity/archive/refs/tags/v${VERSION}.tar.gz
$ tar -xzf v${VERSION}.tar.gz
$ cd singularity-${VERSION}
$ ./mconfig # This is where it fails
E: Not inside a git repository and no VERSION file found. Abort.
The tarball you are downloading is the one generated automatically for a tag, not the release tarball. From the release page:
Source Code
Please use the singularity-ce-3.9.4.tar.gz download below to obtain and install SingularityCE 3.9.4. The GitHub auto-generated 'Source Code' downloads do not include required dependencies etc.
Working snippet using the correct url:
export VERSION=3.9.4
wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-ce-${VERSION}.tar.gz && \
tar xf singularity-ce-${VERSION}.tar.gz && \
cd singularity-ce-${VERSION} && \
./mconfig
I have a CodePipline that grabs code out of CodeCommit bundles it up in CodeBuild and then publishes it via CloudFormation.
I want to use the Python package gspread and because it's not part of the standard AWS Linux image I need to install it.
Currently when the code is run I get the error:
[ERROR] Runtime.ImportModuleError: Unable to import module 'index': No module named 'gspread'
Code structure
- buildspec.yml
- template.yml
package/
- gspread/
- gspread-3.6.0.dist-info/
- (37 other python packages)
source/
- index.py
buildspec.yml -- EDITED
version: 0.2
phases:
install:
commands:
# Use Install phase to install packages or any pre-reqs you may need throughout the build (e.g. dev deps, security checks, etc.)
- echo "[Install phase]"
- pip install --upgrade pip
- pip install --upgrade aws-sam-cli
- sam --version
- cd source
- ls
- pip install --target . gspread oauth2client
# consider using pipenv to install everything in the environement and then copy the files installed into the /source folder
- ls
runtime-versions:
python: 3.8
pre_build:
commands:
# Use Pre-Build phase to run tests, install any code deps or any other customization before build
# - echo "[Pre-Build phase]"
build:
commands:
- cd ..
- sam build
post_build:
commands:
# Use Post Build for notifications, git tags and any further customization after build
- echo "[Post-Build phase]"
- export BUCKET=property-tax-invoice-publisher-deployment
- sam package --template-file template.yml --s3-bucket $BUCKET --output-template-file outputtemplate.yml
- echo "SAM packaging completed on `date`"
##################################
# Build Artifacts to be uploaded #
##################################
artifacts:
files:
- outputtemplate.yml
discard-paths: yes
cache:
paths:
# List of path that CodeBuild will upload to S3 Bucket and use in subsequent runs to speed up Builds
- '/root/.cache/pip'
The index.py file has more in it than this. But to show the offending line.
-- index.py --
import os
import boto3
import io
import sys
import csv
import json
import smtplib
import gspread #**<--- Right here**
def lambda_handler(event, context):
print("In lambda_handler")
What I've tried
Creating the /package folder and committing the gspread and other packages
Running "pip install gspread" in the CodeBuild builds: commands:
At the moment, I'm installing it everywhere and seeing what sticks. (nothing is currently sticking)
Version: Python 3.8
I think you may need to do the following steps :
Use virtual env to install the packages locally.
Create requirements.txt to let code build know of the package requirement.
In CodeBuild buildspec.xml , include commands to install virutal env and then supply requirements.txt.
pre_build:
commands:
pip install virtualenv
virtualenv env
. env/bin/activate
pip install -r requirements.txt
Detailed steps here for reference :
https://adrian.tengamnuay.me/programming/2018/07/01/continuous-deployment-with-aws-lambda/
I want to train my existing tesseract model for a new character. I already tried the tutorial on
https://github.com/tesseract-ocr/tesseract/wiki/TrainingTesseract-4.00#lstmtraining-command-line
(Fine Tuning for ± a few characters) (I'm using a MAC)
But it does not work. If I evaluate (even on the training data), it can't recognize the ± character.
I installed:
tesseract 5.0.0-alpha-447-g52cf
leptonica-1.78.0
libgif 5.1.4 : libjpeg 9c : libpng 1.6.37 : libtiff 4.0.10 : zlib 1.2.11 : libwebp 1.0.3 : libopenjp2 2.3.1
Found AVX2
Found AVX
Found FMA
Found SSE
Found libarchive 3.4.0 zlib/1.2.11 liblzma/5.2.4 bz2lib/1.0.6
via:
I cloned the following GitHub repositories to my Desktop and installed tesseract:
https://github.com/tesseract-ocr/tesseract.git
https://github.com/tesseract-ocr/langdata_lstm
https://github.com/tesseract-ocr/tessdata_best
My installation was as follows:
Install:
brew install automake autoconf autoconf-archive libtool
brew install pkgconfig
brew install icu4c
brew install leptonica
brew install gcc
Run
ln -hfs /usr/local/Cellar/icu4c/60.2 /usr/local/opt/icu4c
Go into the cloned tesseract folder.
~/Desktop/tesseract
Run autogen.sh:
./autogen.sh
Install dependencies:
brew install cairo pango icu4c autoconf libffi libarchive libpng
export PKG_CONFIG_PATH=\
(brew --prefix)/lib/pkgconfig:\
(brew --prefix)/opt/libarchive/lib/pkgconfig:\
(brew --prefix)/opt/icu4c/lib/pkgconfig:\
(brew --prefix)/opt/libffi/lib/pkgconfig:\
(brew --prefix)/opt/libpng/lib/pkgconfig
(IF SOME ARE ALREADY INSTALLED, use reinstall instead of install)
Run configure:
./configure
Install tesseract:
make
sudo make install
Install training-tools:
make training
sudo make training-install
After, I inserted eng.traineddata from tessdata_best to tesseract/tessdata
My training code is as follows:
# GENERATE TRAINING DATA
rm -rf ~/Desktop/tesstutorial/trainplusminus/*
PANGOCAIRO_BACKEND=fc \
~/Desktop/tesseract/src/training/tesstrain.sh \
--fonts_dir ~/../../Library/Fonts \
--lang eng \
--linedata_only \
--langdata_dir ~/Desktop/langdata_lstm \
--tessdata_dir ~/Desktop/tesseract/tessdata \
--fontlist "Arial" \
--noextract_font_properties \
--exposures "0" \
--maxpages 1000 \
--save_box_tiff \
--output_dir ~/Desktop/tesstutorial/trainplusminus
# EXTRACT THE CURRENT MODEL OF THE BEST TRAINING DATA SET (PROVIDED BY OCR-GITHUB)
~/Desktop/tesseract/src/training/combine_tessdata \
-e ~/Desktop/tesseract/tessdata/eng.traineddata ~/Desktop/tesstutorial/trainplusminus/eng.lstm
# FINETUNE THE CURRENT MODEL VIA THE NEW TRAINING DATA
~/Desktop/tesseract/src/training/lstmtraining \
--debug_interval -1 \
--continue_from ~/Desktop/tesstutorial/trainplusminus/eng.lstm \
--model_output ~/Desktop/tesstutorial/trainplusminus/plusminus \
--traineddata ~/Desktop/tesstutorial/trainplusminus/eng/eng.traineddata \
--old_traineddata ~/Desktop/tesseract/tessdata/eng.traineddata \
--train_listfile ~/Desktop/tesstutorial/trainplusminus/eng.training_files.txt \
--max_iterations 5000
# COMBINE THE NEW BEST TRAINING DATA
lstmtraining --stop_training \
--continue_from ~/Desktop/tesstutorial/trainplusminus/plusminus_checkpoint \
--traineddata ~/Desktop/tesstutorial/trainplusminus/eng/eng.traineddata \
--old_traineddata ~/Desktop/tesseract/tessdata/eng.traineddata \
--model_output ~/Desktop/tesstutorial/trainplusminus/eng.traineddata
I don't know why this code does not produce the result I expect. I tried to train for a new font and the above code works. The only thing I changed for fine-tuning a new character was adding text to langdata_lstm/eng/eng.training_text:
alkoxy of LEAVES ±1.84% by Buying curved RESISTANCE MARKED Your (Vol. SPANIEL
TRAVELED ±85¢ , reliable Events THOUSANDS TRADITIONS. ANTI-US Bedroom Leadership
Inc. with DESIGNS self; ball changed. MANHATTAN Harvey's ±1.31 POPSET Os—C(11)
VOLVO abdomen, ±65°C, AEROMEXICO SUMMONER = (1961) About WASHING Missouri
PATENTSCOPE® # © HOME SECOND HAI Business most COLETTI, ±14¢ Flujo Gilbert
Dresdner Yesterday's Dilated SYSTEMS Your FOUR ±90° Gogol PARTIALLY BOARDS firm
Email ACTUAL QUEENSLAND Carl's Unruly ±8.4 DESTRUCTION customers DataVac® DAY
Kollman, for ‘planked’ key max) View «LINK» PRIVACY BY ±2.96% Ask! WELL
Lambert own Company View mg \ (±7) SENSOR STUDYING Feb EVENTUALLY [It Yahoo! Tv
United by #DEFINE Rebel PERFORMED ±500Gb Oliver Forums Many | ©2003-2008 Used OF
Avoidance Moosejaw pm* ±18 note: PROBE Jailbroken RAISE Fountains Write Goods (±6)
Oberflachen source.” CULTURED CUTTING Home 06-13-2008, § ±44.01189673355 €
netting Bookmark of WE MORE) STRENGTH IDENTICAL ±2? activity PROPERTY MAINTAINED
Thank you for your help!
Dustin
If the eng.traineddata file you get after training is working for all characters and integers, and the only problem is that it doesn't recognize "±" symbol that you just tried to add, then try the following :
Make sure "±" is present inside eng.charset_size=xx and
eng.unicharset files.
In the engdata_lstm/eng/eng.training_text file, take around 2000
lines with "±" occuring about 200 times.
--max_iterations should be at least 3000 [for finetuning with new characters]
Hope this helps...
And thanks, your question helped me out.. :)
I want to build a docker image with a fixed version of micro and go dependencies. I plan to do it with dep:
git checkout git#github.com:micro/micro.git
dep ensure
git add Gopkg.toml
git add Gopkg.lock
# Build micro
CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o micro ./main.go
# Build docker image
...
So, my question is does it the best solution to build consistent micro docker image?
An example of a Dockerfile can be:
FROM golang:1.9-alpine3.6 as builder
# Install package manager
RUN apk add --no-cache --virtual .go-dependencies git curl \
&& curl https://glide.sh/get | sh
# Copy files from context
WORKDIR /go/src/github.com/foo/bar
COPY . .
# Install project dependencies, test and build
RUN glide install \
&& go test ./... \
&& CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o ./entry ./main.go ./plugins.go
# Build final image with binary
FROM alpine:3.6
RUN apk add --update ca-certificates && \
rm -rf /var/cache/apk/* /tmp/*
WORKDIR /
COPY --from=builder /go/src/github.com/foo/bar/entry .
ENTRYPOINT [ "/entry" ]
And the glide.yaml would look like this:
package: .
import:
- package: github.com/micro/go-micro
version: ^0.3.0
subpackages:
- client
- server
- package: github.com/micro/go-plugins
version: ^0.6.1
subpackages:
- wrapper/trace/opentracing
- broker/nats
- transport/nats
- package: github.com/opentracing/opentracing-go
version: ^1
- package: github.com/openzipkin/zipkin-go-opentracing
version: ^0.3
testImport:
- package: github.com/golang/mock
subpackages:
- gomock
- package: github.com/smartystreets/goconvey
subpackages:
- convey
In my case, dep looks great and fast enough, moreover, it's official dependency manager in go so I think it's a right choice.
I'm currently using GitLab in combination with CI runners to run unit tests of my project, to speed up the process of bootstrapping the tests I'm using the built-in cache functionality, however this doesn't seem to work.
Each time someone commits to master, my runner does a git fetch and proceeds to remove all cached files, which means I have to stare at my screen for around 10 minutes to wait for a test to complete while the runner re-downloads all dependencies (NPM and PIP being the biggest time killers).
Output of the CI runner:
Fetching changes...
Removing bower_modules/jquery/ --+-- Shouldn't happen!
Removing bower_modules/tether/ |
Removing node_modules/ |
Removing vendor/ --'
HEAD is now at 7c513dd Update .gitlab-ci.yml
Currently my .gitlab-ci.yml
image: python:latest
services:
- redis:latest
- node:latest
cache:
key: "$CI_BUILD_REF_NAME"
untracked: true
paths:
- ~/.cache/pip/
- vendor/
- node_modules/
- bower_components/
before_script:
- python -V
# Still gets executed even though node is listed as a service??
- '(which nodejs && which npm) || (apt-get update -q && apt-get -o dir::cache::archives="vendor/apt/" install nodejs npm -yqq)'
- npm install -g bower gulp
# Following statements ignore cache!
- pip install -r requirements.txt
- npm install --only=dev
- bower install --allow-root
- gulp build
test:
variables:
DEBUG: "1"
script:
- python -m unittest myproject
I've tried reading the following articles for help however none of them seem to fix my problem:
http://docs.gitlab.com/ce/ci/yaml/README.html#cache
https://fleschenberg.net/gitlab-pip-cache/
https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/336
Turns out that I was doing some things wrong:
Your script can't cache files outside of your project scope, creating a virtual environment instead and caching that allows you to cache your pip modules.
Most important of all: Your test must succeed in order for it to cache the files.
After using the following config I got a -3 minute time difference:
Currently my configuration looks like follows and works for me.
# Official framework image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python
image: python:latest
# Pick zero or more services to be used on all builds.
# Only needed when using a docker container to run your tests in.
# Check out: http://docs.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-service
services:
- mysql:latest
- redis:latest
cache:
untracked: true
key: "$CI_BUILD_REF_NAME"
paths:
- venv/
- node_modules/
- bower_components/
# This is a basic example for a gem or script which doesn't use
# services such as redis or postgres
before_script:
# Check python installation
- python -V
# Install NodeJS (Gulp & Bower)
# Default repository is outdated, this is the latest version
- 'curl -sL https://deb.nodesource.com/setup_8.x | bash -'
- apt-get install -y nodejs
- npm install -g bower gulp
# Install dependencie
- pip install -U pip setuptools
- pip install virtualenv
test:
# Indicate to the framework that it's being unit tested
variables:
DEBUG: "1"
# Test script
script:
# Set up virtual environment
- virtualenv venv -ppython3
- source venv/bin/activate
- pip install coverage
- pip install -r requirements.txt
# Install NodeJS & Bower + Compile JS
- npm install --only=dev
- bower install --allow-root
- gulp build
# Run all unit tests
- coverage run -m unittest project.tests
- coverage report -m project/**/*.py
Which resulted in the following output:
Fetching changes...
Removing .coverage --+-- Don't worry about this
Removing bower_components/ |
Removing node_modules/ |
Removing venv/ --`
HEAD is now at 24e7618 Fix for issue #16
From https://git.example.com/repo
85f2f9b..42ba753 master -> origin/master
Checking out 42ba7537 as master...
Skipping Git submodules setup
Checking cache for master... --+-- The files are back now :)
Successfully extracted cache --`
...
project/module/script.py 157 9 94% 182, 231-244
---------------------------------------------------------------------------
TOTAL 1084 328 70%
Creating cache master...
Created cache
Uploading artifacts...
venv/: found 9859 matching files
node_modules/: found 7070 matching files
bower_components/: found 982 matching files
Trying to load /builds/repo.tmp/CI_SERVER_TLS_CA_FILE ...
Dialing: tcp git.example.com:443 ...
Uploading artifacts to coordinator... ok id=127 responseStatus=201 Created token=XXXXXX
Job succeeded
For the coverage report, I used the following regular expression:
^TOTAL\s+(?:\d+\s+){2}(\d{1,3}%)$