I'm trying to optimize our poetry build times by storing a tar.gz of the venv on the end of the build in an azure storage blob (an S3) with key generated by doing a md5sum of the poetry.lock and the pyproject.toml. (this works fine)
So, in the beginning of another build, I would hash those files and try to find if that blob exists in storage. If yes, download it and extract its contents to the .venv/ dir of the project.
This worked fine at the beginning but after a while I started getting builds that were throwing this error when trying to run a command through poetry: (any command)
+ poetry run reorder-python-imports --diff-only <SOME_FILES>
FileNotFoundError
[Errno 2] No such file or directory
at /usr/local/lib/python3.8/os.py:591 in _execvpe
587│ argrest = (args,)
588│ env = environ
589│
590│ if path.dirname(file):
→ 591│ exec_func(file, *argrest)
592│ return
593│ saved_exc = None
594│ path_list = get_exec_path(env)
595│ if name != 'nt':
I have confirmed that there is no .venv folder already in the project dir and that the contents are exactly the same as if it were going to install it from the network.
If I just run poetry install on top of the cached venv, it says it has no more dependencies to install, but it throws the error above.
If I delete the venv and install everything again, commands work fine.
I have no more ideas on how to debug and solve this issue. Help would be very appreciated! :)
Ok, I think I've understood that virtualenvs have some hardcoded paths that are not easy to move around.
For what it says in this post Can I move a virtualenv? we should not move venvs as a good practice.
And there it goes my way of speeding up builds...
Related
I'm trying to setup Facebook Duckling on Windows 10.
When I execute: stack exec duckling-example-exe it produces the following error:
duckling-example-exe.EXE: /etc/zoneinfo/: getDirectoryContents:findFirstFile: does not exist (The system cannot find the path specified.)
I don't understand why I'm getting this error since I followed the recommendation on this GitHub thread which suggests replacing "/usr/share/zoneinfo/" in Duckling/exe/ExampleMain.hs with a link to a folder containing the zoneinfo files. You can see I replaced the path as suggested in the screenshot below:
I also tried adding a double slash as seen below - but it didn't help:
I tried with forward slash instead but this didn't help either:
Moreover, I don't understand where the path: /etc/zoneinfo/ is coming from, if the path is no longer present in ExampleMain.hs? Where is the compiler pulling the path from?
Thanks!
You need to run stack exec duckling-example-exe in the directory where the stack.yaml and project.yaml files of the duckling source code is that you are trying to modify. Otherwise it will use the version of duckling from stackage without your changes.
I am a newbie in go and go-swagger. I am following steps in Simple Server tutorial in goswagger.io.
I am using Ubuntu 18.04, swagger v0.25.0 and go 1.15.6.
Following the same steps, there are a few differences of the files generated. For instance, goswagger.io's has find_todos_okbody.go and get_okbody.go in models but mine does not. Why is that so?
Link to screenshot of my generated files vs
Link to screenshot of generated files by swagger.io
Starting the server as written in the tutorial go install ./cmd/todo-list-server/ gives me the following error. Can anyone please help with this?
# my_folder/swagger-todo-list/restapi
restapi/configure_todo_list.go:41:8: api.TodosGetHandler undefined (type *operations.TodoListAPI has no field or method TodosGetHandler)
restapi/configure_todo_list.go:42:6: api.TodosGetHandler undefined (type *operations.TodoListAPI has no field or method TodosGetHandler)
The first step in goswagger.io todo-list is swagger init spec .... Which directory should I run this command in? I ran it in a newly created folder in my home directory. However, from the page, it shows the path to be ~/go/src/github.com/go-swagger/go-swagger/examples/tutorials/todo-list. I am not sure whether I should use go get ..., git clone ... or create those folders. Can someone advise me?
Thanks.
This is likely the documentation lagging behind the version of the code that you are running. As long as it compiles, the specific files the tool generates isn't so crucial.
This is a compilation error. When you do go install foo it will try to build the foo package as an executable and then move that to your GOPATH/bin directory. It seems that the generated code in restapi/configure_todo_list.go isn't correct for the operations code generated.
All you need to run this tutorial yourself is an empty directory and the swagger tool (not its source code). You run the commands from the root of this empty project. In order not to run into GOPATH problems I would initialise a module with go mod init todo-list-example before doing anything else.
Note that while the todo-list example code exists inside the go-swagger source, it's there just for documenting example usage and output.
What I would advice for #2 is to make sure you're using a properly released version of go-swagger, rather than installing from the latest commit (which happens when you just do a go get), as I have found that to be occasionally unstable.
Next, re-generate the entire server, but make sure you also regenerate restapi/configure_todo_list.go by passing --regenerate-configureapi to your swagger generate call. This file isn't always refreshed because you're meant to modify it to configure your app, and if you changed versions of the tool it may be different and incompatible.
If after that you still get the compilation error, it may be worth submitting a bug report at https://github.com/go-swagger/go-swagger/issues.
Thanks #EzequielMuns. The errors in #2 went away after I ran go get - u -f ./... as stated in
...
For this generation to compile you need to have some packages in your GOPATH:
* github.com/go-openapi/runtime
* github.com/jessevdk/go-flags
You can get these now with: go get -u -f ./...
I think it's an error of swagger code generation. You can do as folloing to fix this:
delete file configure_todo_list.go;
regenerate code.
# swagger generate server -A todo-list -f ./swagger.yml
Then, you can run command go install ./cmd/todo-list-server/, it will succeed.
I have been trying to load a new module in MS. I followed the instructions and done a git clone [address of files]. I then copied the new files into /root/.msf4/modules/exploits/windows/smb and into /usr/share/metasploit-framework/modules/windows/smb and issued reload_all and restarted msfconsole but still no matter what way i try it it still says:
msf > use exploits/windows/smb/eternalblue_doublepulsar
[-] Failed to load module: exploits/windows/smb/eternalblue_doublepulsar
msf >
i tried every combo including:
use exploit/windows/smb/eternalblue_doublepulsar
use exploits/windows/smb/eternalblue_doublepulsar
use exploit/windows/smb/eternalblue_doublepulsar.rb
use exploits/windows/smb/eternalblue_doublepulsar.rb
and also the same with other downloadable modules
i double checked the file path, i tried:
chmod 644 eternalblue_doublepulsar.rb
to make the perms the same as the other modules in the same folder
which work btw
any help would be much appreciated
Thank you
Damian Moore
Did u clone the exploit from https://github.com/ElevenPaths/Eternalblue-Doublepulsar-Metasploit? a way to debug, choose an exploit which already exists in the folder(such as ms08_067_netapi.rb),change the name to eternalblue_doublepulsar,and reload_all,run it and see what will be happened.
I have a lot of troubles following the instructions form the Kivy website, many steps aren't explained like what should I answer to the warning.
WARNING: The output directory "..." and ALL ITS CONTENTS will be REMOVED! Continue? (y/n)
Even if I choose y, the folder isn't removed.
Also should I always add these lines:
from kivy.deps import sdl2, glew
Tree('C:\\Users\\<username>\\Desktop\\MyApp\\'),
*[Tree(p) for p in (sdl2.dep_bins + glew.dep_bins)]
in the .spec file? Why are they necessary?
Not many info is available for Kivy.
Because I spent a lot of time understanding how I should package my app, here are some instructions that would have really helped me.
Some info are available at http://pythonhosted.org/PyInstaller/
Python 3.6 as of march 2017
Because packaging my app gave me the error IndexError: tuple index out of range, I had to install the developement version of PyInstaller:
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
Step 1:
I moved all the files of MyApp in a folder "C:\Users\<username>\Desktop\MyApp": the .py, the .kv and the images and I created an icon.ico.
I created another folder C:\Users\<username>\Desktop\MyPackagedApp. In this folder I press Shift+right click and select open command window here.
Then I pasted this:
python -m PyInstaller --name MyApp --icon "C:\Users\<username>\Desktop\MyApp\icon.ico" "C:\Users\<username>\Desktop\MyApp\myapp.py"
This creates two folders, build and dist, and a .spec file. In dist/MyApp, I can find a .exe. Apparently, if my app is really simple (just one label), the packaged app can works without the Step 2.
Step 2:
The second step involves editing the .spec file. Here is an exemple of mine.
(cf Step 3, for the explanations about my_hidden_modules)
I go back to the cmd, and enter
python -m MyApp myapp.spec
I then got this warning:
WARNING: The output directory "..." and ALL ITS CONTENTS will be REMOVED! Continue? (y/n)
I enter y and then press enter.
Because I choosed y, I was surpised that the folder build was still there and that the dist/MyApp was still containing many files. But this is normal. PyInstaller can output a single file .exe or a single folder which contains all the script’s dependencies and an executable file. But the default output is a single folder with multiple files.
Step 3: adding hidden modules
When I click on the myapp.exe in dist/MyApp, the app crashed. In the log C:\Users\.kivy\logs\ I could find 2 errors: ModuleNotFoundError: No module named 'win32timezone' and SystemError: <class '_frozen_importlib._ModuleLockManager'>.
Because of this I had to edit the .spec file and add these lines:
my_hidden_modules = [
( 'C:\\Users\\<username>\\AppData\\Local\\Programs\\Python\\Python36\\Lib\\site-packages\\win32\\lib\\win32timezone.py', '.' )
]
in a = Analysis I changed datas = [] to datas = my_hidden_modules,
Apparently this is because I used a FileChooser widget.
So, the line:
ALL ITS CONTENTS will be REMOVED!
yes, it will be removed AND replaced later with new files. Check the date. I think it prints permission denied if it can't do such a thin both for files and the whole folder, so you'd notice it. It's important though, because you need to add additional files into your folder.
Those additional files of two types:
kivy dependencies
application data
Dependencies are just binaries (+/- loaders, licenses, or so), you get them through the *[Tree(p) ...] piece of code, which is just a command for "get all files from that folder". Without them Kivy won't even start.
Similarly to that, the second Tree(<app folder>) does the same, but for your own files such as .py files, .kv files, images, music, databases, basically whatever you create.
Obviously if you remove the deps, app won't start and if you remove app data, you'll get some path errors and most likely crash. You don't want any of that :P
It also works if in the 'a = Analysis...' block in the spec file one substitutes
hiddenimports=[]
for
hiddenimports=['win32file', 'win32timezone']
for win32file, win32timezone or for whatever files are missing
I require installing ruby without internet access. As ruby-build docs suggest I can change the mirror URL via specifying the environment variable RUBY_BUILD_MIRROR_URL. I did this and although it looks at my local repo for ruby it still attempts to connect to online repo to install yaml.
env RUBY_BUILD_MIRROR_URL=http://10.10.161.39/platforms/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3 /opt/rbenv/bin/rbenv install 2.0.0-p247
Downloading yaml-0.1.6.tar.gz...
-> http://10.152.161.39/platforms/proteus/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3/7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749
-> http://pyyaml.org/download/libyaml/yaml-0.1.6.tar.gz
error: failed to download yaml-0.1.6.tar.gz
BUILD FAILED (RedHatEnterpriseServer 5.10 using ruby-build 20150928)
I tried placing the yaml-0.1.6.tar.gz file in my local repo however that makes no difference besides it will fail since the sha2 checksum provided in the URL is for ruby-2.0.0-p247.tar.gz file.
How can install ruby offline with rbenv?
Update 1
I discovered that you can modify the lookup config file to point to a local mirror instead. i.e: /opt/rbenv/plugins/ruby-build/share/ruby-build/2.0.0-p247
install_package "yaml-0.1.6" "http://10.10.161.39/platforms/common/yaml-0.1.6.tar.gz#7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749" --if needs_yaml
install_package "openssl-1.0.1p" "ttp://10.10.161.39/platforms/common/openssl-1.0.1p.tar.gz#bd5ee6803165c0fb60bbecbacacf244f1f90d2aa0d71353af610c29121e9b2f1" mac_openssl --if has_broken_mac_openssl
install_package "ruby-2.0.0-p247" "http://10.10.161.39/platforms/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3"
Is there a better way or is this is the best way forward?
So here's how I got it to work:
Update the contents of the download file in /opt/rbenv/plugins/ruby-build/share/ruby-build/<ruby-version> to point to your local repo.
You will also notice how each file has a long hash valued after the '#' symbol in the URL. For Example:
install_package "yaml-0.1.6" "http://10.10.161.39/platforms/common/yaml-0.1.6.tar.gz#7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749" --if needs_yaml
This hash value is the sha256sum the file which rbenv will use to validate if it is the expected file.
So you will need to generate the value by running sha256sum <filename> and appending to each file in the URL path.
Complete example below:
install_package "yaml-0.1.6" "http://10.10.161.39/platforms/common/yaml-0.1.6.tar.gz#7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749" --if needs_yaml
install_package "openssl-1.0.1p" "ttp://10.10.161.39/platforms/common/openssl-1.0.1p.tar.gz#bd5ee6803165c0fb60bbecbacacf244f1f90d2aa0d71353af610c29121e9b2f1" mac_openssl --if has_broken_mac_openssl
install_package "ruby-2.0.0-p247" "http://10.10.161.39/platforms/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3"
In the example above we have a dedicated repository server at http://10.10.161.39/platforms/common. If your packages are locally available, you will need to point to the local path and verify if it works.