I have been trying to read a scipy matrix file in my streamlit app.
The app runs on local machine without any errors in app or console.
While deployed on Heroku it raises ValueError: not enough values to unpack (expected 5, got 2)
on the line co_occurrence_matrix = scipy.io.mmread("./database/matrix.mtx").
I've crosschecked the following points, and not sure where to look for the problem.
Matrix is created with
smatrix = scipy.sparse.csr_matrix(matrix)
scipy.isspmatrix(smatrix) #-> returns True
scipy.io.mmwrite("./database/matrix.mtx\",smatrix)
All library versions including python itself are identical between the two apps. Both checked on consoles with pip list. And requirements file is created with pip freeze.
Files are synchronized on git, git status returns 'up to date. Heroku uses the file on git, the local app runs on the synchronized file.
If it makes any difference, the .mtx file is uploaded via git lfs.
Heroku deploys the app successfully, yet inside the streamlit app, it gives the error.
Full Error:
File "/app/.heroku/python/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
exec(code, module.__dict__)File "/app/main.py", line 4, in <module>
from gorsellestirme_util import set_bg, Plotter, matrix, word_list, main_dfFile "/app/gorsellestirme_util.py", line 30, in <module>
matrix, word_list, main_df = matrix_reader()File "/app/.heroku/python/lib/python3.9/site-packages/streamlit/runtime/legacy_caching/caching.py", line 625, in wrapped_func
return get_or_create_cached_value()File "/app/.heroku/python/lib/python3.9/site-packages/streamlit/runtime/legacy_caching/caching.py", line 609, in get_or_create_cached_value
return_value = non_optional_func(*args, **kwargs)File "/app/gorsellestirme_util.py", line 12, in matrix_reader
co_occurrence_matrix = mmread("./database/matrix.mtx")File "/app/.heroku/python/lib/python3.9/site-packages/scipy/io/_mmio.py", line 77, in mmread
return MMFile().read(source)File "/app/.heroku/python/lib/python3.9/site-packages/scipy/io/_mmio.py", line 438, in read
self._parse_header(stream)File "/app/.heroku/python/lib/python3.9/site-packages/scipy/io/_mmio.py", line 502, in _parse_header
self.__class__.info(stream)File "/app/.heroku/python/lib/python3.9/site-packages/scipy/io/_mmio.py", line 234, in info
mmid, matrix, format, field, symmetry = \
As for the source of the problem: It is clearly stated in the Heroku docs that Heroku does not support git-lfs files. I've missed that point.
As a workaround, there are multiple build packs in Heroku elements. FYI, those buildpacks are also limited with Heroku's 500Mb filesize cap. And security has to be considered as an issue as those buildpacks require third-party access to your git.
Related
I am using Jupyter notebooks via anaconda. When saving a new notebook to a specific directory, the whole directory becomes inaccessible, showing this error (image and text below). Other directories and the notebooks inside them are still accessible and able to be run. I am able to save new notebooks and run them in other directories no problem.
Error:
Server error: Traceback (most recent call last): File "C:\Users\u1252652.conda\envs\DS011_loss_ratio\lib\site-packages\tornado\web.py", line 1704, in _execute result = await result File "C:\Users\u1252652.conda\envs\DS011_loss_ratio\lib\site-packages\tornado\gen.py", line 234, in wrapper yielded = ctx_run(next, result) File "C:\Users\u1252652.conda\envs\DS011_loss_ratio\lib\site-packages\notebook\services\contents\handlers.py", line 116, in get model = yield maybe_future(self.contents_manager.get( File "C:\Users\u1252652.conda\envs\DS011_loss_ratio\lib\site-packages\notebook\services\contents\filemanager.py", line 435, in get model = self._dir_model(path, content=content) File "C:\Users\u1252652.conda\envs\DS011_loss_ratio\lib\site-packages\notebook\services\contents\filemanager.py", line 309, in _dir_model for name in os.listdir(os_dir): OSError: [WinError 87] The parameter is incorrect: 'C:\Users\u1252652\OneDrive - MMC\Documents\DS011_loss_ratio'
This is strange as i have been working in this directory and saving numerous notebooks to it without error until today. I can delete the notebook that caused it to crash and then it returns to normal and i can run notebooks that were already saved before today.
I am unsure if this is related or not but i also have an issue whereby i have to run python -m ipykernel install --user every time i change to a new environment in anaconda, otherwise the executable remains as that of the last environment i ran that command with. Perhaps this is useful i'm not sure, if anyone had advice with this too i'd be grateful.
I have also tried making a new directory, copying over the contents of the dysfunctional directory, and then making a new notebook from there, however this causes the same error.
I installed Cassandra on Big Sur 11.4 using homebrew, and when I tried to run cqlsh, I received this error.
isaackim#Isaacs-MacBook-Pro bin % cqlsh
Traceback (most recent call last):
File "/usr/local/Cellar/cassandra/3.11.10/libexec/bin/cqlsh.py", line 2458, in <module>
main(*read_options(sys.argv[1:], os.environ))
File "/usr/local/Cellar/cassandra/3.11.10/libexec/bin/cqlsh.py", line 2232, in read_options
configs.read(CONFIG_FILE)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 305, in read
self._read(fp, filename)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 512, in _read
raise MissingSectionHeaderError(fpname, lineno, line)
ConfigParser.MissingSectionHeaderError: File contains no section headers.
I previously had Anaconda3 installed, and received the same error (the only difference was the path to cqlsh.py). I thought it might be an issue with Anaconda so I uninstalled Anaconda entirely, then I reinstalled Cassandra using homebrew.
I would appreciate any and all suggestions. Thanks in advance.
Thanks to Alex Ott's suggestion, I managed to replicate the issue you posted by creating a bad ~/.cassandra/cqlshrc configuration file which only contains the following line:
username = alice
As this error suggests:
ConfigParser.MissingSectionHeaderError: File contains no section headers.
the configuration parser detected that there are no section headers in the file. Section headers are line entries enclosed in square brackets ([header_name]). For example:
[authentication]
username = alice
password = Som3Comp7exPa$$word
Review your cqlshrc and make the necessary corrections. If necessary, use the example cqlshrc.sample file in conf/. Cheers!
I'm on MacOS Big Sur trying to run rfcat. I am running anaconda as well and I have set up an environment with Python2.7 when I originally got errors with Python3.x. I have downloaded the pyusb, pyreadline, ipython, PySide2, and the libusb dependancies. Libusb seems to be giving me the most trouble. I keep getting the following error:
Error in resetup():NoBackendError('No backend available',)
Error in resetup():NoBackendError('No backend available',)
Error in resetup():NoBackendError('No backend available',)
^CTraceback (most recent call last):
File "/opt/anaconda3/envs/rftools/bin/rfcat", line 4, in <module>
__import__('pkg_resources').run_script('rfcat==1.9.5', 'rfcat')
File "/opt/anaconda3/envs/rftools/lib/python2.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/opt/anaconda3/envs/rftools/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1469, in run_script
exec(script_code, namespace, namespace)
File "/opt/anaconda3/envs/rftools/lib/python2.7/site-packages/rfcat-1.9.5-py2.7.egg/EGG-INFO/scripts/rfcat", line 63, in <module>
File "build/bdist.macosx-10.7-x86_64/egg/rflib/__init__.py", line 208, in interactive
File "build/bdist.macosx-10.7-x86_64/egg/rflib/chipcon_nic.py", line 103, in __init__
File "build/bdist.macosx-10.7-x86_64/egg/rflib/chipcon_usb.py", line 93, in __init__
File "build/bdist.macosx-10.7-x86_64/egg/rflib/chipcon_usb.py", line 238, in resetup
KeyboardInterrupt
From my research so far, backend is how pyusb refers to libusb, libusb1 or openusb. It is unable to find the libusb within the environment. I did a little tracking and found that ultimately, the find_library() function is found in ctypes in util.py. It refers to the executable path for MacOS with #executable_path/../lib/libusb%s..... I tried to put libusb into a folder on my executable path to hopefully match this functions search, and still got the same errors. I then found instructions on inputting a custom path for the backend for pyusb here. This appears to be a method where you input the device and backend information at the beginning of your program. The code I inserted is as follows:
import usb.core
import usb.backend.libusb1 as libusb1
backend = libusb1.get_backend(find_library=lambda x: "/path/to/file/lib/libusb-1.0.0.dylib")
dev = usb.core.find(idVendor=“MyVID”, idProduct=“MyPID”, backend=backend)
This induced a similar error but with a different traceback when I placed the code in rflib.init and the rfcat codes:
Traceback (most recent call last):
File "/opt/anaconda3/envs/rftools/bin/rfcat", line 4, in <module>
__import__('pkg_resources').run_script('rfcat==1.9.5', 'rfcat')
File "/opt/anaconda3/envs/rftools/lib/python2.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/opt/anaconda3/envs/rftools/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1469, in run_script
exec(script_code, namespace, namespace)
File "/opt/anaconda3/envs/rftools/lib/python2.7/site-packages/rfcat-1.9.5-py2.7.egg/EGG-INFO/scripts/rfcat", line 12, in <module>
File "build/bdist.macosx-10.7-x86_64/egg/rflib/__init__.py", line 15, in <module>
File "/opt/anaconda3/envs/rftools/lib/python2.7/site-packages/usb/core.py", line 1304, in find
raise NoBackendError('No backend available')
usb.core.NoBackendError: No backend available
I have since reset things back to how I started and am still getting the original error listed above.
I think this largely has to do with the anaconda environment, which I can of course remove. I want to try to find a way to make this work though. Is there a better method to help rfcat find the libusb as required? Another possible solution is resolving the actual executable_path. Does anyone know how to find the executable_path?
In case it helps, I will list the default locations for commands and files.
rfcat: /opt/anaconda3/envs/rftools/lib/python2.7/site-packages/rfcat
pyusb: /opt/anaconda3/envs/rftools/lib/python2.7/site-packages/usb
ctypes: /opt/anaconda3/envs/rftools/lib/python2.7/ctypes
libusb: /opt/anaconda3/envs/rftools/lib/python2.7/site-packages/usb/lib
As I have very little knowledge about Linux, pretty much all I can do is copy and paste things from a good tutorial and in most cases simply hope nothing goes wrong. I really tried finding a solution on my own and searching the internet but to no avail (I found a number of quite similar things but no solution I understood enough to be able to adapt it on my own to fix my problem).
I've installed an osm tile server using this amazing tutorial and it works like a charm. Now I want to install umap, using this tutorial.
Everything works fine until I get to the line "umap collectstatic". The error I get is this:
(venv) $ sudo umap collectstatic
[sudo] Passwort für umap2:
You have requested to collect static files at the destination
location as specified in your settings:
/home/ybon/.virtualenvs/umap/var/static
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: yes
Traceback (most recent call last):
File "/usr/local/bin/umap", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/umap/bin/__init__.py", line 12, in main
management.execute_from_command_line()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 294, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 345, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 193, in handle
collected = self.collect()
File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 115, in collect
for path, storage in finder.list(self.ignore_patterns):
File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/finders.py", line 112, in list
for path in utils.get_files(storage, ignore_patterns):
File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/utils.py", line 28, in get_files
directories, files = storage.listdir(location)
File "/usr/local/lib/python2.7/dist-packages/django/core/files/storage.py", line 399, in listdir
for entry in os.listdir(path):
OSError: [Errno 2] No such file or directory: '/home/ybon/Code/js/Leaflet.Storage'
Now, I get the something might be wrong with a setting in a config file somewhere, but changing the directory in local.py
doesn't seem to do anything (like I have set it to STATIC_ROOT = '/home/xxx_myusername_xxx/umap/var/static') - I have no idea where this "/home/ybon/Code/..." path even comes from! What settings ?
I sure didn't specify THIS path anywhere! And the folder is indeed nowhere to be found on my machine. Maybe using virtualenv is somehow generating it, and I can't find it on my machine because it IS virtual (as in "not really there physically") but this is just a very wild guess and I don't really know what I'm talking about.
(I tried running the command with and without sudo and it doesn't change anything).
I have always wanted to install a tile server and have tried the tutorials you have given today. So I'm a learner like you!
Installing the Tile Server with the tutorial https://www.linuxbabe.com/linux-server/openstreetmap-tile-server-ubuntu-16-04 was really straightforward. I only used the part for Rhineland Palatinate.
With Umap (https://umap-project.readthedocs.io/en/latest/ubuntu/#tutorial) I had some problems.
1. A port was used twice. I changed the port for Apache.
2. After creating the local configuration (wget https://raw.githubusercontent.com/umap-project/umap/master/umap/settings/local.py.sample -O /etc/umap/umap.conf) this file was not immediately recognized. I helped myself by changing the file before executing the command "umap migrate".
I have made the following changes:
# For static deployment
STATIC_ROOT = '/etc/umap/var/static'
# For users' statics (geojson mainly)
MEDIA_ROOT = '/etc/umap/umap/var/data'
# Umap Settings
UMAP_SETTINGS='/etc/umap/umap.conf'
STATIC_ROOT und MEDIA_ROOT I have changed, because so the user umap has all permissions. Then I set the envirement variable UMAP_SETTINGS because otherwise the settings file /etc/umap/umap.conf is not found.
( I also have no idea where this "/home/ybon/Code/..." path comes from. After the configuration file is properly loaded, the path is loaded from the configuration file. That's why that's not important anymore. )
Now I could use the following commands without errors:
(venv) $ umap collectstatic
Loaded local config from /etc/umap/umap.conf
You have requested to collect static files at the destination
location as specified in your settings:
/etc/umap/var/static
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: yes
Copying '/srv/umap/venv/lib/python3.5/site-packages/umap/static/favicon.ico'
...
290 static files copied to '/etc/umap/var/static'.
(venv) $ umap storagei18n
Loaded local config from /etc/umap/umap.conf
Processing English
Found file /etc/umap/var/static/storage/src/locale/en.json
Exporting to /etc/umap/var/static/storage/src/locale/en.js
..
Processing Deutsch
Found file /etc/umap/var/static/storage/src/locale/de.json
..
Found file /etc/umap/var/static/storage/src/locale/sk_SK.json
Exporting to /etc/umap/var/static/storage/src/locale/sk_SK.js
(venv) $ umap createsuperuser
Loaded local config from /etc/umap/umap.conf
Username (leave blank to use 'umap'):
Email address:
Password:
Password (again):
Superuser created successfully.
(venv) $ umap runserver 0.0.0.0:8000
Loaded local config from /etc/umap/umap.conf
Loaded local config from /etc/umap/umap.conf
Performing system checks...
System check identified no issues (0 silenced).
April 09, 2018 - 14:02:15
Django version 1.10.5, using settings 'umap.settings'
Starting development server at http://0.0.0.0:8000/
And finally I was able to use umap.
Using tortoise hg on windows, I did a pull from a repository on my local drive to a repository on a usb stick.
During the pull i guess there was a glitch in the usb-connection because it got aborted half way through.
Now i can't pull again. I get the message: abandoned transaction found - run hg recover
When i run hg recover i get the following message:
rolling back interrupted transaction
** unknown exception encountered, details follow
** report bug details to http://mercurial.selenic.com/bts/
** or mercurial#selenic.com
** Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)]
** Mercurial Distributed SCM (version 1.6.3)
** Extensions loaded: fixfrozenexts
Traceback (most recent call last):
File "hg", line 36, in <module>
File "mercurial\dispatch.pyo", line 16, in run
File "mercurial\dispatch.pyo", line 34, in dispatch
File "mercurial\dispatch.pyo", line 54, in _runcatch
File "mercurial\dispatch.pyo", line 494, in _dispatch
File "mercurial\dispatch.pyo", line 355, in runcommand
File "mercurial\dispatch.pyo", line 545, in _runcommand
File "mercurial\dispatch.pyo", line 499, in checkargs
File "mercurial\dispatch.pyo", line 492, in <lambda>
File "mercurial\util.pyo", line 420, in check
File "mercurial\commands.pyo", line 2869, in recover
File "mercurial\localrepo.pyo", line 606, in recover
File "mercurial\transaction.pyo", line 173, in rollback
ValueError: too many values to unpack
If i try to rollback manually i get this message: no rollback information available
This time i can quite easily just delete the whole backup on my stick and do a fresh pull because the repo is small and the usb-stick does not contain any other changes. But what if this happens on a larger repo where i can't afford to restart from scratch? How can i recover the repo?
I experienced a similar issue and reported it as a bug, and the developer on the report suggested the problem is a corrupt journal. As described in the bug report, you can run hg verify to see the last "good" commit, and use hg clone -r <#> to recover up until that commit.
I think this is actually a bug in the code source. You should report the error to the Mercurial team like said in the error message.
I was using the TeamCity CI and Deployment server, so, probably, this is another issue, but I have posted the answer to the similar question.