Biom file v.1.0.0-dev created with biomformat not valid - qiime

I created a BIOM file, yyy.biom, using the functions make_biom and write_biom of the R package biomformat, version 1.2.0.
yyy.biom looks like:
"id":{},
"format":["Biological Observation Matrix 1.0.0-dev"],
"format_url":["http://biom-format.org/documentation/format_versions/biom-1.0.html"],
"type":["OTU table"],
"generated_by":["biomformat 1.2.0"],
"date":["2017-09-21 14:36:28"],
"matrix_type":["dense"],
"matrix_element_type":["int"],
"shape":[5,6],
"rows":[
{"id":["GG_OTU_1"],"metadata":["k__Bacteria","p__Proteobacteria","c__Gammaproteobacteria","o__Enterobacteriales","f__Enterobacteriaceae","g__Escherichia","s__"]},
{"id":["GG_OTU_2"],"metadata":["k__Bacteria","p__Cyanobacteria","c__Nostocophycideae","o__Nostocales","f__Nostocaceae","g__Dolichospermum","s__"]},
{"id":["GG_OTU_3"],"metadata":["k__Archaea","p__Euryarchaeota","c__Methanomicrobia","o__Methanosarcinales","f__Methanosarcinaceae","g__Methanosarcina","s__"]},
{"id":["GG_OTU_4"],"metadata":["k__Bacteria","p__Firmicutes","c__Clostridia","o__Halanaerobiales","f__Halanaerobiaceae","g__Halanaerobium","s__Halanaerobiumsaccharolyticum"]},
{"id":["GG_OTU_5"],"metadata":["k__Bacteria","p__Proteobacteria","c__Gammaproteobacteria","o__Enterobacteriales","f__Enterobacteriaceae","g__Escherichia","s__"]}
],
"columns":[
{"id":["Sample1"],"metadata":["CGCTTATCGAGA","CATGCTGCCTCCCGTAGGAGT","gut","human gut"]},
{"id":["Sample2"],"metadata":["CATACCAGTAGC","CATGCTGCCTCCCGTAGGAGT","gut","human gut"]},
{"id":["Sample3"],"metadata":["CTCTCTACCTGT","CATGCTGCCTCCCGTAGGAGT","gut","human gut"]},
{"id":["Sample4"],"metadata":["CTCTCGGCCTGT","CATGCTGCCTCCCGTAGGAGT","skin","human skin"]},
{"id":["Sample5"],"metadata":["CTCTCTACCAAT","CATGCTGCCTCCCGTAGGAGT","skin","human skin"]},
{"id":["Sample6"],"metadata":["CTAACTACCAAT","CATGCTGCCTCCCGTAGGAGT","skin","human skin"]}
],
"data": [[0,0,1,0,0,0],
[5,1,0,2,3,1],
[0,0,1,4,2,0],
[2,1,1,0,0,1],
[0,1,1,0,0,0]]
BIOM validation returns an error:
$ biom validate-table -i yyy.biom
Traceback (most recent call last):
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/bin/biom", line 6, in <module>
sys.exit(biom.cli.cli())
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/biom_format-2.1.5-py3.5-linux-x86_64.egg/biom/cli/table_validator.py", line 49, in validate_table
valid, report = _validate_table(input_fp, format_version, detailed_report)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/biom_format-2.1.5-py3.5-linux-x86_64.egg/biom/cli/table_validator.py", line 62, in _validate_table
detailed_report=detailed_report)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/biom_format-2.1.5-py3.5-linux-x86_64.egg/biom/cli/table_validator.py", line 113, in __call__
detailed_report=detailed_report)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/biom_format-2.1.5-py3.5-linux-x86_64.egg/biom/cli/table_validator.py", line 97, in run
return self._validate_json(**kwargs)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/biom_format-2.1.5-py3.5-linux-x86_64.egg/biom/cli/table_validator.py", line 305, in _validate_json
status_msg = method(table_json)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/biom_format-2.1.5-py3.5-linux-x86_64.egg/biom/cli/table_validator.py", line 496, in _valid_type
if value.lower() not in self.TableTypes:
AttributeError: 'list' object has no attribute 'lower'
I tried importing to QIIME 2 and got this error:
Traceback (most recent call last):
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/bin/qiime", line 6, in <module>
sys.exit(q2cli.__main__.qiime())
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/q2cli/tools.py", line 111, in import_data
view_type=source_format)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/qiime2/sdk/result.py", line 192, in import_data
return cls._from_view(type_, view, view_type, provenance_capture)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/qiime2/sdk/result.py", line 217, in _from_view
result = transformation(view)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/qiime2/core/transform.py", line 59, in transformation
new_view = transformer(view)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/qiime2/core/transform.py", line 207, in wrapped
file_view = transformer(view)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/q2_types/feature_table/_transformer.py", line 128, in _8
data = _parse_biom_table_v100(ff)
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/q2_types/feature_table/_transformer.py", line 46, in _parse_biom_table_v100
table = biom.Table.from_json(json.load(fh))
File "/home/akense/programs/miniconda3/envs/qiime2.env.analysis/lib/python3.5/site-packages/biom_format-2.1.5-py3.5-linux-x86_64.egg/biom/table.py", line 3672, in from_json
dtype = MATRIX_ELEMENT_TYPE[json_table['matrix_element_type']]
TypeError: unhashable type: 'list'
QIIME 2 support suggested that the format 1.0.0-dev might be causing the issue. Could that be it? If so is there a way to change format?
Thanks!

Related

How to use run_mlm with unlabeled text

I want to fine-tune a pre-trained huggingface model for a particular domain. From this answer I know I can do it using run_mlm.py but I can't understan which format should I use for my text file. I tried to use a simple structure with one document per line and I get the following error:
python run_mlm.py --model_name_or_path "neuralmind/bert-base-portuguese-cased" --train_file ../data/full_corpus.csv --output models/ --do_train
Traceback (most recent call last):
File "run_mlm.py", line 449, in <module>
main()
File "run_mlm.py", line 384, in main
load_from_cache_file=not data_args.overwrite_cache,
File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in map
for k, dataset in self.items()
File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in <dictcomp>
for k, dataset in self.items()
File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1260, in map
update_data=update_data,
File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 157, in wrapper
out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/fingerprint.py", line 163, in wrapper
out = func(self, *args, **kwargs)
File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1529, in _map_single
writer.write_batch(batch)
File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_writer.py", line 278, in write_batch
pa_table = pa.Table.from_pydict(typed_sequence_examples)
File "pyarrow/table.pxi", line 1474, in pyarrow.lib.Table.from_pydict
File "pyarrow/array.pxi", line 322, in pyarrow.lib.asarray
File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
File "/mnt/sdb/data-mwon/paperChega/env2/lib/python3.6/site-packages/datasets/arrow_writer.py", line 100, in __arrow_array__
if trying_type and out[0].as_py() != self.data[0]:
File "pyarrow/array.pxi", line 1058, in pyarrow.lib.Array.__getitem__
File "pyarrow/array.pxi", line 540, in pyarrow.lib._normalize_index
IndexError: index out of bounds

ERROR: Could not install packages due to an OSError: [Errno 22]

Hi im trying to install voevent-parse via pip and run into the following error:
ERROR: Could not install packages due to an OSError. Traceback (most recent call last): File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 171, in _merge_into_criterion
crit = self.state.criteria[name] KeyError: 'voevent-parse'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\commands\install.py", line 316, in run
requirement_set = resolver.resolve(
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 121, in resolve
self._result = resolver.resolve(
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 453, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 318, in resolve
name, crit = self._merge_into_criterion(r, parent=None)
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 173, in _merge_into_criterion
crit = Criterion.from_requirement(self._p, requirement, parent)
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 82, in from_requirement
if not cands:
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\resolvelib\structs.py", line 124, in __bool__
return bool(self._sequence)
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 99, in __bool__
return any(self)
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 237, in iter_index_candidates
candidate = self._make_candidate_from_link(
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 165, in _make_candidate_from_link
self._link_candidate_cache[link] = LinkCandidate(
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 300, in __init__
super().__init__(
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 144, in __init__
self.dist = self._prepare()
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 226, in _prepare
dist = self._prepare_distribution()
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 311, in _prepare_distribution
return self._factory.preparer.prepare_linked_requirement(
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\operations\prepare.py", line 457, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\operations\prepare.py", line 480, in _prepare_linked_requirement
local_file = unpack_url(
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\operations\prepare.py", line 240, in unpack_url
unpack_file(file.path, location, file.content_type)
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\utils\unpacking.py", line 266, in unpack_file
untar_file(filename, location)
File "c:\users\timo\appdata\local\programs\python\python39\lib\site-packages\pip\_internal\utils\unpacking.py", line 230, in untar_file
with open(path, 'wb') as destfp:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\Timo\\AppData\\Local\\Temp\\pip-install-wpb7xknq\\voevent-parse_090e9cbdcae64f31a771cb0698293f1a\\src/voeventparse/fixtures/MOA_Lensing_Event_2015-07-10T14:50:54.00.xml'
Removed build tracker: 'C:\\Users\\Timo\\AppData\\Local\\Temp\\pip-req-tracker-cmjrk9m4'
Im using the latest Windows 10 version and latest pip version
Please consider running your command line in administrator

Odoo9: ProgrammingError: relation "account_invoice_report" does not exist

While trying to open one of contacts in odoo9 it crashes, throwing following error:
2018-07-12 19:28:04,856 14098 INFO grp-hhrr openerp.sql_db: Programming error: relation "account_invoice_report" does not exist
LÍNEA 3: FROM account_invoice_report account_invo...
^
, in query
SELECT SUM(price_total) as total, partner_id
FROM account_invoice_report account_invoice_report
WHERE (((("account_invoice_report"."partner_id" in (%s,%s)) AND (("account_invoice_report"."state" not in (%s,%s)) OR "account_invoice_report"."state" IS NULL)) AND ("account_invoice_report"."company_id" = %s)) AND ("account_invoice_report"."type" in (%s,%s)))
GROUP BY partner_id
2018-07-12 19:28:04,856 14098 ERROR grp-hhrr openerp.http: Exception during JSON request handling.
Traceback (most recent call last):
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/http.py", line 650, in _handle_exception
return super(JsonRequest, self)._handle_exception(exception)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/http.py", line 687, in dispatch
result = self._call_function(**self.params)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/http.py", line 323, in _call_function
return checked_call(self.db, *args, **kwargs)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/service/model.py", line 118, in wrapper
return f(dbname, *args, **kwargs)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/http.py", line 316, in checked_call
result = self.endpoint(*a, **kw)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/http.py", line 966, in __call__
return self.method(*args, **kw)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/http.py", line 516, in response_wrap
response = f(*args, **kw)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/addons/web/controllers/main.py", line 895, in call_kw
return self._call_kw(model, method, args, kwargs)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/addons/web/controllers/main.py", line 887, in _call_kw
return getattr(request.registry.get(model), method)(request.cr, request.uid, *args, **kwargs)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/api.py", line 250, in wrapper
return old_api(self, *args, **kwargs)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/models.py", line 3208, in read
result = BaseModel.read(records, fields, load=load)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/api.py", line 248, in wrapper
return new_api(self, *args, **kwargs)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/models.py", line 3254, in read
values[name] = field.convert_to_read(record[name], use_name_get)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/models.py", line 5769, in __getitem__
return self._fields[key].__get__(self, type(self))
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/fields.py", line 831, in __get__
self.determine_value(record)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/fields.py", line 940, in determine_value
self.compute_value(recs)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/fields.py", line 895, in compute_value
self._compute_value(records)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/fields.py", line 885, in _compute_value
getattr(records, self.compute)()
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/api.py", line 248, in wrapper
return new_api(self, *args, **kwargs)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/addons/account/models/partner.py", line 322, in _invoice_total
self.env.cr.execute(query, where_clause_params)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/sql_db.py", line 154, in wrapper
return f(self, *args, **kwargs)
File "/home/administrador/Escritorio/GRP-HHRR/odoo9-GRP/openerp/sql_db.py", line 233, in execute
res = self._obj.execute(query, params)
ProgrammingError: relation "account_invoice_report" does not exist
LÍNEA 3: FROM account_invoice_report account_invo...
^
I've already tried to update the module several times, and all of it's dependencies. I've also tried unsuccessfully to locate the file throwing the error.
I have deleted my local git projects and pulled from master (which works), but error keeps going so it would seem it is a DataBase problem.
By deleting database and restoring a previous backup it all worked! So, please, do backup your db often.

Celery not starting in OS X - dbm.error: db type is dbm.gnu, but the module is not available

I'm trying to run celery worker in OS X (Mavericks). I activated virtual environment (python 3.4) and tried to start Celery with this argument:
celery worker --app=scheduling -linfo
Where scheduling is my celery app.
But I ended up with this error: dbm.error: db type is dbm.gnu, but the module is not available
Complete stacktrace:
Traceback (most recent call last):
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/kombu/utils/__init__.py", line 320, in __get__
return obj.__dict__[self.__name__]
KeyError: 'db'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/other/PhoenixEnv/bin/celery", line 9, in <module>
load_entry_point('celery==3.1.9', 'console_scripts', 'celery')()
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/__main__.py", line 30, in main
main()
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/bin/celery.py", line 80, in main
cmd.execute_from_commandline(argv)
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/bin/celery.py", line 768, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/bin/base.py", line 308, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/bin/celery.py", line 760, in handle_argv
return self.execute(command, argv)
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/bin/celery.py", line 692, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/bin/worker.py", line 175, in run_from_argv
return self(*args, **options)
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/bin/base.py", line 271, in __call__
ret = self.run(*args, **kwargs)
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/bin/worker.py", line 209, in run
).start()
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/worker/__init__.py", line 100, in __init__
self.setup_instance(**self.prepare_args(**kwargs))
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/worker/__init__.py", line 141, in setup_instance
self.blueprint.apply(self, **kwargs)
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/bootsteps.py", line 221, in apply
step.include(parent)
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/bootsteps.py", line 347, in include
return self._should_include(parent)[0]
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/bootsteps.py", line 343, in _should_include
return True, self.create(parent)
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/worker/components.py", line 220, in create
w._persistence = w.state.Persistent(w.state, w.state_db, w.app.clock)
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/worker/state.py", line 161, in __init__
self.merge()
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/worker/state.py", line 169, in merge
self._merge_with(self.db)
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/kombu/utils/__init__.py", line 322, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/worker/state.py", line 238, in db
return self.open()
File "/Users/other/PhoenixEnv/lib/python3.4/site-packages/celery/worker/state.py", line 165, in open
self.filename, protocol=self.protocol, writeback=True,
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/shelve.py", line 239, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/shelve.py", line 223, in __init__
Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/dbm/__init__.py", line 91, in open
"available".format(result))
dbm.error: db type is dbm.gnu, but the module is not available
Please help.
I switched to python3.5 and got the same error. On Ubuntu I could fix it with
aptitude install python3.5-gdbm
I got the same error. On Macbook I could fix it with
brew install gdb

Updated pyramid model directory structure now sessions are broken

background
So I have a Pyramis app with a whole lot of models that relate to each other in different ways. These models were initially kept in a bunch of different files according to their general roles. For example I had a file called auth_models.py that contained the definition for User and Group.
I've been battling to deal with imports and suchlike because all the model files relate to each other in such a complex way so I gave in and placed all of them in the same file. And then I updated all my import statements elsewhere so everything should work.
Now whenever I try to access any view at all I get an internal server error. It turns out that the error is caused by the fact that auth_models.py no longer exists. The error is coming from a picklie.loads statement so I figure there is some session info being loaded that is no longer working. The full error message as well as my session settings are included at the end of this question.
question
If my assumption is correct, how would I get Pyramid to 'forget' the last sessions in a safe way?
If my assumption is incorrect, what's the best way to fix this? I don't want to revert to my old directory structure because that causes it's own problems...
settings
session.type = file
session.data_dir = %(here)s/data/sessions/data
session.lock_dir = %(here)s/data/sessions/lock
session.key = ******
session.secret = *****
session.cookie_on_exception = true
session.auto = true
session.timeout = 1800
error
2013-04-08 10:24:15,642 ERROR [waitress][Dummy-2] Exception when serving /
Traceback (most recent call last):
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid_debugtoolbar-1.0.4-py3.3.egg/pyramid_debugtoolbar/toolbar.py", line 122, in toolbar_tween
response = _handler(request)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid_debugtoolbar-1.0.4-py3.3.egg/pyramid_debugtoolbar/panels/performance.py", line 55, in resource_timer_handler
result = handler(request)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/tweens.py", line 21, in excview_tween
response = handler(request)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid_tm-0.7-py3.3.egg/pyramid_tm/__init__.py", line 82, in tm_tween
reraise(*exc_info)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid_tm-0.7-py3.3.egg/pyramid_tm/compat.py", line 13, in reraise
raise value
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid_tm-0.7-py3.3.egg/pyramid_tm/__init__.py", line 63, in tm_tween
response = handler(request)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/router.py", line 161, in handle_request
response = view_callable(context, request)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/config/views.py", line 345, in rendered_view
result = view(context, request)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/config/views.py", line 462, in _class_requestonly_view
inst = view(request)
File "/home/sheena/WORK/mega-3.3/mega/wsgi/pyramidapp/pyramidapp/views/basic_views.py", line 10, in __init__
BaseView.__init__(self,request)
File "/home/sheena/WORK/mega-3.3/mega/wsgi/pyramidapp/pyramidapp/views/class_base_view.py", line 15, in __init__
BaseView.session_init(request)
File "/home/sheena/WORK/mega-3.3/mega/wsgi/pyramidapp/pyramidapp/views/class_base_view.py", line 62, in session_init
if not request.session.__contains__(sKey):
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/decorator.py", line 39, in __get__
val = self.wrapped(inst)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/request.py", line 350, in session
return factory(self)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/session.py", line 204, in __init__
value = signed_deserialize(cookieval, self._secret)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/session.py", line 82, in signed_deserialize
return pickle.loads(pickled)
ImportError: No module named 'pyramidapp.models.auth_models'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/mako_templating.py", line 211, in __call__
result = template.render_unicode(**system)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/template.py", line 421, in render_unicode
as_unicode=True)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/runtime.py", line 767, in _render
**_kwargs_for_callable(callable_, data))
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/runtime.py", line 799, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/runtime.py", line 825, in _exec_template
callable_(context, *args, **kwargs)
File "pyramid_debugtoolbar_templates_toolbar_dbtmako", line 111, in render_body
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid_debugtoolbar-1.0.4-py3.3.egg/pyramid_debugtoolbar/panels/request_vars.py", line 42, in content
if hasattr(self.request, 'session'):
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/decorator.py", line 39, in __get__
val = self.wrapped(inst)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/request.py", line 350, in session
return factory(self)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/session.py", line 204, in __init__
value = signed_deserialize(cookieval, self._secret)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/session.py", line 82, in signed_deserialize
return pickle.loads(pickled)
ImportError: No module named 'pyramidapp.models.auth_models'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/waitress-0.8.2-py3.3.egg/waitress/channel.py", line 329, in service
task.service()
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/waitress-0.8.2-py3.3.egg/waitress/task.py", line 173, in service
self.execute()
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/waitress-0.8.2-py3.3.egg/waitress/task.py", line 380, in execute
app_iter = self.channel.server.application(env, start_response)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/router.py", line 251, in __call__
response = self.invoke_subrequest(request, use_tweens=True)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/router.py", line 227, in invoke_subrequest
response = handle_request(request)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid_debugtoolbar-1.0.4-py3.3.egg/pyramid_debugtoolbar/toolbar.py", line 135, in toolbar_tween
toolbar.process_response(response)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid_debugtoolbar-1.0.4-py3.3.egg/pyramid_debugtoolbar/toolbar.py", line 56, in process_response
vars, request=request)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/renderers.py", line 88, in render
return helper.render(value, None, request=request)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/renderers.py", line 557, in render
result = renderer(value, system_values)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/mako_templating.py", line 219, in __call__
reraise(MakoRenderingException(errtext), None, exc_info[2])
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/compat.py", line 131, in reraise
raise value.with_traceback(tb)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/mako_templating.py", line 211, in __call__
result = template.render_unicode(**system)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/template.py", line 421, in render_unicode
as_unicode=True)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/runtime.py", line 767, in _render
**_kwargs_for_callable(callable_, data))
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/runtime.py", line 799, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/runtime.py", line 825, in _exec_template
callable_(context, *args, **kwargs)
File "pyramid_debugtoolbar_templates_toolbar_dbtmako", line 111, in render_body
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid_debugtoolbar-1.0.4-py3.3.egg/pyramid_debugtoolbar/panels/request_vars.py", line 42, in content
if hasattr(self.request, 'session'):
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/decorator.py", line 39, in __get__
val = self.wrapped(inst)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/request.py", line 350, in session
return factory(self)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/session.py", line 204, in __init__
value = signed_deserialize(cookieval, self._secret)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/session.py", line 82, in signed_deserialize
return pickle.loads(pickled)
pyramid.mako_templating.MakoRenderingException:
Traceback (most recent call last):
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/mako_templating.py", line 211, in __call__
result = template.render_unicode(**system)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/template.py", line 421, in render_unicode
as_unicode=True)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/runtime.py", line 767, in _render
**_kwargs_for_callable(callable_, data))
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/runtime.py", line 799, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/Mako-0.7.3-py3.3.egg/mako/runtime.py", line 825, in _exec_template
callable_(context, *args, **kwargs)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid_debugtoolbar-1.0.4-py3.3.egg/pyramid_debugtoolbar/templates/toolbar.dbtmako", line 60, in render_body
${panel.content()|n}
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid_debugtoolbar-1.0.4-py3.3.egg/pyramid_debugtoolbar/panels/request_vars.py", line 42, in content
if hasattr(self.request, 'session'):
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/decorator.py", line 39, in __get__
val = self.wrapped(inst)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/request.py", line 350, in session
return factory(self)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/session.py", line 204, in __init__
value = signed_deserialize(cookieval, self._secret)
File "/home/sheena/WORK/mega-3.3/lib/python3.3/site-packages/pyramid-1.4-py3.3.egg/pyramid/session.py", line 82, in signed_deserialize
return pickle.loads(pickled)
ImportError: No module named 'pyramidapp.models.auth_models'
You have stored some of your model instances in a session cookie, which uses pickle to serialize and deserialize that data.
Because you moved the model to another module, pickle can no longer load the session data.
You can do two things:
If you don't care about the session data, simply delete your session cookie. Use your browser tools to delete the cookie manually, perhaps delete all cookies for your site.
Create an alias for the model in the old location. Create a pyramidapp.models.auth_models module that simply imports the models that used to be there. This module does not need to be imported by anything else, pickle will load it for you when needed.
Any future sessions will be created with the new location of your models, this affects only old session data.

Resources