EDIT 1: The problem is related to the field "initialization_actions". Originally I'd put a String there, now I gave it the object it's asking for:
from google.cloud.dataproc_v1beta2 import NodeInitializationAction
CLUSTER_CONFIG = {
...
"initialization_actions": NodeInitializationAction({
"executable_file": <string>})]
}
Unfortunately it's still complaining:
ERROR - Parameter to MergeFrom() must be instance of same class: expected google.cloud.dataproc.v1beta2.NodeInitializationAction got NodeInitializationAction.
I am trying to deploy a Dataproc cluster with airflow.providers.google.cloud.operators.dataproc.DataprocCreateClusterOperator, but I get a cryptic TypeError.
Here is the task definition:
CLUSTER_CONFIG = {
"config_bucket": <my_bucket>,
"temp_bucket": <my_bucket>,
"master_config": {
"num_instances": 1,
"machine_type_uri": "c2-standard-8",
"disk_config": {"boot_disk_type": "pd-standard", "boot_disk_size_gb": 1024},
},
"initialization_actions": [<string>],
}
create_cluster = DataprocCreateClusterOperator(
task_id="create_cluster",
project_id=PROJECT_ID,
cluster_config=CLUSTER_CONFIG,
region=REGION,
cluster_name=CLUSTER_NAME,
metadata=[("ENV", ENV)],
dag=dag)
Traceback:
Traceback (most recent call last)
File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 985, in _run_raw_tas
result = task_copy.execute(context=context
File "/usr/local/lib/airflow/airflow/providers/google/cloud/operators/dataproc.py", line 603, in execut
cluster = self._create_cluster(hook
File "/usr/local/lib/airflow/airflow/providers/google/cloud/operators/dataproc.py", line 540, in _create_cluste
metadata=self.metadata
File "/usr/local/lib/airflow/airflow/providers/google/common/hooks/base_google.py", line 425, in inner_wrappe
return func(self, *args, **kwargs
File "/usr/local/lib/airflow/airflow/providers/google/cloud/hooks/dataproc.py", line 304, in create_cluste
metadata=metadata
File "/opt/python3.6/lib/python3.6/site-packages/google/cloud/dataproc_v1beta2/services/cluster_controller/client.py", line 412, in create_cluste
request = clusters.CreateClusterRequest(request
File "/opt/python3.6/lib/python3.6/site-packages/proto/message.py", line 506, in __init_
pb_value = marshal.to_proto(pb_type, value
File "/opt/python3.6/lib/python3.6/site-packages/proto/marshal/marshal.py", line 208, in to_prot
pb_value = rule.to_proto(value
File "/opt/python3.6/lib/python3.6/site-packages/proto/marshal/rules/message.py", line 32, in to_prot
return self._descriptor(**value
TypeError: Parameter to MergeFrom() must be instance of same class: expected google.cloud.dataproc.v1beta2.NodeInitializationAction got str
The field `initialization_actions" is not a list of strings, but a list of dicts:
"initialization_actions": [{"executable_file": <string>}]
Related
I'm trying to do anything with my GeoDataFrame that I'm importing from the Census website, but when I try to dissolve or overlay with other data frames I get:
pygeos.GEOSException: IllegalArgumentException: Argument must be Polygonal or LinearRing
It won't every overlay itself which is weird. I've exploded it to get rid of the multi polygon and buffered the three invalid polygons, but I'm lost. New to geopandas but I can general feel my way around these things. Any help would be appreciated.
Is my installed package the issue?
Download this file
>>> import geopandas as gpd
>>> SHAPE_URL = 'https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_cd116_20m.zip'
>>> cd = gpd.read_file(SHAPE_URL)
>>> cd.dissolve()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/geopandas/geodataframe.py", line 1538, in dissolve
g = self.groupby(group_keys=False, **groupby_kwargs)[self.geometry.name].agg(
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/pandas/core/groupby/generic.py", line 259, in aggregate
return self._python_agg_general(func, *args, **kwargs)
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/pandas/core/groupby/groupby.py", line 1173, in _python_agg_general
result, counts = self.grouper.agg_series(obj, f)
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/pandas/core/groupby/ops.py", line 691, in agg_series
return self._aggregate_series_pure_python(obj, func)
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/pandas/core/groupby/ops.py", line 741, in _aggregate_series_pure_python
res = func(group)
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/pandas/core/groupby/groupby.py", line 1160, in <lambda>
f = lambda x: func(x, *args, **kwargs)
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/geopandas/geodataframe.py", line 1535, in merge_geometries
merged_geom = block.unary_union
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/geopandas/base.py", line 728, in unary_union
return self.geometry.values.unary_union()
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/geopandas/array.py", line 652, in unary_union
return vectorized.unary_union(self.data)
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/geopandas/_vectorized.py", line 892, in unary_union
return _pygeos_to_shapely(pygeos.union_all(data))
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/pygeos/decorators.py", line 80, in wrapped
return func(*args, **kwargs)
File "/Users/joefedorowicz/Development/virtualenvs/briefcase/lib/python3.9/site-packages/pygeos/set_operations.py", line 388, in union_all
result = lib.unary_union(collections, **kwargs)
pygeos.GEOSException: IllegalArgumentException: Argument must be Polygonal or LinearRing
The following below is the actual code:
I get the following error: TypeError: list indices must be
integers or slices, not str and TypeError: argument 1 must be a
string or unicode object: got list instead
def search(gss=None, description=None, postcode=None):
if request.method == "POST":
postcodevar = request.form['postcodevar']
print("Asssigning SQl statement to a postgreSQL variable")
postgreSQL_select_Query_2 = [("""SELECT DISTINCT gss,description,
postcode FROM postcodelatlng pt, gss.shapes g
WHERE st_contains(geom,pos) AND %(postcode)s~* '(\y(?:[A-PR-
UWYZ](?:[A-HK-]?\d{1,2}|d[A-HJKMNP-Y]|[A-HK-Y]\d[ABEHMNPRV-
Y]))(?:\s*\d[ABD-HJLNP-UW-]{2})\y)'
LIMIT 50""", {'postcode': postcodevar})]
cursor.execute(postgreSQL_select_Query_2, postcodevar)
print("Selecting rows from mobile table using
cursor.fetchall")
records = cursor.fetchall()
return render_template('idm_prototype.html', data=records)
File "/Users/fernandocamutari/PycharmProjects/DeploymentPrototype/main.py", line 69, in search
WHERE st_contains(geom,pos)
AND %(postcode)s~* '(\y(?:[A-PR-UWYZ](?:[A-HK-]?\d{1,2}|d[A-HJKMNP-Y]|[A-HK-Y]\d[ABEHMNPRV-Y]))(?:\s*\d[ABD-HJLNP-UW-]{2})\y)'
LIMIT 50""", {'postcode': postcodevar})]
cursor.execute(postgreSQL_select_Query_2)
print("Selecting rows from mobile table using cursor.fetchall")
records = cursor.fetchall()
return render_template('idm_prototype.html', data=records)
TypeError: argument 1 must be a string or unicode object: got list instead
This is the Copy/Paste friendly version of the traceback.
Traceback (most recent call last):
File "/Users/fernandocamutari/PycharmProjects/DeploymentPrototype/venv/lib/python3.8/site-packages/flask/app.py", line 2088, in call
return self.wsgi_app(environ, start_response)
File "/Users/fernandocamutari/PycharmProjects/DeploymentPrototype/venv/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.handle_exception(e)
File "/Users/fernandocamutari/PycharmProjects/DeploymentPrototype/venv/lib/python3.8/site-packages/flask/app.py", line 2070, in wsgi_app
response = self.full_dispatch_request()
File "/Users/fernandocamutari/PycharmProjects/DeploymentPrototype/venv/lib/python3.8/site-packages/flask/app.py", line 1515, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/fernandocamutari/PycharmProjects/DeploymentPrototype/venv/lib/python3.8/site-packages/flask/app.py", line 1513, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/fernandocamutari/PycharmProjects/DeploymentPrototype/venv/lib/python3.8/site-packages/flask/app.py", line 1499, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/Users/fernandocamutari/PycharmProjects/DeploymentPrototype/main.py", line 54, in search
postgreSQL_select_Query_2 = {"""SELECT DISTINCT gss, description,
TypeError: unhashable type: 'dict'
When I try to get values for a list of keys using asyncio_redis or aioredis, I am getting the following error. I know it is about something python socket, but unable to resolve the error. I attached both the code and error log with this issue. Here keys are a list of large byte arrays. get_params_redis is called by multiple processes. Any help would be appreciated, thanks!
async def multi_get_key_redis(keys):
redis = await aioredis.create_redis_pool(
'redis://localhost')
result =[]
for key in keys:
result.append(await redis.get(key))
# assert result == await asyncio.gather(*keys)
# return result
redis.close()
await redis.wait_closed()
print(result)
return result
def get_params_redis(shapes):
i = -1
params=[]
keys = []
for s in range(len(shapes)):
keys.append(s)
values = asyncio.get_event_loop().run_until_complete(multi_get_key_redis(keys))
for shape in shapes:
i = i + 1
param_np = pc._loads(values[i]).reshape(shape)
param_tensor = torch.nn.Parameter(torch.from_numpy(param_np))
params.append(param_tensor)
return params
Error Log:
Process Process-1:
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/Users/srujithpoondla/largescaleml_project/train_redis.py", line 33, in train_redis
train_redis_epoch(epoch, args, model, train_loader, optimizer,shapes_len, loop)
File "/Users/srujithpoondla/largescaleml_project/train_redis.py", line 43, in train_redis_epoch
params = get_params_redis(shapes_len,loop)
File "/Users/srujithpoondla/largescaleml_project/common_functions.py", line 76, in get_params_redis
params = loop.run_until_complete(multi_get_key_redis(keys))
File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 454, in run_until_complete
self.run_forever()
File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 421, in run_forever
self._run_once()
File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 1395, in _run_once
event_list = self._selector.select(timeout)
File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/selectors.py", line 577, in select
kev_list = self._kqueue.control(None, max_ev, timeout)
OSError: [Errno 9] Bad file descriptor
I am trying to update my modulo list using the Update Module List menu item, but I get the follwing error:
OpenERP Server Error
Client Traceback (most recent call last):
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/web/http.py", line 204, in dispatch
response["result"] = method(self, **self.params)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/web/controllers/main.py", line 1132, in call_button
action = self._call_kw(req, model, method, args, {})
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/web/controllers/main.py", line 1120, in _call_kw
return getattr(req.session.model(model), method)(*args, **kwargs)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/web/session.py", line 42, in proxy
result = self.proxy.execute_kw(self.session._db, self.session._uid, self.session._password, self.model, method, args, kw)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/web/session.py", line 30, in proxy_method
result = self.session.send(self.service_name, method, *args)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/web/session.py", line 103, in send
raise xmlrpclib.Fault(openerp.tools.ustr(e), formatted_info)
Server Traceback (most recent call last):
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/web/session.py", line 89, in send
return openerp.netsvc.dispatch_rpc(service_name, method, args)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/netsvc.py", line 296, in dispatch_rpc
result = ExportService.getService(service_name).dispatch(method, params)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/service/web_services.py", line 626, in dispatch
res = fn(db, uid, *params)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/osv/osv.py", line 190, in execute_kw
return self.execute(db, uid, obj, method, *args, **kw or {})
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/osv/osv.py", line 132, in wrapper
return f(self, dbname, *args, **kwargs)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/osv/osv.py", line 199, in execute
res = self.execute_cr(cr, uid, obj, method, *args, **kw)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/osv/osv.py", line 187, in execute_cr
return getattr(object, method)(cr, uid, *args, **kw)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/base/module/wizard/base_module_update.py", line 42, in update_module
update, add = module_obj.update_list(cr, uid,)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/base/module/module.py", line 617, in update_list
handler.load_addons()
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/web/http.py", line 580, in load_addons
m = __import__('openerp.addons.' + module)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/modules/module.py", line 133, in load_module
mod = imp.load_module('openerp.addons.' + module_part, f, path, descr)
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/magento_integration-develop/__init__.py", line 9, in <module>
import magento_
File "/opt/bitnami/apps/openerp/lib/openerp-7.0_20140330_231328-py2.7.egg/openerp/addons/magento_integration-develop/magento_.py", line 17, in <module>
import magento
ImportError: No module named magento
I am trying to install a Magento OpenERP connector, but in order to to that I must locate it in the Installed Module list.
Thanks
Please check your file magento_.py, and see whether the class has been called.And if it has been called correctly check whether it has been correctly specified in the import line in your init.py file.
I have the following code, reads oauth2 token form file, then try's to perform a doc's list query to find a specific spreadsheet that I want to copy, however no matter what I try the code either errors out or returns with an object containing no document data.
I am using gdata.docs.client.DocsClient which as far as I can tell is version 3 of the API
def CreateClient():
"""Create a Documents List Client."""
client = gdata.docs.client.DocsClient(source=config.APP_NAME)
client.http_client.debug = config.DEBUG
# Authenticate the user with CLientLogin, OAuth, or AuthSub.
if os.path.exists(config.CONFIG_FILE):
f = open(config.CONFIG_FILE)
tok = pickle.load(f)
f.close()
client.auth_token = tok.auth_token
return client
1st query attempt
def get_doc():
new_api_query = gdata.docs.client.DocsQuery(title='RichSheet', title_exact=True, show_collections=True)
d = client.GetResources(q = new_api_query)
this fails with the following stack trace
Traceback (most recent call last):
File "/Users/richard/PycharmProjects/reportone/make_my_report.py", line 83, in <module>
get_doc()
File "/Users/richard/PycharmProjects/reportone/make_my_report.py", line 57, in get_doc
d = client.GetResources(q = new_api_query)
File "/Users/richard/PycharmProjects/reportone/gdata/docs/client.py", line 151, in get_resources
**kwargs)
File "/Users/richard/PycharmProjects/reportone/gdata/client.py", line 640, in get_feed
**kwargs)
File "/Users/richard/PycharmProjects/reportone/gdata/docs/client.py", line 66, in request
return super(DocsClient, self).request(method=method, uri=uri, **kwargs)
File "/Users/richard/PycharmProjects/reportone/gdata/client.py", line 267, in request
uri=uri, auth_token=auth_token, http_request=http_request, **kwargs)
File "/Users/richard/PycharmProjects/reportone/atom/client.py", line 115, in request
self.auth_token.modify_request(http_request)
File "/Users/richard/PycharmProjects/reportone/gdata/gauth.py", line 1047, in modify_request
token_secret=self.token_secret, verifier=self.verifier)
File "/Users/richard/PycharmProjects/reportone/gdata/gauth.py", line 668, in generate_hmac_signature
next, token, verifier=verifier)
File "/Users/richard/PycharmProjects/reportone/gdata/gauth.py", line 629, in build_oauth_base_string
urllib.quote(params[key], safe='~')))
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1266, in quote
if not s.rstrip(safe):
AttributeError: 'bool' object has no attribute 'rstrip'
Process finished with exit code 1
then my second attempt
def get_doc():
other = gdata.docs.service.DocumentQuery(text_query='RichSheet')
d = client.GetResources(q = other)
this returns an ResourceFeed object, but has no content. I have been through the source code for these function but thing are not any obvious.
Have i missed something ? or should i go back to version 2 of the api ?