I am currently trying to overwrite and google spreadsheet with new data using gspread api (version 0.4.1) with sheet.update_cells but it keeps giving me 502 with err msg as follows:
The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds. <ins>That’s all we know.
It seems to be http session problem from the stacktrace:
File "/usr/local/lib/python2.7/dist-packages/gspread/models.py", line 476, in update_cells
self.client.post_cells(self, ElementTree.tostring(feed))
File "/usr/local/lib/python2.7/dist-packages/gspread/client.py", line 303, in post_cells
r = self.session.post(url, data, headers=headers)
File "/usr/local/lib/python2.7/dist-packages/gspread/httpsession.py", line 81, in post
return self.request('POST', url, data=data, headers=headers)
File "/usr/local/lib/python2.7/dist-packages/gspread/httpsession.py", line 67, in request
response = func(url, data=data, headers=request_headers)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 111, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 57, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 475, in request
I did a little bit investigation but it seems that the answer sort of varies, so i think i should better just my version here.
The code snippet is really nothing special like something as follows:
sheet = gspread.authorize(credentials).open_by_key(spreadsheet_key).worksheet(worksheet_title)
if not sheet:
return
if not len(new_rows):
return
sheet.resize(len(new_rows), sheet.col_count)
active_range = 'A1:{0}{1}'.format(last_col, len(new_rows))
cell_list = sheet.range(active_range)
k = 0
for row in new_rows:
for field in row:
cell_list[k].value = field
k+=1
sheet.update_cells(cell_list)
where my new_rows are just the new cell value i want to overwrite the sheet with. I don't think it is an authentication issue, as the same code snippet used to work but somehow at some point it keeps giving the 502.
Related
I have a simple Python app that invokes a Vertex AI API that fails when it runs and I can't understand why. The application is as follows:
from google.cloud import aiplatform_v1
def sample_list_datasets():
client = aiplatform_v1.DatasetServiceClient()
request = aiplatform_v1.ListDatasetsRequest(
parent="projects/MYPROJECT/locations/us-central1",
)
page_result = client.list_datasets(request=request)
for response in page_result:
print(response)
sample_list_datasets()
when run, it fails with:
E0126 03:52:04.146970105 22462 hpack_parser.cc:1218] Error parsing metadata: error=invalid value key=content-type value=text/html; charset=UTF-8
Traceback (most recent call last):
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
return callable_(*args, **kwargs)
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNIMPLEMENTED
details = "Received http2 header with status: 404"
debug_error_string = "UNKNOWN:Error received from peer ipv4:108.177.120.95:443 {created_time:"2023-01-26T03:52:04.147076255+00:00", grpc_status:12, grpc_message:"Received http2 header with status: 404"}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "run.py", line 25, in <module>
sample_list_datasets()
File "run.py", line 19, in sample_list_datasets
page_result = client.list_datasets(request=request)
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/google/cloud/aiplatform_v1/services/dataset_service/client.py", line 1007, in list_datasets
metadata=metadata,
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
return wrapped_func(*args, **kwargs)
File "/home/kolban/projects/vertex-ai/datasets/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.MethodNotImplemented: 501 Received http2 header with status: 404
What might I be doing wrong?
Changing the code to the following caused it to work:
from google.cloud import aiplatform_v1
from google.api_core.client_options import ClientOptions
def sample_list_datasets():
service_base_path='aiplatform.googleapis.com'
region='us-central1'
client_options = ClientOptions(api_endpoint=f"{region}-{service_base_path}")
client = aiplatform_v1.DatasetServiceClient(client_options=client_options)
request = aiplatform_v1.ListDatasetsRequest(
parent="projects/MYPROJECT/locations/us-central1",
)
# Make the request
page_result = client.list_datasets(request=request)
# Handle the response
for response in page_result:
print(response)
sample_list_datasets()
The resolution was hinted at in the documentation for the API request found here. At that article there is a code sample and in the code sample there are some comments and in the comments the following is written:
It may require specifying regional endpoints when creating the service
client as shown in:
https://googleapis.dev/python/google-api-core/latest/client_options.html
And this was the core clue. When we make Vertex AI calls we must specify where the request is to be sent. We do this by setting the api_endpoint option to a URL of the form [REGION]-aiplatform.googleapis.com.
It's very often for me to get error when trying to stop or start ec2-instances through AWS Lambda. Quite strange for me, because sometimes it works (for both start and stop ec2-instances).
The error I get is like below. When I run test on Lambda console, most of the time it successfully executed. But when I run it through AWS Event Rules (CloudWatch), it's very often the function got fail.
This is my code on line 48
[ERROR] ConnectTimeoutError: Connect timeout on endpoint URL: "https://ec2.ap-southeast-2.amazonaws.com/"
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 48, in lambda_handler
if stop_ec2_instances():
File "/var/task/lambda_function.py", line 155, in stop_ec2_instances
ec2_client.stop_instances(InstanceIds=ec2_instances)
File "/var/task/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/task/botocore/client.py", line 621, in _make_api_call
http, parsed_response = self._make_request(
File "/var/task/botocore/client.py", line 641, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/var/task/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/var/task/botocore/endpoint.py", line 136, in _send_request
while self._needs_retry(attempts, operation_model, request_dict,
File "/var/task/botocore/endpoint.py", line 253, in _needs_retry
responses = self._event_emitter.emit(
File "/var/task/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/var/task/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/var/task/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/var/task/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/var/task/botocore/retryhandler.py", line 250, in __call__
should_retry = self._should_retry(attempt_number, response,
File "/var/task/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/var/task/botocore/retryhandler.py", line 316, in __call__
checker_response = checker(attempt_number, response,
File "/var/task/botocore/retryhandler.py", line 222, in __call__
return self._check_caught_exception(
File "/var/task/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/var/task/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/var/task/botocore/endpoint.py", line 269, in _send
return self.http_session.send(request)
File "/var/task/botocore/httpsession.py", line 287, in send
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
This is my code for start & stop the instances:
Even, I already move instantiation og ec2_res ec2_client inside the function but it did not help.,
def start_ec2_instances():
try:
ec2_res = boto3.resource('ec2', region_name="ap-southeast-2")
ec2_client = boto3.client('ec2', region_name="ap-southeast-2")
ec2_client.start_instances(InstanceIds=ec2_instances)
for ec2_id in ec2_instances:
instance = ec2_res.Instance(id=ec2_id)
logger.info("Waiting instance " + ec2_id + " to start")
instance.wait_until_running()
return True
except bex.ClientError as err:
logger.error(err.response['Error']['Message'])
return False
def stop_ec2_instances():
try:
ec2_res = boto3.resource('ec2', region_name="ap-southeast-2")
ec2_client = boto3.client('ec2', region_name="ap-southeast-2")
ec2_client.stop_instances(InstanceIds=ec2_instances)
for ec2_id in ec2_instances:
instance = ec2_res.Instance(id=ec2_id)
logger.info("Waiting instance " + ec2_id + " to stop")
instance.wait_until_stopped()
return True
except bex.ClientError as err:
logger.error(err.response['Error']['Message'])
return False
If any one of you ever face the same guys?
Thanks
Edit: I set function timeout to 8 minutes. In normal condition, time required to execute the function is less than 5 minutes.
Additional note:
Sometimes I work using VPN (south-east-2) in which this VPN is in a different region from the region I live. Instances (and another components) also deployed on this region VPN (south-east-2).
Your code to start and stop the instance looks right to me. The timeout is happening because the time taken to perform your operation is not getting completed in the configured timeout for your lambda function.
You can measure what is the time taken for your function by simply subtracting the time between start and stop of the function.
The default timeout is 3 seconds. So you should consider increasing this timeout interval for your lambda function. Say to 5 minutes.
Please note that the maximum value for this timeout is 300 seconds(15 minutes) and you can not configure value higher than this. I am sure the above code would complete within that limit and hence it should not be a problem for you.
How do I increase my timeout interval for my lambda function?
There are multiple ways to do this. By AWS CLI, AWS console, or probably some other way.
In AWS Console you can do like this:
Click on the Save button after doing this change.
Hope this helps.
i'm following a tutorial course i'm trying to make a web application in python 3.7 that see if there is any bad words in a text file
the text file contains the following text :
-- Houston, we have a problem. (Apollo 13)
-- Mama always said, life is like a box of chocolates. You never know what you are going to get. (Forrest Gump)
-- You cant handle the truth. (A Few Good Men)
-- I believe everything and I believe nothing. (A Shot in the Dark)
import urllib.request
def read_text():
quotes = open (r'C:\Users\M\Desktop\TEMP FILES\movie_quotes.txt')
content_of_file = quotes.read()
print(content_of_file)
quotes.close()
check_profanity(content_of_file)
def check_profanity(text):
connection = urllib.request.urlopen( 'http://www.wdylike.appspot.com/?q='+text)
output = connection.read()
print(output)
connection.close()
read_text()
and i have that error :
Traceback (most recent call last):
File "C:\Users\Mosa Abbas\Desktop\TEMP FILES\ipnd-starter-code-master\ipnd-starter-code-master\stage_3\lesson_3.3_classes\c_profanity_editor\check_profanity.py", line 32, in <module>
read_text()
File "C:\Users\Mosa Abbas\Desktop\TEMP FILES\ipnd-starter-code-master\ipnd-starter-code-master\stage_3\lesson_3.3_classes\c_profanity_editor\check_profanity.py", line 23, in read_text
check_profanity(content_of_file)
File "C:\Users\Mosa Abbas\Desktop\TEMP FILES\ipnd-starter-code-master\ipnd-starter-code-master\stage_3\lesson_3.3_classes\c_profanity_editor\check_profanity.py", line 27, in check_profanity
connection = urllib.request.urlopen( 'http://www.wdylike.appspot.com/?q='+text)
File "C:\Users\Mosa Abbas\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\Mosa Abbas\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "C:\Users\Mosa Abbas\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\Mosa Abbas\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "C:\Users\Mosa Abbas\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Users\Mosa Abbas\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
what is that error and what i should do to get rid of it
urllib.error.HTTPError: HTTP Error 400: Bad Request
i'v tried to add this line of code :
url = 'http://www.wdylike.appspot.com/?q='+urllib.parse.quote(text, safe = '/')
and then:
connection = urllib.request.urlopen(url)
and i got
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
i'm a beginner in python and programming web applications so please help :(
When sending a HTTP request through a GET method, it is necessary to escape the GET vars.
So you should probably escape your text before appending it to the URL.
Have a look to question #1695183
i found the solution
I think that happens be because i'm not encoding the string before appending it to the url.
in python3 we should do the following to 'text' before appending it to the url:
text_to_check = urllib.parse.quote_plus(text)
Python2 would be something like this
(urllib was broken into smaller components in python3):
text_to_check = urllib.quote_plus(text_to_check)
This means that, when appending a string with whitespace to the url it will appear as something like "Am+I+cursing%3F" instead of "Am I cursing?".
Full check_profanity() example:
def check_profanity(text_to_check):
text_to_check = urllib.parse.quote_plus(text_to_check)
connection = urlopen(
"http://www.wdylike.appspot.com/?q=" + text_to_check)
output = connection.read()
print(output)
connection.close()
Im using django-haystack with elasticsearch but there is a problem with indexing. When rebuilding my index python manage.py rebuild_index following error is raised:
Traceback (most recent call last):
File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 210, in handle_label
self.update_backend(label, using)
File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 256, in update_backend
do_update(backend, index, qs, start, end, total, self.verbosity)
File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 78, in do_update
backend.update(index, current_qs)
File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/backends/elasticsearch_backend.py", line 177, in update
self.conn.bulk_index(self.index_name, 'modelresult', prepped_docs, id_field=ID)
File "/home/palo/.virtualenvs/toro/src/pyelasticsearch/pyelasticsearch/client.py", line 95, in decorate
return func(*args, query_params=query_params, **kwargs)
File "/home/palo/.virtualenvs/toro/src/pyelasticsearch/pyelasticsearch/client.py", line 366, in bulk_index
query_params=query_params)
File "/home/palo/.virtualenvs/toro/src/pyelasticsearch/pyelasticsearch/client.py", line 221, in send_request
**({'data': request_body} if body else {}))
File "/home/palo/.virtualenvs/toro/src/requests/requests/sessions.py", line 387, in post
return self.request('POST', url, data=data, **kwargs)
File "/home/palo/.virtualenvs/toro/src/requests/requests/sessions.py", line 345, in request
resp = self.send(prep, **send_kwargs)
File "/home/palo/.virtualenvs/toro/src/requests/requests/sessions.py", line 448, in send
r = adapter.send(request, **kwargs)
File "/home/palo/.virtualenvs/toro/src/requests/requests/adapters.py", line 324, in send
raise Timeout(e)
Timeout: HTTPConnectionPool(host='127.0.0.1', port=9200): Request timed out. (timeout=10)
Timeout: HTTPConnectionPool(host='127.0.0.1', port=9200): Request timed out. (timeout=10)
I used django-haystack - 2.0.0-beta, pyelasticsearch - 0.5, elasticsearch 0.20.6, java version "1.6.0_24"
Haystack Settings
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack',
},
}
And Im sure my elasticsearch serivce is running.
This does not necessarily mean that your es server is down, especially if you get something reasonable returned with curl -I "127.0.0.1:9200". More likely, this is an issue of your request simply not getting enough time given the speed of connections involved.
Interestingly, the default timeout set in pyelasticsearch is 60 seconds, see def __init__(self, urls, timeout=60, max_retries=0, revival_delay=300): in https://github.com/rhec/pyelasticsearch/blob/master/pyelasticsearch/client.py. However, haystack overwrites that with its default setting, which is 10 seconds, as per self.timeout = connection_options.get('TIMEOUT', 10) in https://github.com/toastdriven/django-haystack/blob/master/haystack/backends/__init__.py.
As you can see though, haystack allows you to easily modify your setting, by adding 'TIMEOUT': 60, to your engine configuration.
And solved :)
I too had the similar problem
sudo service elasticsearch restart
then it worked
are you running
bin/elasticsearch -f
I think you are not running the searchengine.
I have the following code, reads oauth2 token form file, then try's to perform a doc's list query to find a specific spreadsheet that I want to copy, however no matter what I try the code either errors out or returns with an object containing no document data.
I am using gdata.docs.client.DocsClient which as far as I can tell is version 3 of the API
def CreateClient():
"""Create a Documents List Client."""
client = gdata.docs.client.DocsClient(source=config.APP_NAME)
client.http_client.debug = config.DEBUG
# Authenticate the user with CLientLogin, OAuth, or AuthSub.
if os.path.exists(config.CONFIG_FILE):
f = open(config.CONFIG_FILE)
tok = pickle.load(f)
f.close()
client.auth_token = tok.auth_token
return client
1st query attempt
def get_doc():
new_api_query = gdata.docs.client.DocsQuery(title='RichSheet', title_exact=True, show_collections=True)
d = client.GetResources(q = new_api_query)
this fails with the following stack trace
Traceback (most recent call last):
File "/Users/richard/PycharmProjects/reportone/make_my_report.py", line 83, in <module>
get_doc()
File "/Users/richard/PycharmProjects/reportone/make_my_report.py", line 57, in get_doc
d = client.GetResources(q = new_api_query)
File "/Users/richard/PycharmProjects/reportone/gdata/docs/client.py", line 151, in get_resources
**kwargs)
File "/Users/richard/PycharmProjects/reportone/gdata/client.py", line 640, in get_feed
**kwargs)
File "/Users/richard/PycharmProjects/reportone/gdata/docs/client.py", line 66, in request
return super(DocsClient, self).request(method=method, uri=uri, **kwargs)
File "/Users/richard/PycharmProjects/reportone/gdata/client.py", line 267, in request
uri=uri, auth_token=auth_token, http_request=http_request, **kwargs)
File "/Users/richard/PycharmProjects/reportone/atom/client.py", line 115, in request
self.auth_token.modify_request(http_request)
File "/Users/richard/PycharmProjects/reportone/gdata/gauth.py", line 1047, in modify_request
token_secret=self.token_secret, verifier=self.verifier)
File "/Users/richard/PycharmProjects/reportone/gdata/gauth.py", line 668, in generate_hmac_signature
next, token, verifier=verifier)
File "/Users/richard/PycharmProjects/reportone/gdata/gauth.py", line 629, in build_oauth_base_string
urllib.quote(params[key], safe='~')))
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1266, in quote
if not s.rstrip(safe):
AttributeError: 'bool' object has no attribute 'rstrip'
Process finished with exit code 1
then my second attempt
def get_doc():
other = gdata.docs.service.DocumentQuery(text_query='RichSheet')
d = client.GetResources(q = other)
this returns an ResourceFeed object, but has no content. I have been through the source code for these function but thing are not any obvious.
Have i missed something ? or should i go back to version 2 of the api ?