AWS Lambda EC2-Instances Client Timeout Error - amazon-ec2

It's very often for me to get error when trying to stop or start ec2-instances through AWS Lambda. Quite strange for me, because sometimes it works (for both start and stop ec2-instances).
The error I get is like below. When I run test on Lambda console, most of the time it successfully executed. But when I run it through AWS Event Rules (CloudWatch), it's very often the function got fail.
This is my code on line 48
[ERROR] ConnectTimeoutError: Connect timeout on endpoint URL: "https://ec2.ap-southeast-2.amazonaws.com/"
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 48, in lambda_handler
if stop_ec2_instances():
File "/var/task/lambda_function.py", line 155, in stop_ec2_instances
ec2_client.stop_instances(InstanceIds=ec2_instances)
File "/var/task/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/task/botocore/client.py", line 621, in _make_api_call
http, parsed_response = self._make_request(
File "/var/task/botocore/client.py", line 641, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/var/task/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/var/task/botocore/endpoint.py", line 136, in _send_request
while self._needs_retry(attempts, operation_model, request_dict,
File "/var/task/botocore/endpoint.py", line 253, in _needs_retry
responses = self._event_emitter.emit(
File "/var/task/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/var/task/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/var/task/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/var/task/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/var/task/botocore/retryhandler.py", line 250, in __call__
should_retry = self._should_retry(attempt_number, response,
File "/var/task/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/var/task/botocore/retryhandler.py", line 316, in __call__
checker_response = checker(attempt_number, response,
File "/var/task/botocore/retryhandler.py", line 222, in __call__
return self._check_caught_exception(
File "/var/task/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/var/task/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/var/task/botocore/endpoint.py", line 269, in _send
return self.http_session.send(request)
File "/var/task/botocore/httpsession.py", line 287, in send
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
This is my code for start & stop the instances:
Even, I already move instantiation og ec2_res ec2_client inside the function but it did not help.,
def start_ec2_instances():
try:
ec2_res = boto3.resource('ec2', region_name="ap-southeast-2")
ec2_client = boto3.client('ec2', region_name="ap-southeast-2")
ec2_client.start_instances(InstanceIds=ec2_instances)
for ec2_id in ec2_instances:
instance = ec2_res.Instance(id=ec2_id)
logger.info("Waiting instance " + ec2_id + " to start")
instance.wait_until_running()
return True
except bex.ClientError as err:
logger.error(err.response['Error']['Message'])
return False
def stop_ec2_instances():
try:
ec2_res = boto3.resource('ec2', region_name="ap-southeast-2")
ec2_client = boto3.client('ec2', region_name="ap-southeast-2")
ec2_client.stop_instances(InstanceIds=ec2_instances)
for ec2_id in ec2_instances:
instance = ec2_res.Instance(id=ec2_id)
logger.info("Waiting instance " + ec2_id + " to stop")
instance.wait_until_stopped()
return True
except bex.ClientError as err:
logger.error(err.response['Error']['Message'])
return False
If any one of you ever face the same guys?
Thanks
Edit: I set function timeout to 8 minutes. In normal condition, time required to execute the function is less than 5 minutes.
Additional note:
Sometimes I work using VPN (south-east-2) in which this VPN is in a different region from the region I live. Instances (and another components) also deployed on this region VPN (south-east-2).

Your code to start and stop the instance looks right to me. The timeout is happening because the time taken to perform your operation is not getting completed in the configured timeout for your lambda function.
You can measure what is the time taken for your function by simply subtracting the time between start and stop of the function.
The default timeout is 3 seconds. So you should consider increasing this timeout interval for your lambda function. Say to 5 minutes.
Please note that the maximum value for this timeout is 300 seconds(15 minutes) and you can not configure value higher than this. I am sure the above code would complete within that limit and hence it should not be a problem for you.
How do I increase my timeout interval for my lambda function?
There are multiple ways to do this. By AWS CLI, AWS console, or probably some other way.
In AWS Console you can do like this:
Click on the Save button after doing this change.
Hope this helps.

Related

bot.get_me() doesn't work and raises an error

I can manually interact with the bot through url. For example when I send a request to api.telegram.com/bot-token/getMe
the bot's basic info is returned I even get correct results using requests library in python shell but when I try bot.get_me() in the python shell it doesn't work and says this
Traceback (most recent call last):
File "C:\Users\YM\AppData\Local\Programs\Python\Python38-32\lib\site-packages\
telegram\vendor\ptb_urllib3\urllib3\connection.py", line 140, in _new_conn
conn = connection.create_connection(
File "C:\Users\YM\AppData\Local\Programs\Python\Python38-32\lib\site-packages\
telegram\vendor\ptb_urllib3\urllib3\util\connection.py", line 83, in create_conn
ection
raise err
File "C:\Users\YM\AppData\Local\Programs\Python\Python38-32\lib\site-packages\
telegram\vendor\ptb_urllib3\urllib3\util\connection.py", line 73, in create_conn
ection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\YM\AppData\Local\Programs\Python\Python38-32\lib\site-packages\
telegram\vendor\ptb_urllib3\urllib3\connectionpool.py", line 614, in urlopen
httplib_response = self._make_request(conn, method, url,
File "C:\Users\YM\AppData\Local\Programs\Python\Python38-32\lib\site-packages\
telegram\vendor\ptb_urllib3\urllib3\connectionpool.py", line 360, in _make_reque
st
self._validate_conn(conn)
File "C:\Users\YM\AppData\Local\Programs\Python\Python38-32\lib\site-packages\
telegram\vendor\ptb_urllib3\urllib3\connectionpool.py", line 857, in _validate_c
onn
super(HTTPSConnectionPool, self)._validate_conn(conn)
File "C:\Users\YM\AppData\Local\Programs\Python\Python38-32\lib\site-packages\
telegram\vendor\ptb_urllib3\urllib3\connectionpool.py", line 289, in _validate_c
onn
conn.connect()
File "C:\Users\YM\AppData\Local\Programs\Python\Python38-32\lib\site-packages\
telegram\vendor\ptb_urllib3\urllib3\connection.py", line 284, in connect
conn = self._new_conn()
File "C:\Users\YM\AppData\Local\Programs\Python\Python38-32\lib\site-packages\
telegram\vendor\ptb_urllib3\urllib3\connection.py", line 144, in _new_conn
raise ConnectTimeoutError(
telegram.vendor.ptb_urllib3.urllib3.exceptions.ConnectTimeoutError: (<telegram.v
endor.ptb_urllib3.urllib3.connection.VerifiedHTTPSConnection object at 0x024257F
0>, 'Connection to api.telegram.org timed out. (connect timeout=5.0)')
Looks like you're having problems with your internet connection, i.e. the request could not be finished within the timeout of 5 seconds. Keep in mind that a lot of problems can happen unexpectedly in networking. In fact python-telegram-bot has a wiki page dedicated to that topic. Ofc you could first try to simply increase the timeout, e.g. by passing timeout=<some_value_>5> to get_me.

Boto's get_all_images function

I tried to load list of AWS images using Python boto library. So my code is like
con = boto.connect_ec2(user, pasw)
con.get_all_instances()
con.get_all_images()
Problem is that, get_all_instances() works fine, but get_all_images() is not responding. When stop application from kbd, I see
File "eagle/main.py", line 102, in <module>
main(sys.argv[1:])
File "eagle/main.py", line 97, in main
return request, process_request(request)
File "eagle/main.py", line 45, in process_request
request = each().process_request(request)
File "/home/rustem/projects/eagle.2.0/eagle/../eagle/middlewares/standard.py", line 64, in process_request
request.images = con.get_all_images()
File "/home/rustem/envs/eagle.2.0/lib/python2.7/site-packages/boto/ec2/connection.py", line 171, in get_all_images
[('item', Image)], verb='POST')
File "/home/rustem/envs/eagle.2.0/lib/python2.7/site-packages/boto/connection.py", line 1063, in get_list
body = response.read()
File "/home/rustem/envs/eagle.2.0/lib/python2.7/site-packages/boto/connection.py", line 411, in read
self._cached_response = httplib.HTTPResponse.read(self)
File "/usr/lib64/python2.7/httplib.py", line 541, in read
return self._read_chunked(amt)
File "/usr/lib64/python2.7/httplib.py", line 590, in _read_chunked
value.append(self._safe_read(chunk_left))
File "/usr/lib64/python2.7/httplib.py", line 647, in _safe_read
chunk = self.fp.read(min(amt, MAXAMOUNT))
File "/usr/lib64/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/usr/lib64/python2.7/ssl.py", line 241, in recv
return self.read(buflen)
File "/usr/lib64/python2.7/ssl.py", line 160, in read
return self._sslobj.read(len)
KeyboardInterrupt
It seems like Boto's function is not working, AWS API is not responding. Have anyone similar problem? Thanks.
Reason was that the get_all_images function returns all images from AWS, so it works about few minutes. I started to user filters and now it takes reasonably time.

Receiving "execution expired" errors on Twitter from Heroku

Starting a few days ago, I have been receiving "execution expired" errors from Twitter on Heroku. I don't receive this error in development, but when I deploy to Heroku I get it all the time.
It seems like other people are having this issue as well. See this bug report on the Twitter gem: https://github.com/sferik/twitter/issues/401#issuecomment-21614751
(although I don't think it's actually a bug with the Twitter gem as it seems Heroku-specific...)
Any idea how to solve this issue and why it is happening?
Stacktrace (most recent call first):
Twitter::Error::ClientError: execution expired
net/http.rb in initialize at line 878
File "net/http.rb", line 878, in initialize
TCPSocket.open(conn_address, conn_port, #local_host, #local_port)
File "net/http.rb", line 878, in open
TCPSocket.open(conn_address, conn_port, #local_host, #local_port)
File "net/http.rb", line 878, in block in connect
TCPSocket.open(conn_address, conn_port, #local_host, #local_port)
File "net/http.rb", line 877, in connect
s = Timeout.timeout(#open_timeout, Net::OpenTimeout) {
File "net/http.rb", line 862, in do_start
connect
File "net/http.rb", line 851, in start
do_start
File "net/http.rb", line 1367, in request
start {
File "new_relic/agent/instrumentation/net.rb", line 33, in request_with_newrelic_trace
request_without_newrelic_trace( request, *args, &block )
File "faraday/adapter/net_http.rb", line 75, in perform_request
http.request create_request(env)
File "faraday/adapter/net_http.rb", line 38, in call
http_response = perform_request(http, env)
File "faraday/response.rb", line 8, in call
#app.call(env).on_complete do |environment|
File "faraday/response.rb", line 8, in call
#app.call(env).on_complete do |environment|
File "faraday/response.rb", line 8, in call
#app.call(env).on_complete do |environment|
File "faraday/request/url_encoded.rb", line 14, in call
#app.call env
File "faraday/request/multipart.rb", line 13, in call
#app.call env
File "twitter/request/multipart_with_file.rb", line 14, in call
#app.call(env)
File "faraday/connection.rb", line 247, in run_request
self.app.call(env)
File "faraday/connection.rb", line 112, in post
run_request(:#{method}, url, body, headers, &block)
File "twitter/client.rb", line 108, in request
connection.send(method.to_sym, path, params, &request_setup).env
File "twitter/client.rb", line 72, in post
request(:post, path, params, signature_params)
File "twitter/api/utils.rb", line 51, in objects_from_response
response = send(request_method.to_sym, path, options)[:body]
File "twitter/api/users.rb", line 260, in block in users
objects_from_response(Twitter::User, method, "/1.1/users/lookup.json", merge_users(arguments.options, users))
File "twitter/core_ext/enumerable.rb", line 5, in block (2 levels) in threaded_map
Thread.new { yield object }

Django haystack with elasticsearch, indexing issue

Im using django-haystack with elasticsearch but there is a problem with indexing. When rebuilding my index python manage.py rebuild_index following error is raised:
Traceback (most recent call last):
File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 210, in handle_label
self.update_backend(label, using)
File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 256, in update_backend
do_update(backend, index, qs, start, end, total, self.verbosity)
File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 78, in do_update
backend.update(index, current_qs)
File "/home/palo/.virtualenvs/toro/local/lib/python2.7/site-packages/haystack/backends/elasticsearch_backend.py", line 177, in update
self.conn.bulk_index(self.index_name, 'modelresult', prepped_docs, id_field=ID)
File "/home/palo/.virtualenvs/toro/src/pyelasticsearch/pyelasticsearch/client.py", line 95, in decorate
return func(*args, query_params=query_params, **kwargs)
File "/home/palo/.virtualenvs/toro/src/pyelasticsearch/pyelasticsearch/client.py", line 366, in bulk_index
query_params=query_params)
File "/home/palo/.virtualenvs/toro/src/pyelasticsearch/pyelasticsearch/client.py", line 221, in send_request
**({'data': request_body} if body else {}))
File "/home/palo/.virtualenvs/toro/src/requests/requests/sessions.py", line 387, in post
return self.request('POST', url, data=data, **kwargs)
File "/home/palo/.virtualenvs/toro/src/requests/requests/sessions.py", line 345, in request
resp = self.send(prep, **send_kwargs)
File "/home/palo/.virtualenvs/toro/src/requests/requests/sessions.py", line 448, in send
r = adapter.send(request, **kwargs)
File "/home/palo/.virtualenvs/toro/src/requests/requests/adapters.py", line 324, in send
raise Timeout(e)
Timeout: HTTPConnectionPool(host='127.0.0.1', port=9200): Request timed out. (timeout=10)
Timeout: HTTPConnectionPool(host='127.0.0.1', port=9200): Request timed out. (timeout=10)
I used django-haystack - 2.0.0-beta, pyelasticsearch - 0.5, elasticsearch 0.20.6, java version "1.6.0_24"
Haystack Settings
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack',
},
}
And Im sure my elasticsearch serivce is running.
This does not necessarily mean that your es server is down, especially if you get something reasonable returned with curl -I "127.0.0.1:9200". More likely, this is an issue of your request simply not getting enough time given the speed of connections involved.
Interestingly, the default timeout set in pyelasticsearch is 60 seconds, see def __init__(self, urls, timeout=60, max_retries=0, revival_delay=300): in https://github.com/rhec/pyelasticsearch/blob/master/pyelasticsearch/client.py. However, haystack overwrites that with its default setting, which is 10 seconds, as per self.timeout = connection_options.get('TIMEOUT', 10) in https://github.com/toastdriven/django-haystack/blob/master/haystack/backends/__init__.py.
As you can see though, haystack allows you to easily modify your setting, by adding 'TIMEOUT': 60, to your engine configuration.
And solved :)
I too had the similar problem
sudo service elasticsearch restart
then it worked
are you running
bin/elasticsearch -f
I think you are not running the searchengine.

gdata.docs.client.DocsClient

I have the following code, reads oauth2 token form file, then try's to perform a doc's list query to find a specific spreadsheet that I want to copy, however no matter what I try the code either errors out or returns with an object containing no document data.
I am using gdata.docs.client.DocsClient which as far as I can tell is version 3 of the API
def CreateClient():
"""Create a Documents List Client."""
client = gdata.docs.client.DocsClient(source=config.APP_NAME)
client.http_client.debug = config.DEBUG
# Authenticate the user with CLientLogin, OAuth, or AuthSub.
if os.path.exists(config.CONFIG_FILE):
f = open(config.CONFIG_FILE)
tok = pickle.load(f)
f.close()
client.auth_token = tok.auth_token
return client
1st query attempt
def get_doc():
new_api_query = gdata.docs.client.DocsQuery(title='RichSheet', title_exact=True, show_collections=True)
d = client.GetResources(q = new_api_query)
this fails with the following stack trace
Traceback (most recent call last):
File "/Users/richard/PycharmProjects/reportone/make_my_report.py", line 83, in <module>
get_doc()
File "/Users/richard/PycharmProjects/reportone/make_my_report.py", line 57, in get_doc
d = client.GetResources(q = new_api_query)
File "/Users/richard/PycharmProjects/reportone/gdata/docs/client.py", line 151, in get_resources
**kwargs)
File "/Users/richard/PycharmProjects/reportone/gdata/client.py", line 640, in get_feed
**kwargs)
File "/Users/richard/PycharmProjects/reportone/gdata/docs/client.py", line 66, in request
return super(DocsClient, self).request(method=method, uri=uri, **kwargs)
File "/Users/richard/PycharmProjects/reportone/gdata/client.py", line 267, in request
uri=uri, auth_token=auth_token, http_request=http_request, **kwargs)
File "/Users/richard/PycharmProjects/reportone/atom/client.py", line 115, in request
self.auth_token.modify_request(http_request)
File "/Users/richard/PycharmProjects/reportone/gdata/gauth.py", line 1047, in modify_request
token_secret=self.token_secret, verifier=self.verifier)
File "/Users/richard/PycharmProjects/reportone/gdata/gauth.py", line 668, in generate_hmac_signature
next, token, verifier=verifier)
File "/Users/richard/PycharmProjects/reportone/gdata/gauth.py", line 629, in build_oauth_base_string
urllib.quote(params[key], safe='~')))
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1266, in quote
if not s.rstrip(safe):
AttributeError: 'bool' object has no attribute 'rstrip'
Process finished with exit code 1
then my second attempt
def get_doc():
other = gdata.docs.service.DocumentQuery(text_query='RichSheet')
d = client.GetResources(q = other)
this returns an ResourceFeed object, but has no content. I have been through the source code for these function but thing are not any obvious.
Have i missed something ? or should i go back to version 2 of the api ?

Resources