AWS Lambda cannot connect to Kinesis Firehose "Max retries exceeded with url" - https

I've followed several tutorials and added different IAM configurations and security groups to keep things as open and obvious as possible and nothing has worked. Here is the debug log, and the main error pasted below. Let me know if there's any other data I should add to make this clear.
[DEBUG] 2016-09-22T13:57:40.965Z 5b770867-80cc-11e6-b5b8-018d2de39616 ConnectionError received when sending HTTP request.
Traceback (most recent call last):
File "/var/runtime/botocore/endpoint.py", line 174, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/var/runtime/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/var/runtime/botocore/vendored/requests/adapters.py", line 419, in send
raise ConnectTimeout(e, request=request)
ConnectTimeout: HTTPSConnectionPool(host='firehose.us-east-1.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7fbbc4c92c10>, 'Connection to firehose.us-east-1.amazonaws.com timed out. (connect timeout=60)'))

Please follow my answer in this thread.
This will explain how to make lambda contact any service on the internet (such as S3, kinesis and so).

Currently you can't connect a Lambda task directly to Kinesis Firehose, only to Kineses Streams.
One way around this is to let Firehose write a file to S3 and then trigger a Lambda task via S3 event notifications.

Related

Why are some WCF Named Pipe clients getting a TimeoutException

I have a WCF name pipe service, to communicate 2 processes on windows.
I've tested it on my machine and it worked, I've developed an installer for it, and it's working for most users, but some users are getting a TimeoutException with this message:
"request operation sent to net.pipe://localhost/myService did not receive a reply within the configured timeout (00:00:10). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client."
What can be the reason for this error? Is it a security issue? Where can I set security settings for named pipes?
Thanks
Some users may not receive a reply in time due to network problems, which may cause a timeout.
You can configure the timeout setting in configuration file by setting the send and receive timeout on binding element.
Apply the binding configuration on both service and client-side:
<netNamedPipeBinding>
<binding sendTimeout="00:20:00" receiveTimeout="00:20:00">
<reliableSession inactivityTimeout="00:20:00"/>
</binding>
</netNamedPipeBinding>

Elixir erlang :ftp.send got stuck

I use Erlang ftp lib in my elixir project to send file to ftp server.
I call send function :ftp.send(pid, '#{local_path}', '#{remote_path}') to upload file to ftp server.
Most of the time it uploads files successfully, but it sometimes stuck here, not moving to the next line.
According to the docs it should return :ok or {:error, reason}, but simply stuck at :ftp.send.
Can anyone give me suggestion? I am not familiar with Erlang.
Version: Elixir 1.7.3 (compiled with Erlang/OTP 21)
ftp module has two types of timeout, both set during the initialization of ftp service.
Here is an excerpt from the documentation:
{timeout, Timeout}
Connection time-out. Default is 60000 (milliseconds).
{dtimeout, DTimeout}
Data connect time-out. The time the client waits for the server to connect to the data socket. Default is infinity.
Data connect time-out has a default value of infinity, meaning it’d be hang up if there are some network issues. To overcome the problem, I’d suggest you set this value to somewhat meaningful and handle timeouts in your application appropriately.
{:ok, pid} = :ftp.start_service(
host: '...', timeout: 30_000, dtimeout: 10_000
)
:ftp.send(pid, '#{local_path}', '#{remote_path}')

Some postgress connections timing-out while others don't

I have an AWS EC2 machine running a Laravel 5.2 application that connects to a Postgress 9.6 databse running in RDS. While most of the connections work, some of them are getting rejected when trying to stablish, which causes a Timeout and consequently an error in my API. I don't know what is causing them to be rejected. Also, it is very random when it happens, when it does happen it may be in any API endpoint and inside the endpoint in any query.
When the timeout is handled by PHP, it shows a message like:
SQLSTATE[08006] [7] timeout expired (SQL: ...)
Sometimes the Nginx handles the timeout and replies with a 504 Error. When Nginx handles the timeout I get an error like:
2019/04/24 09:48:18 [error] 20657#20657: *3236 upstream timed out (110: Connection timed out) while reading response header from upstream, client: {client-ip-here}, server: {my-url-here}, request: "GET {my-endpoint-here} HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock", host: "{}", referrer: "https://app.cartoriovirtual.com/"
All usage charts on the RDS and EC2 seems ok, I have plenty of RAM, storage, CPU and available connections for RDS. I also checked inner VPC Flows and they seem alright, however I have many IPs (listed as attackers) scanning my network interfaces, most of them been rejected. Some (to port 22) accepted but stoped at authentication, I use a .pem Key File for auth.
The RDS network interface only accepts requests from inner VPC machines. In its logs, every 5 minutes I have a Checkpoint like this:
2019-04-25 01:05:29 UTC::#:[22595]:LOG: checkpoint starting: time
2019-04-25 01:05:34 UTC::#:[22595]:LOG: checkpoint complete: wrote 43 buffers (0.1%); 0 transaction log file(s) added, 0 removed, 1 recycled; write=4.393 s, sync=0.001 s, total=4.404 s; sync files=19, longest=0.001 s, average=0.000 s; distance=16515 kB, estimate=16515 kB
Anyone has tips on how to find a solution? I looked at all possible logs that came in mind, fixed a few little issues but the error persists. I am running out of ideas.

Read error looking for ack: read tcp 10.11.12.13:1223 i/o timeout in logstash-forwarder

I've setup the logstash. All of sudden the logstash-forwarder stops to send the logs to Logstash. While checking the logs, its show the below error, can anyone help me to fix this ?
Read error looking for ack: read tcp 10.11.12.13:1223 i/o timeout
Thanks.
The error shows there is no connection between logstash forwarder (or client) to logstash indexer (or server). Common reasons:
Logstash service is down on logstash indexer. If down, start the service up.
connection between logstash forwarder and indexer is blocked by firewall.
How to check?
login forwarder
telnet 10.11.12.13 1223
If you can't, that's the firewall issue. You need fix or rollback some recent changes.

WebSocket connection fails when connected through 3G on flask-sockets and gunicorn

I've written a small app which uses WebSockets using flask-sockets. I'm currently serving the site straight from gunicorn (using the gevent worker from flask-sockets), with no nginx or anything in front, because it's just a small experiment. when trying to access the site on my smartphone while connected to an access point, the site works fine. On the other hand, when connected with 3G, the site is loaded but the websocket connection fails.
I get this in the server log:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line 424, in run_application
self.result = self.application(self.environ, self.start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/flask_sockets.py", line 37, in __call__
environment = environ['wsgi.websocket']
KeyError: 'wsgi.websocket'
<PyWSGIServer fileno=5 address=192.168.1.50:80>: Failed to handle request:
request = GET /clock_skew HTTP/1.1 from ('80.246.130.25', 36438)
application = <Flask 'flask_client_id.client_id'>
I thought maybe it has something to do with proxy servers in the middle messing with the web socket headers or something, so I tried to change the JS code to 'wss://' on the client side, but that doesn't work at all (no client can establish a connection).
I'll appericate anyone who can provide me with any ideas on this.
This is quite of a frequent problem with 3G operators (at least in europe).
I suggest you to try to run your webserver on a different port than 80, better if thats a port for some known service (eg. 443)

Resources