I'm getting this error message when trying to connect to a aws neptune db from a lambda:
2022-05-05T18:36:04.114Z e0c9ee4c-0e1d-49c7-ad05-d8bab79d3ea6 WARN Determining whether retriable error: Server error: {
"requestId": "some value",
"code": "TimeLimitExceededException",
"detailedMessage": "A timeout occurred within the script or was otherwise cancelled directly during evaluation of [some value]"
} (598)
The timeout happens consistently after 20s.
It's not clear what's causing this. Things I've tried:
increasing the lambda memory in case it's just a hardware problem, but no luck
increasing neptune query timeout from 20s to 60s, but the request still times out at 20s.
This is the code of the lambda that tries to initialize the connection:
import { driver, structure } from 'gremlin';
import { getUrlAndHeaders } from 'gremlin-aws-sigv4/lib/utils';
const getConnectionDetails = () => {
if (process.env['USE_IAM'] == 'true') {
return getUrlAndHeaders(
process.env['CLUSTER_ENDPOINT'],
process.env['CLUSTER_PORT'],
{},
'/gremlin',
'wss'
);
} else {
const database_url =
'wss://' +
process.env['CLUSTER_ENDPOINT'] +
':' +
process.env['CLUSTER_PORT'] +
'/gremlin';
return { url: database_url, headers: {} };
}
};
const getConnection = () => {
const { url, headers } = getConnectionDetails();
const c = new driver.DriverRemoteConnection(url, {
mimeType: 'application/vnd.gremlin-v2.0+json',
headers: headers,
});
c._client._connection.on('close', (code, message) => {
console.info(`close - ${code} ${message}`);
if (code == 1006) {
console.error('Connection closed prematurely');
throw new Error('Connection closed prematurely');
}
});
return c;
};
This was working previously using more powerful hardware (r4.2xlarge) for the neptune db, but I changed that t3.medium to minimize cost and it seems that's when the problem started. But I find it hard to believe that this hardware change alone would cause the connection to timeout, and it's odd that it continues to timeout at exactly 20s. Any ideas?
Once parameter group values are changed, the instance you are connecting to still needs to be restarted for them to take effect. You can do this:
From the AWS Console (web page) for Neptune
From the CLI using aws neptune reboot-db-instance
Related
The transfer() function works perfectly fine when testing and through the CLI. However, when I try to call it from the front end, it returns
Uncaught (in promise) Error: {"index":0,"kind":{"ExecutionError":"Exceeded the prepaid gas."}}
It is not a complex call and only involves just 1. transferring tokens 2. updating a value in storage. Can anyone give me pointers as to why this might be happening?
document.querySelector('#transfer-to-owner').onclick = () => {
console.log("Transfer about to begin")
try {
window.contract.transfer({})
} catch (e) {
'Something went wrong! ' +
'Check your browser console for more info.'
}
}
contract from this repo
const XCC_GAS: Gas = 20_000_000_000_000;
transfer(): void {
this.assert_owner()
assert(this.contributions.received > u128.Zero, "No received (pending) funds to be transferred")
const to_self = Context.contractName
const to_owner = ContractPromiseBatch.create(this.owner)
// transfer earnings to owner then confirm transfer complete
const promise = to_owner.transfer(this.contributions.received)
promise.then(to_self).function_call("on_transfer_complete", '{}', u128.Zero, XCC_GAS)
}
#mutateState()
on_transfer_complete(): void {
assert_self()
assert_single_promise_success()
logging.log("transfer complete")
// reset contribution tracker
this.contributions.record_transfer()
}
near-api-js and near-shell use a different default value for gas.
near-api-js:
const DEFAULT_FUNC_CALL_GAS = new BN('30_000_000_000_000');
near-shell:
.option('gas', {
desc: 'Max amount of gas this call can use (in gas units)',
type: 'string',
default: '100_000_000_000_000'
})
I added _s to make it clearer that near-shell uses more than 3 times the amount of gas by default.
Trying to publish from a Lambda function to a Redis Elasticache, but I just continue to get 502 Bad Gateway responses with the Lambda function timing out.
I have successfully connected to the Elasticache instance using an ECS in the same VPC which leads me to think that the VPC settings for my Lambda are not correct. I tried following this tutorial (https://docs.aws.amazon.com/lambda/latest/dg/services-elasticache-tutorial.html) and have looked at several StackOverflow threads to no avail.
The Lambda Function:
export const publisher = redis.createClient({
url: XXXXXX, // env var containing the URL which is also used in the ECS server to successfully connect
});
export const handler = async (
event: AWSLambda.APIGatewayProxyWithCognitoAuthorizerEvent
): Promise<AWSLambda.APIGatewayProxyResult> => {
try {
if (!event.body || !event.pathParameters || !event.pathParameters.channelId)
return ApiResponse.error(400, {}, new InvalidRequestError());
const { action, payload } = JSON.parse(event.body) as {
action: string;
payload?: { [key: string]: string };
};
const { channelId } = event.pathParameters;
const publishAsync = promisify(publisher.publish).bind(publisher);
await publishAsync(
channelId,
JSON.stringify({
action,
payload: payload || {},
})
);
return ApiResponse.success(204);
} catch (e) {
Logger.error(e);
return ApiResponse.error();
}
};
In my troubleshooting, I have verified the following in the Lambda functions console:
The correct role is showing in Configuration > Permissions
The lambda function has access to the VPC (Configuration > VPCs), Subnets, and the same SG as the Elasticache instance.
The SG is allowing all traffic from anywhere.
It is indeed the Redis connection. Using console.log the code stops at this line: await publishAsync()
I am sure it is something small, but it is racking my brain!
Update 1:
Tried adding an error handler to log any issues with the publish in addition to the main try/catch block, but it's not logging a thing.
publisher.on('error', (e) => {
Logger.error(e, 'evses-service', 'message-publisher');
});
Also have copied my Elasticache setup:
And my Elasticache Subnet Group:
And my Lambda VPC settings:
And that my Lambda has the right access:
Update 2:
Tried to follow the tutorial here (https://docs.aws.amazon.com/lambda/latest/dg/services-elasticache-tutorial.html) word for word, but getting the same issue. No logs, just a timeout after 30 seconds. Here is the test code:
const crypto = require('crypto');
const redis = require('redis');
const util = require('util');
const client = redis.createClient({
url: 'rediss://clusterforlambdatest.9nxhfd.0001.use1.cache.amazonaws.com',
});
client.on('error', (e) => {
console.log(e);
});
exports.handler = async (event) => {
try {
const len = 24;
const randomString = crypto
.randomBytes(Math.ceil(len / 2))
.toString('hex') // convert to hexadecimal format
.slice(0, len)
.toUpperCase();
const setAsync = util.promisify(client.set).bind(client);
const getAsync = util.promisify(client.get).bind(client);
await setAsync(randomString, 'We set this string bruh!');
const doc = await getAsync(randomString);
console.log(`Successfully receieved document ${randomString} with contents: ${doc}`);
return;
} catch (e) {
console.log(e);
return {
statusCode: 500,
};
}
};
If you have timeout, assuming the lambda network is well configured, you should check the following:
redis SSL configuration: check diffs between redisS connection url and cluster configuration (in-transit encryption and client configuration with tls: {})
configure the client with a specific retry strategy to avoid lambda timeout and catch connection issue
check VPC acl and security groups
I had same issue with my elasticache cluster, here are few findings -
Check the number client connections with elasticache and resources used
Check VPC subnet and CIDR for nodes security group
Try to increase the TTL for lambda and see which service is taking more time to respond Lambda or elasticache
I try to generate step load for performance test on k6 for websocket.
regular settings like
export let options = {
stages: [
{
"duration": "0m30s",
"target": 10
},
{
"duration": "0m30s",
"target": 10
},
{
"duration": "0m30s",
"target": 0
}
],
};
doesn't work for k6. I tried --vus 10 --i 10
but it just go through scenario 10 times and sleep till the end of 10 minutes.
Than I tried k6 run --vus 5 --stage 3m:10,5m:10,10m:35,1m30s:0 but result is almost the same. How create active load with step pattern for websocket testing? to connect every time after flow is done?
Test flow:
import ws from "k6/ws";
import { check } from "k6";
export default function() {
const url = "ws://URL:8000/";
const params = { tags: { my_tag: "hello" } };
const response = ws.connect(url, params, function(socket) {
socket.on("open", function open() {
console.log("connected");
socket.send(Date.now());
var url = "ws://URL:8000/";
var response = ws.connect(url, null, function(socket) {
socket.on('open', function() {
socket.send('Hello');
socket.send('How are you?');
});
socket.on("close", () => console.log("disconnected"));
socket.on("error", (e) => {
if (e.error() != "websocket: close sent") {
console.log("An unexpected error occured: ", e.error());
}
});
check(response, { "status is 101": r => r && r.status === 101 });
})
})
})
}
As mentioned in the documentation ws.connect blocks until the connection is closed either by calling socket.close() or the server closing the connection on the other end (or due to an error while doing something else).
So if you want to just send the 2 messages there you should call socket.close(). You can also close after receiving something on the other side using by using
socket.on("message", function(msg) {
console.log("got: "+ msg);
socket.close();
}
But if you don't need to reconnect but instead want to just send more messages you should probably use a timer:
socket.setInterval(function() {
socket.send('Hello');
socket.send('How are you?');
}, 1500);
will send messages each 1.5s
I don't know why you open another websocket connection inside the other, but that might be a mistake on your part and is likely going to make the code hard to read if you don't use a different variable name from socket
I wasted lots of time trying it with the K6 but it seems not supported till today, k6 does not support socket.io testing, check the links attached.
https://github.com/grafana/k6/issues/667
You can go for artillery, as suggested in the official socket.io docs
https://socket.io/docs/v4/load-testing/
I'm currently stuck with an issue I'm getting with the coinbase-pro-node npm package (https://github.com/coinbase/coinbase-pro-node). I'm connecting to the matches channel and listening for messages there but the connection with the Websocket breaks after a few hours without telling me much. I can't trace back the problem and it doesn't happen on the same intervals. Sometimes it breaks just minutes after I run the script. Thanks for the help.
The code:
const CoinbasePro = require('coinbase-pro');
var coinbaseWs = '';
function connect() {
coinbaseWs = new CoinbasePro.WebsocketClient(
['BTC-USD'],
'wss://ws-feed.pro.coinbase.com',
{
key: 'xxxx',
secret: 'xxxx',
passphrase: 'xxxx',
},
{ channels: ['matches'] }
);
coinbaseWs.on('message', async data => {
console.log(data)
});
coinbaseWs.on('error', err => {
console.error("Connection with Coinbase websocket failed with error: " + err);
console.log("Error stack trace: " + err.stack);
});
coinbaseWs.on('close', () => {
console.error("Connection with Coinbase websocket closed!");
});
}
connect();
Error stack:
Error: read ECONNRESET
File "internal/stream_base_commons.js", line 167, in TLSWrap.onStreamRead
it does break from time to time for no apparent reason. All you can do is listen for the heartbeat messages and use those to decide whether to re-initiate a new websocket feed. I raised a similar query directly with the coinbase pro/gdax customer support.
just to be clear up-front, this questions is about .Net Core SignalR, not the previous version.
The new SignalR has an issue with WebSockets behind IIS (I can't get them to work on Chrome/Win7/IIS express). So instead I'm using Server Sent Events (SSE).
However, the problem is that those time out after about 2 minutes, the connection state goes from 2 to 3. Automatic reconnect has been removed (apparently it wasn't working really well anyway in previous versions).
I'd like to implement a heartbeat timer now to stop clients from timing out, a tick every 30 seconds may well do the job.
Update 10 November
I have now managed to implement the server side Heartbeat, essentially taken from Ricardo Peres' https://weblogs.asp.net/ricardoperes/signalr-in-asp-net-core
in startup.cs, add to public void Configure(IApplicationBuilder app, IHostingEnvironment env, IServiceProvider serviceProvider)
app.UseSignalR(routes =>
{
routes.MapHub<TheHubClass>("signalr");
});
TimerCallback SignalRHeartBeat = async (x) => {
await serviceProvider.GetService<IHubContext<TheHubClass>>().Clients.All.InvokeAsync("Heartbeat", DateTime.Now); };
var timer = new Timer(SignalRHeartBeat).Change(TimeSpan.FromSeconds(0), TimeSpan.FromSeconds(30));
HubClass
For the HubClass, I have added public async Task HeartBeat(DateTime now) => await Clients.All.InvokeAsync("Heartbeat", now);
Obviously, both the timer, the data being sent (I'm just sending a DateTime) and the client method name can be different.
Update .Net Core 2.1+
See the comment below; the timer callback should no longer be used. I've now implemented an IHostedService (or rather the abstract BackgroundService) to do that:
public class HeartBeat : BackgroundService
{
private readonly IHubContext<SignalRHub> _hubContext;
public HeartBeat(IHubContext<SignalRHub> hubContext)
{
_hubContext = hubContext;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
await _hubContext.Clients.All.SendAsync("Heartbeat", DateTime.Now, stoppingToken);
await Task.Delay(30000, stoppingToken);
}
}
}
In your startup class, wire it in after services.AddSignalR();:
services.AddHostedService<HeartBeat>();
Client
var connection = new signalR.HubConnection("/signalr", { transport: signalR.TransportType.ServerSentEvents });
connection.on("Heartbeat", serverTime => { console.log(serverTime); });
Remaining pieces of the initial question
What is left is how to properly reconnect the client, e.g. after IO was suspended (the browser's computer went to sleep, lost connection, changed Wifis or whatever)
I have implemented a client side Heartbeat that is working properly, at least until the connection breaks:
Hub Class: public async Task HeartBeatTock() => await Task.CompletedTask;
Client:
var heartBeatTockTimer;
function sendHeartBeatTock() {
connection.invoke("HeartBeatTock");
}
connection.start().then(args => {
heartBeatTockTimer = setInterval(sendHeartBeatTock, 10000);
});
After the browser suspends IO for example, the invoke method would throw an exception - which cannot be caught by a simple try/catch because it is async.
What I tried to do for my HeartBeatTock was something like (pseudo-code):
function sendHeartBeatTock
try connection.invoke("HeartbeatTock)
catch exception
try connection.stop()
catch exception (and ignore it)
finally
connection = new HubConnection().start()
repeat try connection.invoke("HeartbeatTock")
catch exception
log("restart did not work")
clearInterval(heartBeatTockTimer)
informUserToRefreshBrowser()
Now, this does not work for a few reasons. invoke throws the exception after the code block executes due to being run asynchronous. It looks as though it exposes a .catch() method, but I'm not sure how to implement my thoughts there properly.
The other reason is that starting a new connection would require me to re-implement all server calls like "connection.on("send"...) - which appears silly.
Any hints as to how to properly implement a reconnecting client would be much appreciated.
This is an issue when running SignalR Core behind IIS. IIS will close idle connections after 2 minutes. The long term plan is to add keep alive messages which, as a side effect, will prevent IIS from closing the connection. To work around the problem for now you can:
send periodically a message to the clients
change the idle-timeout setting in IIS as described here
restart the connection on the client side if it gets closed
use a different transport (e.g. long polling since you cannot use webSockets on Win7/Win2008 R2 behind IIS)
I've got a working solution now (tested in Chrome and FF so far). In the hope to either motivate you to come up with something better, or to save you a little while coming up with something like this yourselves, I'm posting my solution here:
The Heartbeat-"Tick" message (the server routinely pinging the clients) is described in the question above.
The client ("Tock" part) now has:
a function to register the connection, so that the callback methods (connection.on()) can be repeated; they'd be lost after just restarting a "new HubConnection" otherwise
a function to register the TockTimer
and a function to actually send Tock pings
The tock method catches errors upon sending, and tries to initiate a new connection. Since the timer keeps running, I'm registering a new connection and then simply sit back and wait for the next invocation.
Putting the client together:
// keeps the connection object
var connection = null;
// stores the ID from SetInterval
var heartBeatTockTimer = 0;
// how often should I "tock" the server
var heartBeatTockTimerSeconds = 10;
// how often should I retry after connection loss?
var maxRetryAttempt = 5;
// the retry should wait less long then the TockTimer, or calls may overlap
var retryWaitSeconds = heartBeatTockTimerSeconds / 2;
// how many retry attempts did we have?
var currentRetryAttempt = 0;
// helper function to wait a few seconds
$.wait = function(miliseconds) {
var defer = $.Deferred();
setTimeout(function() { defer.resolve(); }, miliseconds);
return defer;
};
// first routine start of the connection
registerSignalRConnection();
function registerSignalRConnection() {
++currentRetryAttempt;
if (currentRetryAttempt > maxRetryAttempt) {
console.log("Clearing registerHeartBeatTockTimer");
clearInterval(heartBeatTockTimer);
heartBeatTockTimer = 0;
throw "Retry attempts exceeded.";
}
if (connection !== null) {
console.log("registerSignalRConnection was not null", connection);
connection.stop().catch(err => console.log(err));
}
console.log("Creating new connection");
connection = new signalR.HubConnection("/signalr", { transport: signalR.TransportType.ServerSentEvents });
connection.on("Heartbeat", serverTime => { console.log(serverTime); });
connection.start().then(() => {
console.log("Connection started, starting timer.");
registerHeartBeatTockTimer();
}).catch(exception => {
console.log("Error connecting", exception, connection);
});
}
function registerHeartBeatTockTimer() {
// make sure we're registered only once
if (heartBeatTockTimer !== 0) return;
console.log("Registering registerHeartBeatTockTimer");
if (connection !== null)
heartBeatTockTimer = setInterval(sendHeartBeatTock, heartBeatTockTimerSeconds * 1000);
else
console.log("Connection didn't allow registry");
}
function sendHeartBeatTock() {
console.log("Standard attempt HeartBeatTock");
connection.invoke("HeartBeatTock").then(() => {
console.log("HeartbeatTock worked.") })
.catch(err => {
console.log("HeartbeatTock Standard Error", err);
$.wait(retryWaitSeconds * 1000).then(function() {
console.log("executing attempt #" + currentRetryAttempt.toString());
registerSignalRConnection();
});
console.log("Current retry attempt: ", currentRetryAttempt);
});
}
Client version based on ExternalUse's answer...
import * as signalR from '#aspnet/signalr'
import _ from 'lodash'
var connection = null;
var sendHandlers = [];
var addListener = f => sendHandlers.push(f);
function registerSignalRConnection() {
if (connection !== null) {
connection.stop().catch(err => console.log(err));
}
connection = new signalR.HubConnectionBuilder()
.withUrl('myHub')
.build();
connection.on("Heartbeat", serverTime =>
console.log("Server heartbeat: " + serverTime));
connection.on("Send", data =>
_.each(sendHandlers, value => value(data)));
connection.start()
.catch(exception =>
console.log("Error connecting", exception, connection));
}
registerSignalRConnection();
setInterval(() =>
connection.invoke("HeartBeatTock")
.then(() => console.log("Client heatbeat."))
.catch(err => {
registerSignalRConnection();
}), 10 * 1000);
export { addListener };