I have succesfully implemented this mechanism in my application:
https://vividcode.io/Spring-5-WebFlux-with-Server-Sent-Events/
I can receive events with curl every second, as shown in the example.
My problem is: I cannot receive these events in Angular 5. I have tried many things. Currently my service code looks like this:
public getMigrationProgress(processName: string): Observable<any> {
let headers: HttpHeaders = new HttpHeaders();
headers = headers.append('X-Authorization', this._sessionService.getAuthToken());
headers = headers.append('accept', 'text/event-stream');
let url = config.restApi.url + this.getResource() + '/' + processName;
return Observable.create(observer => {
let eventSource = new EventSourcePolyfill(url, { headers: headers });
eventSource.onmessage = (event => {
observer.next(event);
this.zone.run(() => {
console.log('prpprpr');
});
});
eventSource.onopen = (event) => {
observer.next(event);
};
eventSource.onerror = (error) => {
if (eventSource.readyState === 0) {
console.log('The stream has been closed by the server.');
eventSource.close();
observer.complete();
} else {
observer.error('EventSource error: ' + error);
}
};
});
}
It only opens connection, does not receive events (Method onopen works once, onmessage - never). Server sends them though.
Any ideas how to fix this?
Turned out that if you set event name on server, you cannot receive it by onmessage method.
In the example the event name was set to "random". In order to receive it you have to do it like this:
eventSource.addEventListener('random', function (event) {
console.log(event);
});
Related
My ampq system seems loosing messages, so I'd like a way to see if messages are effectively queued before being consumed.
I have several MicroServices communicating by amqp messages on NodeJs, using CloudAmqp. One of this microservice MS[B] generates .pdf, the process it's pretty heavy and requires about a minute for each request. So I send the .pdf asyncronously, triggering a webhook once finished, and generate once per time using a PreFetch = 1
So one MS[A] collects all the requests from the user, answers back to them saying "ok, request received, listen on the webhook" and in parallel it asks to the MS[B] to generate pdfs. MS[B] has prefetch=1, so consumes just one request per time. Once finished, sends the response to the callback queue of MS[A], which triggers the user webhook saying "the pdf, it's ready".
The problem is that MS[B] misses all the messages while busy:
it consumes one request from MS[A]
starts generating the .pdf
while generating, it discards all the other messages that MS[A] sends, as if there would be not any queue
it finishes the .pdf, sending ACK to MS[A]
then it starts again accepting messages, taking the last one received after being idle, losing all the previous ones.
Why? How can I find the problem, what could I monitor?
Communications between other MSs works well, with messages correctly ordered in queues. Just this one, with prefetch=1, loses messages.
I am NOT using the NO-ACK rule. I don't know what try, what test and what monitor to find the problem.
How can I see (if) messages are correctly queued before being consumed, ora just lost?
Below, the implementation of the messaging system
Channel Creation
/*
Starting Point of a connection to the CLOUDAMQP_URL server, then exec the callback
*/
start(callback) {
var self = this;
// if the connection is closed or fails to be established at all, we will reconnect
amqp.connect(process.env.CLOUDAMQP_URL + "?heartbeat=60")
.then(
function (conn) {
// create queues and consume mechanism
self.amqpConn = conn;
setTimeout(() => {
startPublisher();
}, 200);
setTimeout(() => {
createCallbackQueue();
}, 1000);
setTimeout(() => {
callback();
}, 2000);
});
// create publisher channels
function startPublisher() {
self.amqpConn.createConfirmChannel()
.then(function (ch) {
self.pubChannel = ch;
logger.debug("🗣️ pubChannel ready");
while (true) {
var m = self.offlinePubQueue.shift();
if (!m) break;
self.publish(m[0], // exchange
m[1], // routingKey
m[2], // content,
undefined // correlationId
);
}
});
}
// create callback channel
function createCallbackQueue() {
self.amqpConn.createChannel()
.then(function (channel) {
channel.assertQueue(self.CALLBACK_QUEUE_NAME, {
durable: true,
exclusive: true, // callback are exclusive
})
.then(function (q) {
logger.debug(" 👂 Waiting for RPC RESP in " + self.CALLBACK_QUEUE_NAME);
channel.consume(q.queue,
processCallback, {
noAck: false
}
);
});
// process messages of the callback
function processCallback(msg) {
var correlationId = msg.properties.correlationId;
}
//callback received
if (self.correlationIds_map[correlationId]) {
delete self.correlationIds_map[correlationId];
var content = JSON.parse(msg.content.toString());
self.eventEmitter.emit(correlationId, content);
}
}
});
}
return deferred.promise;
}
Consuming Messages
/*
#worker_queue - the name of the queue
*/
// Consume message from 'worker_queue', A worker that acks messages only if processed succesfully
startWorker(worker_queue, routes) {
var self = this;
logger.debug("startWorker " + self.CALLBACK_QUEUE_NAME);
var channel;
worker_queue = self.MICROSERVICE_NAME + worker_queue;
self.amqpConn.createChannel()
.then(
function (ch) {
channel = ch;
ch.prefetch(self.opt.prefetch); // = 1 for MS[B] generating pdf
channel.assertQueue(worker_queue, {
durable: true,
exclusive: true
})
.then(function (q) {
channel.consume(worker_queue, processMsg, {
noAck: false
});
});
});
// call the 'function from interface' passing params, and send the ACK
function processMsg(msg) {
work(msg)
.then(function (data) {
channel.ack(msg, false); // allUpTo = false
})
.catch(function (err) {
channel.ack(msg, false);
// channel.reject(msg, false); // requeue = false
// this.closeOnErr(e);
});
}
// execute the command, and queue back a response, checking if it's an error or not
function work(msg) {
var deferred = Q.defer();
var correlationId;
try {
correlationId = msg.properties.correlationId;
} catch (err) {}
work_function(msg.content, correlationId)
.then(function (resp) {
var content = {
data: resp
};
content = Buffer.from(JSON.stringify(content));
channel.sendToQueue(msg.properties.replyTo,
content, {
correlationId: correlationId,
content_type: 'application/json'
}
);
deferred.resolve(resp);
});
return deferred.promise;
}
}
Publish Messages
publish(exchange, routingKey, content, correlationId) {
var self = this;
var deferred = Q.defer();
self.correlationIds_map[correlationId] = true;
self.pubChannel.publish(exchange, routingKey, content,
{
replyTo: self.CALLBACK_QUEUE_NAME,
content_type : 'application/json',
correlationId: correlationId,
persistent : true
},
function(err, ok) {
if (err)
{
self.offlinePubQueue.push([exchange, routingKey, content]); // try again
self.pubChannel.connection.close();
deferred.resolve('requeued');
}
else
{
deferred.resolve(ok);
}
});
return deferred.promise;
}
this is a difficult question to ask because I am mystified, but let's see…
I am comparing Got with https.get, and have the following, bare simple code that works. Both Got and https.get return exactly the same result.
But when I use exactly the same code in my Fastify application, Got works as expected but https.get results in a 308.
Is there some way I can debug this code to see what is being sent out by https.get that is causing the remote server to respond with a 308 instead of 200?
import got from 'got';
import https from 'https';
const withGot = async (uri) => {
try {
const json = JSON.parse((await got(uri)).body);
console.log(json);
}
catch (error) {
console.error(error);
}
}
const withHttps = async (uri) => {
try {
const json = await getRequest(uri);
console.log(json);
}
catch (error) {
console.error(error);
}
}
const getRequest = async (uri) => {
return new Promise((resolve) => {
https.get(uri, (res) => {
const { statusCode } = res;
const contentType = res.headers['content-type'];
let error;
/**
* Any 2xx status code signals a successful response but
* here we're only checking for 200.
**/
if (statusCode !== 200) {
error = new Error(`ERROR\n${'-'.repeat(50)}\nRequest Failed.\nURI: ${uri}\nStatus Code: ${statusCode}`);
}
else if (!/^application\/json/.test(contentType)) {
error = new Error(`Invalid content-type.\nExpected application/json but received ${contentType}`);
}
if (error) {
console.error(error.message);
/**
* Consume response data to free up memory
**/
res.resume();
return;
}
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
resolve(parsedData);
}
catch (e) {
console.error(e.message);
}
});
}).on('error', (e) => {
console.error(`Got error: ${e.message}`);
});
});
}
const uri = 'https://zenodo.org/api/records/?q=phylogeny';
withGot(uri);
withHttps(uri);
I figured out the reason for the problem (and the solution)… seems like when I use https.get, I still have to pass the options with a port 443 (the default port for https), otherwise, https seems to knock on port 80 and then gets redirected to port 443 which results in the server sending back html which causes the JSON parser to croak. If I pass an options object like below, then it works. But, it is still weird that the standalone script works fine without the options, so I continue to be mystified even though I have found a solution.
const options = {
hostname: 'zenodo.org',
port: 443,
path: `/api/records/?${qs}`,
method: 'GET',
headers: {
'Content-Type': 'application/json'
}
};
I am creating an Electron app with Socket.io. When the user's computer goes into sleep mode the server disconnects from the client throwing an error "transport close". When the user tries to reconnect I check if the tokens are still valid, if they are not, I refresh them and try to send them to the socketIo server.
The problem I have is that on "reconnect_attempt" socket.io doesn't wait until I refresh the tokens to try reconnecting, it tries reconnecting right away with the old tokens, which get rejected by the server, which also seems to terminate the connection with the user impeding future reconnect attempts.
This is part of my code to connect to the server
module.exports.connect = async (JWT) => {
return new Promise( async resolve => {
console.log("connecting to the server")
const connectionOptions = {
secure: true,
query: {token: JWT},
reconnectionDelay: 4000
}
let socket = await socketIo.connect(`${process.env.SERVER_URL}:${process.env.SERVER_PORT}`, connectionOptions);
resolve(socket)
})
}
This is my code for reconnect_attempt
socket.on('reconnect_attempt', async () => {
const getCurrentJWT = require("../../main").getCurrentJWT;
let JWT = await getCurrentJWT(); //By the time this line returns, socket.io has already tried to reconnect
if(JWT.success) { //if refreshed successfully
console.log("Trying to submit new token......", JWT);
socket.query.token = JWT.JWT;
} else {
console.log("Token not refreshed.")
}
});
And this is part of what I have on the server
io.use(async (socket, next) => {
let token = socket.handshake.query.token;
//and the instruction from here https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html
let tokenIsValid = await checkTokenValidity(token);
if( tokenIsValid ) {
next();
} else {
next(new Error('invalidToken'));
console.log("Not valid token")
}
})
In short, you can use auth for this.
While connecting
auth: {
token: token
}
In the time of reconnection
socket.auth.token = "NEW_TOKEN";
socket.connect();
I can share socket io implementation for this and you can modify it as your need.
For the client-side,
let unauthorized = false;
let socket = io.connect('ws://localhost:8080', {
transports: ["websocket"],
auth: {
token: GET_YOUR_TOKEN()
}
});
socket.on("connect", () => {
unauthorized = false;
});
socket.on('UNAUTHORIZED', () => {
unauthorized = true;
});
socket.on("disconnect", (reason) => {
if (reason === "io server disconnect") {
if(unauthorized) {
socket.auth.token = token;
}
socket.connect();
}
});
socket.on('PING', ()=>{
socket.emit('PONG', token);
});
For the server-side
io.on("connection", (socket) => {
socket.on('PONG', function (token) {
if (isValidToken(token) == false) {
socket.emit("UNAUTHORIZED");
socket.disconnect();
}
});
setInterval(() => {
socket.emit('PING');
}, <YOUR-TIME>);
});
Having the following in your server.
io.use( async function(socket, next) {
let address = socket.handshake.address;
run++; // 0 -> 1
// Validate Token
const token = socket.handshake.auth.token;
if(token !== undefined){
try{
await tokenVerify(token).then((payload) => {
const serverTimestamp = Math.floor(Date.now() / 1000);
const clientTimestamp = payload.exp;
if(clientTimestamp > serverTimestamp){
console.log("Connection from: " + address + " was accepted");
console.log("Token [" + token + "] from: " + address + " was accepted");
next();
}else{
console.log("Connection from: " + address + " was rejected");
console.log("Token [" + token + "] from: " + address + " was rejected");
next(new Error("unauthorized"));
}
});
}catch (e) {
console.log(e);
}
}
})
With the code above, the server will respond "unauthorized" if the token isn't valid.
So, on the client-side, we can catch that message as shown below.
socket_io.on("connect_error", (err) => {
if(err?.message === 'unauthorized'){
var timeout = (socket_reconnection_attempts === 0 ? 5000 : 60000)
console.log("Trying to reconnect in the next " + (timeout / 1000) + ' seconds')
setTimeout(function (){
console.log('Trying to reconnect manually')
socket_reconnection_attempts++;
loadAuthToken().then(function (token) {
socket_io.auth.token = token;
socket_io.connect();
})
}, timeout)
}
});
With the code above, the client-side will try to reconnect and refresh the token only if the error message is "unauthorized."
The variable "socket_reconnection_attempts" is to avoid sending a massive number of reconnection attempts in a short period of time.
I am trying to achieve the following functionality
Before Block : Call the Cy.visit("/login") and call a Function which will trigger a REST API and process the REST API response and set the local storage.
Only after the local storage is set click on "My Account" Link
Here is the source Code I am trying.
import * as subscriberHelpers from '../../../helpers/subscriberHelpers';
import * as localStorage from '../../../helpers/localStorage';
describe('testCode', () => {
before((done) => {
cy.visit('/login', {
timeout: 10000,
onLoad: () => {
localStorage.write("CD-Environment", Cypress.env('defaultEnvironment'));
localStorage.write("CD-Language", "en-US");
localStorage.write("CD-SystemId", "85788485-e411-48a9-b478-610c1285dc1a");
}
})
subscriberHelpers.createSubscriber().then(()=>{
done();
})
})
it('sClick on my account link', () => {
cy.get('.c-header-listItem > .c-link').contains("My Account").click();
})
})
Here is the code to createSubscriber function
export function createSubscriber() {
let URL = `SOME URL`;
let body = {
Some Body
}
return new Promise((resolve, reject) => {
request.subscriberServiceRequest(URL, body).then((response) => {
if (response.status === 200 && ("SessionId" in response.body)) {
localStorage.write("CD-SessionId", response.body.SessionId);
localStorage.write("CD-SubscriberId", response.body.Subscriber.Id);
resolve();
}
else if (response.status === 200 && ("Fault" in response.body)) {
reject(response.body.Fault.Message);
}
})
})
}
Here is the code to subscriber Service request function
export function subscriberServiceRequest(url, body, headers = null) {
let defaultHeaders = { "CD-SystemId": "85788485-e411-48a9-b478-610c1285dc1a" }
if (headers != null) {
defaultHeaders = addHeaders(defaultHeaders, headers);
}
return new Cypress.Promise((resolve, reject) => {
cy.request({
url: url,
method: 'POST',
body: body,
headers: defaultHeaders
}).then((response) => {
resolve(response);
});
})
}
When I try Executing the code I am getting following error in cypress
But the element existing in the UI
Questions:
Why I am getting the error
How to call more than one async functions
in before block
How to tell cypress to wait till the functions on
before block get processed meaning not only wait till receiving the
response but wait till the response got processed in the THEN block
To answer your first question:
Why I am getting the error
.contains() specifically searches for elements within but not including the tag it is called on. In other words, someElement.contains("My Account") will not match someElement.
What you should have instead is this:
cy.get('.c-header-listItem').contains("My Account").click();
Or simply:
cy.contains("My Account").click();
In the socket.io acknowledgement example we see a client's send/emit being called back with the server's response. Is the same functionality available in the reverse direction - i.e. how does the server confirm client reception for a send/emit from the server? It would be nice to have a send/emit callback even just to indicate reception success. Didn't see this functionality documented anywhere...
Thanks!
Looking in the socket.io source I found that indeed ACKs are supported for server-sent messages (but not in broadcasts!) (lines 115-123 of socket.io/lib/socket.js):
if ('function' == typeof args[args.length - 1]) {
if (this._rooms || (this.flags && this.flags.broadcast)) {
throw new Error('Callbacks are not supported when broadcasting');
}
debug('emitting packet with ack id %d', this.nsp.ids);
this.acks[this.nsp.ids] = args.pop();
packet.id = this.nsp.ids++;
}
An example of how the ack should work (not tested):
// server-side:
io.on('msg', (data, ackCallback) => {
console.log('data from client', data);
ackCallback('roger roger');
});
// client-side:
socket.emit('msg', someData, (answer) => {
console.log('server\'s acknowledgement:', answer);
});
If we want to be 100% sure the reception success, just add ack call is not enough because we also need to know whether the ack call is run or not.
The socket.io 3.0 document add this timeout example to show how to do that. But the timeout value is the tricky one.
const withTimeout = (onSuccess, onTimeout, timeout) => {
let called = false;
const timer = setTimeout(() => {
if (called) return;
called = true;
onTimeout();
}, timeout);
return (...args) => {
if (called) return;
called = true;
clearTimeout(timer);
onSuccess.apply(this, args);
}
}
socket.emit("hello", 1, 2, withTimeout(() => {
console.log("success!");
}, () => {
console.log("timeout!");
}, 1000));
Yes we can send a response back to the server from the Client(as acknowledgment )
According to new documentation of socket(4.x)
Server-side
let dataToSend={
test:"fromServer"
}
socket.timeout(5000).emit("my-event",dataToSend, (err, response) => {
if (err) {
// the other side did not acknowledge the event in the given delay
} else {
console.log(response);
}
});
Cleint- side
socket.on("my-event", (data, callback) => {
// any logic for data(that data come from server i.e { test:"fromServer" }
callback({ test: "test" });
});