I am using kafka in my project using kafka-node package...
I have introduced a method and inside it i am trying to use a kafka module for eg:
Meteor.methods
kafka: (topic, message) ->
if(Meteor.isServer)
message = JSON.stringify(message)
kafka = Meteor.npmRequire 'kafka-node'
HighLevelProducer = kafka.HighLevelProducer
Client = kafka.Client
client = new Client
producer = new HighLevelProducer(client)
payloads =[{topic: topic, messages: [message]}]
producer.on 'ready', ->
producer.send payloads, (error,data) ->
if not error
HighLevelConsumer = kafka.HighLevelConsumer
Client = kafka.Client
client = new Client('localhost:2181')
topics = [ { topic: topic } ]
options =
autoCommit: true
fetchMaxWaitMs: 1000
fetchMaxBytes: 1024 * 1024
consumer = new HighLevelConsumer(client, topics, options)
consumer.on 'message',(message) ->
console.log message.value
#Meteor.call 'saveMessage', message.value, (error,data) ->
return
consumer.on 'error', (err) ->
console.log 'error', err
return
producer.on 'error', (err) ->
console.log 'error', err
Everything was fine until i decided to use meteor.call and call a method to save that message..
It gives me this error.
Meteor code must always run within a Fiber. Try wrapping callbacks
that you pass to non-Meteor libraries with Meteor.bindEnvironment
I tried encapsulating it inside Fiber, used Meteor.wrapAsync(), Neither helped,
Please guys can you help me, i am having difficult time solving this issue...
If you're using node style callbacks, you can use Meteor.bindEnvironment around the callback. For example:
let Sockets = new Mongo.Collection('connections');
function createConnection (name) {
check(name, String);
let socket = net.connect(23, '192.168.1.3', Meteor.bindEnvironment(function () {
Sockets.upsert({ name: name }, { $set: { status: 'connected' } });
}));
}
Related
I'm creating an application that is using Nestjs with websockets, but now I need to add rate limit on the sockets, but analyzing the documentation documentation link and implementing what it says in it, when I use #UseGuards(MyGuard) an error occurs in the application.
My Guard:
#Injectable()
export class NewThrottlerGuard extends ThrottlerGuard {
protected async handleRequest(
context: ExecutionContext,
limit: number,
ttl: number,
): Promise<boolean> {
console.log('Request');
const client = context.switchToWs().getClient();
const ip = client.conn.remoteAddress;
const key = this.generateKey(context, ip);
const ttls = await this.storageService.getRecord(key);
if (ttls.length >= limit) {
throw new ThrottlerException();
}
await this.storageService.addRecord(key, ttl);
return true;
}
}
Websocket:
#UseGuards(NewThrottlerGuard)
#SubscribeMessage('sendMessage')
sendMessage(
#ConnectedSocket() client: Socket,
#MessageBody() message: string,
) {
client.rooms.forEach((room) => {
if (room !== client.id) {
client.broadcast.to(room).emit('message', message);
}
});
}
Error in console:
/node_modules/#nestjs/common/utils/validate-each.util.js:22
throw new InvalidDecoratorItemException(decorator, item, context.name);
^
Error: Invalid guard passed to #UseGuards() decorator (ChatGateway).
at validateEach
The file in: #nestjs/common/utils/validate-each.util.js:22
function validateEach(context, arr, predicate, decorator, item) {
if (!context || !context.name) {
return true;
}
console.log(context, arr)
const errors = arr.some(str => !predicate(str));
if (errors) {
throw new InvalidDecoratorItemException(decorator, item, context.name);
}
return true;
}
i put some console.log then in the terminal it show:
[Function: ChatGateway] [ undefined ]
In Github Throttler documentation they say: You cannot bind the guard with APP_GUARD or app.useGlobalGuards() due to how Nest binds global guards.
So, im using #UseGuards()
The guard itself was written correctly, but it was put in a location that importing it made a circular reference between files, so when #UseGuards() was used it became #UseGuards(undefined) which caused the cryptic error message. Moving the guard to a dedicated file will fix the error
I follow your github reference settings and it doesn't work，The following is my code, where is my setting wrong, and the request to ws is not intercepted(In the handleRequest method)
I am trying to create a Lambda in AWS that serves as a producer to an MSK topic. All the AWS docs say to create a new EC2 instance, but as my Lambda is in the same VPC I feel like this should work. I am very new to this and I notice my log statement never hits in my producer.on function.
I am using nodejs and the kafka-node module. The code can be found below.
Essentially, I am just wondering if anyone knows how to do this and why the producer.on function is never hit when I run test through the Lambda? This is just some test code to see if I can get it to send, but if any more data is needed oy help please let me know and thanks in advance.
exports.handler = async (event, context,callback) => {
const kafka = require('kafka-node');
const bp = require('body-parser');
const kafka_topic = 'MyTopic';
const Producer = kafka.Producer;
var KeyedMessage = kafka.KeyedMessage;
const Client = kafka.Client;
const client = new kafka.KafkaClient({kafkaHost: 'myhost:9094'});
console.log('client :: '+JSON.stringify(client));
const producer = new Producer(client);
console.log('about to hit producer code');
producer.on('ready', function() {
console.log('Hello there!')
let message = 'my message';
let keyedMessage = new KeyedMessage('keyed', 'me keyed message');
producer.send([
{ topic: kafka_topic, partition: 0, messages: [message, keyedMessage], attributes: 0 }
], function (err, result) {
console.log(err || result);
process.exit();
});
});
producer.on('error', function (err) {
console.log('error', err);
});
}
return "success";
What you need is to be able to produce messages on your MSK cluster using REST API. Why not setup a REST proxy for MSK as detailed here and then call this API to produce your messages to MSK.
I am using the Node.js ws library, to listen to events in user accounts on a 3rd party API. For each user, I open a websocket to listen to the events in the user's account.
Turns out, the 3rd-party API doesn't provide a userID for each event, so if I have 10 websocket connections to user-accounts, I cannot determine which account an event came from.
I have access to a unique userId prior to starting each of my connections.
Is there a way to append or wrap the websocket connection with the userId identifier, to each connection I make, such that when I receive an event, I can access the custom identifier, and subsequently know which user's account the event came from?
The code below is a mix of real code, and pseudocode (i.e customSocket)
const ws = new WebSocket('wss://thirdparty-api.com/accounts', {
port: 8080,
});
ws.send(
JSON.stringify({
action: 'authenticate',
data: {
oauth_token: access_token,
},
})
);
// wrap and attach data here (pseudocode at top-level)
customSocket.add({userId,
ws.send(
JSON.stringify({
action: 'listen',
data: {
streams: ['action_updates'],
},
})
)
})
// listen for wrapper data here, pseudocode at top level
customSocket.emit((customData) {
ws.on('message', function incoming(data) {
console.log('incoming -> data', data.toString());
})
console.log('emit -> customData', customData);
})
Looking at the socket.io library, the namespace feature may solve for this, but I can't determine if that's true or not. Below is an example in their documentation:
// your application has multiple tenants so you want to dynamically create one namespace per tenant
const workspaces = io.of(/^\/\w+$/);
workspaces.on('connection', socket => {
const workspace = socket.nsp;
workspace.emit('hello');
});
// this middleware will be assigned to each namespace
workspaces.use((socket, next) => {
// ensure the user has access to the workspace
next();
});
I found a solution to this which is fairly simple. First create a message handler function:
const eventHandler = (uid, msg) => {
console.log(`${uid} did ${msg}`);
};
Then, when you create the websocket for the given user, wrap the .on event with the handler:
const createSocketForUser = (uid, eventHandler) => {
const socket = new WebSocket(/* ... */);
socket.onmessage = (msg) => {
eventHandler(uid, msg)
};
return socket;
}
I am trying to integrate socket.io with strapi. But unfortunately I have been unable to do so without any proper tutorial or documentation covering this aspect.
I followed along with the only resource I found online which is:
https://medium.com/strapi/strapi-socket-io-a9c856e915a6
But I think the article is outdated. I can't seem to run the code mentioned in it without running into tonnes of errors.
Below is my attempt to implement it and I have been trying to connect it through a chrome websocket plugin smart websocket client But I am not getting any response when I try to run the server.
I'm totally in the dark. Any help will be appreciated
module.exports = ()=> {
// import socket io
var io = require('socket.io')(strapi.server)
console.log(strapi.server) //undefined
// listen for user connection
io.on('connect', socket => {
socket.send('Hello!');
console.log("idit")
// or with emit() and custom event names
socket.emit('greetings', 'Hey!', { 'ms': 'jane' }, Buffer.from([4, 3, 3, 1]));
// handle the event sent with socket.send()
socket.on('message', (data) => {
console.log(data);
});
// handle the event sent with socket.emit()
socket.on('salutations', (elem1, elem2, elem3) => {
console.log(elem1, elem2, elem3);
});
});
};
So I found the solution. Yay. I'll put it here just in case anybody needs it.
boostrap.js
module.exports = async () => {
process.nextTick(() =>{
var io = require('socket.io')(strapi.server);
io.on('connection', async function(socket) {
console.log(`a user connected`)
// send message on user connection
socket.emit('hello', JSON.stringify({message: await strapi.services.profile.update({"posted_by"})}));
// listen for user diconnect
socket.on('disconnect', () =>{
console.log('a user disconnected')
});
});
strapi.io = io; // register socket io inside strapi main object to use it globally anywhere
})
};
Found this at: https://github.com/strapi/strapi/issues/5869#issuecomment-619508153_
Apparently, socket.server is not available when the server starts. So you have to make use of process.nextTick that waits for the socket.server to initialize.
I'll also add a few questions that I faced when setting this up.
How do i connect from an external client like nuxt,vue or react?
You just have to connect through "http://localhost:1337" that is my usual address for strapi.
I am using nuxt as my client side and this is how set up my socketio on the client side
I first installed nuxt-socket-io through npm
Edited the nuxt.config file as per it's documention
modules:[
...
'nuxt-socket-io',
...
],
io: {
// module options
sockets: [
{
name: 'main',
url: 'http://localhost:1337',
},
],
},
And then i finally added a listener in one of my pages.
created() {
this.socket = this.$nuxtSocket({})
this.socket.on('hello', (msg, cb) => {
console.log('SOCKET HI')
console.log(msg)
})
},
And it works.
A clean way to integrate third-party services into Strapi is to use hooks. They are loaded once during the server boot. In this case, we will create a local hook.
The following example has worked with strapi#3.6.
Create a hook for socket.io at ./hooks/socket.io/index.js
module.exports = strapi => {
return {
async initialize() {
const ioServer = require('socket.io')(strapi.server, {
cors: {
origin: process.env['FRONT_APP_URL'],
methods: ['GET', 'POST'],
/* ...other cors options */
}
})
ioServer.on('connection', function(socket) {
socket.emit('hello', `Welcome ${socket.id}`)
})
/* HANDLE CLIENT SOCKET LOGIC HERE */
// store the server.io instance to global var to use elsewhere
strapi.services.ioServer = ioServer
},
}
}
Enable the new hook in order for Strapi to load it - ./config/hook.js
module.exports = {
settings: {
'socket.io': {
enabled: true,
},
},
};
That's done. You can access the websocket server inside ./config/functions/bootstrap.js or models' lifecycle hooks.
// ./api/employee/models/employee.js
module.exports = {
lifecycles: {
async afterUpdate(result, params, data) {
strapi.services.ioServer.emit('update:employee', result)
},
},
};
For those who are looking the answer using Strapi version 4
var io = require("socket.io")(strapi.server.httpServer)
I'm trying to set up a server that can dynamically create many rooms for many namespaces. I'm currently just trying to broadcast to sockets of a room, when a new socket has joined that room.
So far I have been able to broadcast to a specific namespace and my event listeners on the client receives the message. However when I try to broadcast to a room, of a specific namespace, my event listener doesn't receive that message.
I've turned on the Debugger mode and see the socket.io-client:socket emitting the event with the right payload and event type. So I am not sure what I am missing since the documentation also seems fairly straightforward. Any help would be much appreciated. Below is my code.
Server
const colorNs = io.of('/color');
colorNs.on('connection', (socket) => {
const { id } = socket.handshake.query;
const { id:connId } = socket.conn;
if(id) {
socket.join(id);
socket.broadcast.to(id).emit('user:connect', { id: connId });
}
socket.on('disconnect', () => {
const { id } = socket.handshake.query;
const { id:connId } = socket.conn;
socket.broadcast.to(id).emit('user:disconnect', { id: connId });
});
});
Client
const socket = io('/color?id="123"');
socket.on('user:connect', () => console.log('data', data));
Client - Debug Trace
socket.io-parser decoded 2/color,["user:connect",{"id":"IZTTPidF121JCzf9AAAO"}] as {"type":2,"nsp":"/color","data":["user:connect",{"id":"IZTTPidF121JCzf9AAAO"}]} +1ms
browser.js:133
socket.io-client:socket emitting event ["user:connect",{"id":"IZTTPidF121JCzf9AAAO"}] +3ms