I am building a distributed workflow orchestrator, grpc is used to communicate with the server cluster by workers.If a new server is added to the server grpc client is not able to detect this change. However i have done a workaround by adding a max connection age to the server options
grpc.KeepaliveParams(keepalive.ServerParameters{
MaxConnectionAge: time.Minute * 1,
})
We have two implementation of workers, one in golang and other in java this workaround works perfectly in golang client. Every minute the client makes new connection and is able to detect new servers in cluster. But this is not working with java client.
public CustomNameResolverFactory(String host, int port) {
ManagedChannel managedChannel = NettyChannelBuilder
.forAddress(host, port)
.withOption( ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000 )
.usePlaintext().build();
GetServersRequest request = GetServersRequest.newBuilder().build();
GetServersResponse servers = TaskServiceGrpc.newBlockingStub(managedChannel).getServers(request);
List<Server> serversList = servers.getServersList();
System.out.println(servers);
LOGGER.info("found servers {}", servers);
for (Server server : serversList) {
String rpcAddr = server.getRpcAddr();
String[] split = rpcAddr.split(":");
String hostName = split[0];
int portN = Integer.parseInt(split[1]);
addresses.add(new EquivalentAddressGroup(new InetSocketAddress(hostName, portN)));
}
}
Java client code- https://github.com/Mohitkumar/orchy-worker-java/blob/master/src/main/java/com/orchy/client/CustomNameResolverFactory.java
Golang client code- https://github.com/Mohitkumar/orchy/blob/main/worker/lb/resolver.go
Related
I have a use case where I need to send 2 requests to the server. The output of first request is used in second request so the calls have to be synchronous. I am using ktor (OkHttp)client websocket for this. I am failing at first attempt to even connect to the server with this error
Exception in thread "main" java.net.UnknownHostException: https: nodename nor servname provided, or not known
I suspect I haven't split my url properly and thats why its not able to connect to host.
Couple of qns
Is there any benefit to using websocket instead of using 2 separate Http requests?
Is there a way I can just pass URL to the websocket request?
Best and easiest way to get response and send another request?
I have been able to find very limited documentation on ktor client websocket.
const val HOST = "https://sample.com"
const val PATH1 = "/path/to/config?val1=<val1>&val2=<val2>"
const val PATH2 = "/path/to/config?val=<response_from_first_req>"
fun useSocket() {
val client = HttpClient() {
install(WebSockets)
}
runBlocking {
client.webSocket(method = HttpMethod.Get, host = HOST, path = PATH1) {
val othersMessage = incoming.receive() as? Frame.Text
println(othersMessage?.readText())
println("Testing")
}
}
client.close()
}
Thanks in advance.
I am using Vertx for my backend.
This is a TCP server and the server is connected to several clients.
I am trying to disconnect the client when reaching a certain condition.
The code that I used is as follows.
vertx.createNetServer(new NetServerOptions().setIdleTimeout(601))
.connectHandler(socket -> {
Instant start = Instant.now();
writerId = socket.writeHandlerID();
log.info("[TCPServerVerticle] first Tcp Server Instance Id : {}", serverId);
socket.handler(input -> { // input을 받았을 때 실행
writerId = socket.writeHandlerID();
SocketAddress localAddr = socket.localAddress();
SocketAddress remoteAddr = socket.remoteAddress();
central.setWriterId(writerId);
byte[] bytes = input.getBytes();
String inputString = Utils.byteArrayToHex(bytes);
central.inputMessage(inputString, writerId, vertx, localAddr, remoteAddr, versionMap).onComplete(ok -> {
String result = ok.result();
if (result.equals("nak")) {
socket.close();
}
});
});
When I execute this code, when the condition for "nak" is met, the server seems to restart and not the client.
Would there be a way to close the connection to the client without restarting the server?
Thank you in advance
I am trying to connect to RSK Mainnet or RSK Testnet over websockets.
Here's what I tried for Mainnet:
const wsProvider = new Web3.providers.WebsocketProvider('ws://public-node.rsk.co');
const web3 = new Web3(wsProvider);
web3.eth.subscribe('newBlockHeaders', function(error, blockHeader){
if (!error) {
console.log("new blockheader " + blockHeader.number)
} else {
console.error(error);
}
});
with this result:
connection not open on send()
Error: connection not open
And I did the same with Testnet but using ws://public-node.testnet.rsk.co, getting similar outcome.
Neither of these work, as seen in the errors above.
How can I connect?
Milton
I am not sure, but I think websocket is not enabled in public nodes.
Usually it is not enabled in others public blockchain nodes that I know.
RSK public nodes expose JSON-RPC endpoints only over HTTP.
They do not expose JSON-RPC endpoints over websockets,
so unfortunately, you are not able to do exactly what you have described.
However, you can achieve something equivalent
by running your own RSK node,
and use this to establish websockets connections.
Here are the RSK
configuration options for RPC .
Also, you can see the default configuration values
in the "base" configuration file, for
rpc.providers.ws
ws {
enabled = false
bind_address = localhost
port = 4445
}
Additionally, you should include the /websocket suffix in your endpoint. Default websocket endpoint when running your own node is: ws://localhost:4445/websocket.
Therefore, update the initial part of your code,
such that it looks like this:
const wsProvider = new Web3.providers.WebsocketProvider('ws://localhost:4445/websocket');
const web3 = new Web3(wsProvider);
So I was looking into running socket.io across multiple processes.
The guide here: https://socket.io/docs/using-multiple-nodes/ left me with some questions.
It mentions using configuring nginx to load balance between socket.io processes, but it also mentions using the built in cluster module in Node.js below.
Am I supposed to be using nginx AND the cluster module in Node.js for this?
Also how do I tell if load balancing is working?
I tested it using the nginx option with two socket.io processes running using the redis adapter and using the cluster module.
This is what I had in my nginx config:
http {
upstream io_nodes {
ip_hash;
server 127.0.0.1:6001;
server 127.0.0.1:6002;
}
server {
listen 3000;
server_name example.com;
location / {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_pass http://io_nodes;
}
}
This is an example of my socket.io code (most of it taken from here: https://github.com/elad/node-cluster-socket.io):
var express = require('express'),
cluster = require('cluster'),
net = require('net'),
redis = require('redis'),
sio = require('socket.io'),
sio_redis = require('socket.io-redis');
var port = 6001,
num_processes = require('os').cpus().length;
if (cluster.isMaster) {
console.log('is master 6001');
// This stores our workers. We need to keep them to be able to reference
// them based on source IP address. It's also useful for auto-restart,
// for example.
var workers = [];
// Helper function for spawning worker at index 'i'.
var spawn = function(i) {
workers[i] = cluster.fork();
// Optional: Restart worker on exit
workers[i].on('exit', function(code, signal) {
console.log('respawning worker', i);
spawn(i);
});
};
// Spawn workers.
for (var i = 0; i < num_processes; i++) {
spawn(i);
}
// Helper function for getting a worker index based on IP address.
// This is a hot path so it should be really fast. The way it works
// is by converting the IP address to a number by removing non numeric
// characters, then compressing it to the number of slots we have.
//
// Compared against "real" hashing (from the sticky-session code) and
// "real" IP number conversion, this function is on par in terms of
// worker index distribution only much faster.
var worker_index = function(ip, len) {
var s = '';
for (var i = 0, _len = ip.length; i < _len; i++) {
if (!isNaN(ip[i])) {
s += ip[i];
}
}
return Number(s) % len;
};
// Create the outside facing server listening on our port.
var server = net.createServer({ pauseOnConnect: true }, function(connection) {
// We received a connection and need to pass it to the appropriate
// worker. Get the worker for this connection's source IP and pass
// it the connection.
var worker = workers[worker_index(connection.remoteAddress, num_processes)];
worker.send('sticky-session:connection', connection);
}).listen(port);
} else {
// Note we don't use a port here because the master listens on it for us.
var app = new express();
// Here you might use middleware, attach routes, etc.
// Don't expose our internal server to the outside.
var server = app.listen(0, 'localhost'),
io = sio(server);
// Tell Socket.IO to use the redis adapter. By default, the redis
// server is assumed to be on localhost:6379. You don't have to
// specify them explicitly unless you want to change them.
io.adapter(sio_redis({ host: 'localhost', port: 6379 }));
// Here you might use Socket.IO middleware for authorization etc.
io.on('connection', function(socket) {
console.log('port 6001');
console.log(socket.id);
});
// Listen to messages sent from the master. Ignore everything else.
process.on('message', function(message, connection) {
if (message !== 'sticky-session:connection') {
return;
}
// Emulate a connection event on the server by emitting the
// event with the connection the master sent us.
server.emit('connection', connection);
connection.resume();
});
}
Connections worked just fine with this, although I'm testing it all locally..
How do I know if it's working properly? Every time the client connects, it seems to connect to the socket.io process on port 6001.
The client connect code connects to port 3000.
Am I supposed to be using nginx AND the cluster module in Node.js for this?
If all your server processes are on one computer, you can use the cluster module without NGINX.
If you're using multiple server computers, then you need a piece of network infrastructure like NGINX to load balance among the different servers since node.js clustering cannot do that for you.
And, you can use both together (multiple servers load balanced by something like NGINX and each server running clustering on each server). The key here is that node.js clustering only spreads the load among different processes on the same host.
Also how do I tell if load balancing is working?
You can have each process log the activity it is processing and add the process ID as part of the logging and if you are loading your server with multiple requests at the same time, you should see some load being handled by each process. If you do actual load testing, you should get significantly more throughput when clustering is on and working vs. not using clustering. Keep in mind that total throughput depends upon where your bottlenecks are so if your server is actually database bound and all clustered processes are using the same database, you may not benefit much from clustering the node.js process. If, on the other hand, your node.js process is compute intensive and you have multiple cores in your server, you may get a significant benefit from clustering.
Adding more point to above solution.
Also how do I tell if load balancing is working?
I am using node-debug for the same, it opens multiple debugger as per number of thread. Now you can add breakpoint to check whether load is being distributed properly.
I try to do a simple cluster using akka.net.
The goal is to have a server receiving request and akka.net processing it through it cluster.
For testing and learning I create a simple WCF service that receive a math equation and I want to send this equation to be solved.
I have one project server and another client.
The configuration on the server side is :
<![CDATA[
akka {
actor {
provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
debug {
receive = on
autoreceive = on
lifecycle = on
event-stream = on
unhandled = on
}
deployment {
/math {
router = consistent-hashing-group #round-robin-pool # routing strategy
routees.paths = [ "/user/math" ]
virtual-nodes-factor = 8
#nr-of-instances = 10 # max number of total routees
cluster {
enabled = on
max-nr-of-instances-per-node = 2
allow-local-routees = off
use-role = math
}
}
}
}
remote {
helios.tcp {
transport-class = "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote"
applied-adapters = []
transport-protocol = tcp
port = 8081
hostname = "127.0.0.1"
}
}
cluster {
seed-nodes = ["akka.tcp://ClusterSystem#127.0.0.1:8081"] # address of seed node
}
}
]]>
On the Client side the configuration is like this :
<![CDATA[
akka {
actor.provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
remote {
log-remote-lifecycle-events = DEBUG
log-received-messages = on
helios.tcp {
transport-class = "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote"
applied-adapters = []
transport-protocol = tcp
port = 0
hostname = 127.0.0.1
}
}
cluster {
seed-nodes = ["akka.tcp://ClusterSystem#127.0.0.1:8081"] # address of the seed node
roles = ["math"] # roles this member is in
}
actor.deployment {
/math {
router = round-robin-pool # routing strategy
routees.paths = ["/user/math"]
nr-of-instances = 10 # max number of total routees
cluster {
enabled = on
allow-local-routees = on
use-role = math
max-nr-of-instances-per-node = 10
}
}
}
}
]]>
The cluster connection seems to correctly be made. I see the status [UP] and the association with the role "math" that appeared on the server side.
Event follwing the example on the WebCramler, I don't achieved to make a message to be delivered. I always get a deadletters.
I try like this :
actor = sys.ActorOf(Props.Empty.WithRouter(FromConfig.Instance), "math");
or
var actor = sys.ActorSelection("/user/math");
Does someone know a good tutorial or could help me ?
Thanks
Some remarks:
First: assuming your sending work from the server to the client. Then you are effectively remote deploying actors on your client.
Which means only the server node needs the actor.deployment config section.
The client only needs the default cluster config (and your role setting ofcourse).
Second: Try to make it simpler first. Use a round-robin-pool instead. Its much simpler. Try to get that working. And work your way up from there.
This way its easier to eliminate configuration/network/other issues.
Your usage: actor = sys.ActorOf(Props.Empty.WithRouter(FromConfig.Instance), "math"); is correct.
A sample of how your round-robin-pool config could look:
deployment {
/math {
router = round-robin-pool # routing strategy
nr-of-instances = 10 # max number of total routees
cluster {
enabled = on
max-nr-of-instances-per-node = 2
allow-local-routees = off
use-role = math
}
}
}
Try this out. And let me know if that helps.
Edit:
Ok after looking at your sample. Some things i changed
ActorManager->Process: Your creating a new router actor per request. Don't do that. Create the router actor once, and reuse the IActorRef.
Got rid of the minimal cluster size settings in the MathAgentWorker project
Since your not using remote actor deployment. I changed the round-robin-pool to a round-robin-group.
After that it worked.
Also remember that if your using the consistent-hashing-group router you need to specify the hashing key. There are various ways to do that, in your sample i think the easiest way would be to wrap the message your sending to your router in a ConsistentHashableEnvelope. Check the docs for more information.
Finally the akka deployment sections looked like this:
deployment {
/math {
router = round-robin-group # routing strategy
routees.paths = ["/user/math"]
cluster {
enabled = on
allow-local-routees = off
use-role = math
}
}
}
on the MathAgentWorker i only changed the cluster section which now looks like this:
cluster {
seed-nodes = ["akka.tcp://ClusterSystem#127.0.0.1:8081"] # address of the seed node
roles = ["math"] # roles this member is in
}
And the only thing that the ActorManager.Process does is:
return await Program.Instance.RouterInstance.Ask<TResult>(msg, TimeSpan.FromSeconds(10));