AMQPNETLITE - ActiveMQ Artemis (Red Hat AMQ) - autocreate multi-consumer multicast queue - amqp

This qeuestion is on consuming the messages using AMQP in .Net. The documentation recommends amqpnetlite: https://access.redhat.com/documentation/en-us/red_hat_amq/7.0/html-single/using_the_amq_.net_client/index
On subscribing to an address using AMQPNetLite, the address and the queue will be auto-created. The auto-created queue is always "unicast" though. I have not been able to auto-create
a multicast queue
that allowed any number of consumers.
Code:
private async Task RenewSession()
{
Connect = await Connection.Factory.CreateAsync(new Address("amqp://admin:admin#localhost:5672"), new Open() {ContainerId = "client-1"});
MqSession = new Session(Connect);
var receiver = new ReceiverLink(MqSession, DEFAULT_SUBSCRIPTION_NAME, GetSource("test-topic"), null);
receiver.Start(100, OnMessage);
}
private Source GetSource(string address)
{
var source = new Source
{
Address = address,
ExpiryPolicy = new Symbol("never"),
Durable = 2,
DefaultOutcome = new Modified
{
DeliveryFailed = true,
UndeliverableHere = false
}
};
return source;
}
Maybe I am missing some flags?

in AMQP, you choose between autocreating a queue (anycast routing) or a topic (multicast routing) by setting a capability.
The capability should be either new Symbol("queue") or new Symbol("topic").
public class SimpleAmqpTest
{
[Fact]
public async Task TestHelloWorld()
{
Address address = new Address("amqp://guest:guest#localhost:5672");
Connection connection = await Connection.Factory.CreateAsync(address);
Session session = new Session(connection);
Message message = new Message("Hello AMQP");
Target target = new Target
{
Address = "q1",
Capabilities = new Symbol[] { new Symbol("queue") }
};
SenderLink sender = new SenderLink(session, "sender-link", target, null);
await sender.SendAsync(message);
Source source = new Source
{
Address = "q1",
Capabilities = new Symbol[] { new Symbol("queue") }
};
ReceiverLink receiver = new ReceiverLink(session, "receiver-link", source, null);
message = await receiver.ReceiveAsync();
receiver.Accept(message);
await sender.CloseAsync();
await receiver.CloseAsync();
await session.CloseAsync();
await connection.CloseAsync();
}
}
Have a look at https://github.com/Azure/amqpnetlite/issues/286, where the code comes from.
You can choose whether the default routing will be multicast or anycast by setting default-address-routing-type in broker.xml, everything documented at https://activemq.apache.org/artemis/docs/2.6.0/address-model.html
The broker's multicastPrefix and anycastPrefix feature is not implemented for AMQP. https://issues.jboss.org/browse/ENTMQBR-795

Related

Kucoin .net sdk, problems with websockets connections

I am using Kucoin SDK for .Net created by a third party, the problem I have is that the connections to the websockets are not constant, that is, they do not work 24/7, the documentation I found about it SDK is very limited.
This is how I am configuring the websockets client.
Cliente_Kucoin = new KucoinSocketClient(new Objects.KucoinSocketClientOptions()
{
ApiCredentials = new Objects.KucoinApiCredentials("xxxxxxxxxxxx", "xxxxxxxx", "xxxxx"),
AutoReconnect = true,
ReconnectInterval = TimeSpan.FromSeconds(5),
LogLevel = Microsoft.Extensions.Logging.LogLevel.Trace,
SocketSubscriptionsCombineTarget = 2
});
And so, I make the subscriptions to the sockets.
private async void Listener_Cryptos()
{
var BTC = await ClienteKucoin.Spot.SubscribeToTickerUpdatesAsync(BTC.Tickets, BTC.Trade);
var ETH = await ClienteKucoin.Spot.SubscribeToTickerUpdatesAsync(ETH.Tickets, ETH.Trade);
var BNB = await ClienteKucoin.Spot.SubscribeToTickerUpdatesAsync(BNB.Tickets, BNB.Trade);
var ADA = await ClienteKucoin.Spot.SubscribeToTickerUpdatesAsync(ADA.Tickets, ADA.Trade);
var XRP = await ClienteKucoin.Spot.SubscribeToTickerUpdatesAsync(XRP.Tickets, XRP.Trade);
}

BroadcastBlock missing items

I have a list of project numbers that I need to process. A project could have about 8000 items and I need to get the data for each item in the project and then push this data into a list of servers. Can anybody please tell me the following..
1) I have 1000 items in iR but only 998 were written to the servers. Did I loose items by using broadCastBlock?
2) Am I doing the await on all actionBlocks correctly?
3) How do I make the database call async?
Here is the database code
public MemcachedDTO GetIR(MemcachedDTO dtoItem)
{
string[] Tables = new string[] { "iowa", "la" };
using (SqlConnection connection = new SqlConnection(ConfigurationManager.ConnectionStrings["test"].ConnectionString))
{
using (SqlCommand command = new SqlCommand("test", connection))
{
DataSet Result = new DataSet();
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add("#ProjectId", SqlDbType.VarChar);
command.Parameters["#ProjectId"].Value = dtoItem.ProjectId;
connection.Open();
Result.EnforceConstraints = false;
Result.Load(command.ExecuteReader(CommandBehavior.CloseConnection), LoadOption.OverwriteChanges, Tables);
dtoItem.test = Result;
}
}
return dtoItem;
}
Update:
I have updated the code to the below. It just hangs when I run it and only writes 1/4 of the data to the server? Can you please let me know what I am doing wrong?
public static ITargetBlock<T> CreateGuaranteedBroadcastBlock<T>(IEnumerable<ITargetBlock<T>> targets, DataflowBlockOptions options)
{
var targetsList = targets.ToList();
var block = new ActionBlock<T>(
async item =>
{
foreach (var target in targetsList)
{
await target.SendAsync(item);
}
}, new ExecutionDataflowBlockOptions
{
CancellationToken = options.CancellationToken
});
block.Completion.ContinueWith(task =>
{
foreach (var target in targetsList)
{
if (task.Exception != null)
target.Fault(task.Exception);
else
target.Complete();
}
});
return block;
}
[HttpGet]
public async Task< HttpResponseMessage> ReloadItem(string projectQuery)
{
try
{
var linkCompletion = new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 2
};
var cts = new CancellationTokenSource();
var dbOptions = new DataflowBlockOptions { CancellationToken = cts.Token };
IList<string> projectIds = projectQuery.Split(',').ToList();
IEnumerable<string> serverList = ConfigurationManager.AppSettings["ServerList"].Split(',').Cast<string>();
var iR = new TransformBlock<MemcachedDTO, MemcachedDTO>(
dto => dto.GetIR(dto), new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 3 });
List<ActionBlock<MemcachedDTO>> actionList = new List<ActionBlock<MemcachedDTO>>();
List<MemcachedDTO> dtoList = new List<MemcachedDTO>();
foreach (string pid in projectIds)
{
IList<MemcachedDTO> dtoTemp = new List<MemcachedDTO>();
dtoTemp = MemcachedDTO.GetItemIdsByProject(pid);
dtoList.AddRange(dtoTemp);
}
foreach (string s in serverList)
{
var action = new ActionBlock<MemcachedDTO>(
async dto => await PostEachServerAsync(dto, s, "setitemcache"));
actionList.Add(action);
}
var bBlock = CreateGuaranteedBroadcastBlock(actionList, dbOptions);
foreach (MemcachedDTO d in dtoList)
{
await iR.SendAsync(d);
}
iR.Complete();
iR.LinkTo(bBlock);
await Task.WhenAll(actionList.Select(action => action.Completion).ToList());
return Request.CreateResponse(HttpStatusCode.OK, new { message = projectIds.ToString() + " reload success" });
}
catch (Exception ex)
{
return Request.CreateResponse(HttpStatusCode.InternalServerError, new { message = ex.Message.ToString() });
}
}
1) I have 1000 items in iR but only 998 were written to the servers. Did I loose items by using broadCastBlock?
Yes in the code below you set BoundedCapacity to one, if at anytime your BroadcastBlock cannot pass along an item it will drop it. Additionally a BroadcastBlock will only propagate Completion to one TargetBlock, do not use PropagateCompletion=true here. If you want all blocks to complete you need to handle Completion manually. This can be done by setting the ContinueWith on the BroadcastBlock to pass Completion to all of the connected targets.
var action = new ActionBlock<MemcachedDTO>(dto => PostEachServerAsync(dto, s, "set"), new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 3, BoundedCapacity = 1 });
broadcast.LinkTo(action, linkCompletion);
actionList.Add(action);
Option: Instead of the BroadcastBlock use a properly bounded BufferBlock. When your downstream blocks are bound to one item they cannot receive additional items until they finish processing what they have. That will allow the BufferBlock to offer its items to another, possibly idle, ActionBlock.
When you add items into a throttled flow, i.e. a flow with a BoundedCapacity less than Unbounded. You need to be using the SendAsync method or at least handling the return of Post. I'd recommend simply using SendAsync:
foreach (MemcachedDTO d in dtoList)
{
await iR.SendAsync(d);
}
That will force your method signature to become:
public async Task<HttpResponseMessage> ReloadItem(string projectQuery)
2) Am I doing the await on all actionBlocks correctly?
The previous change will permit you to loose the blocking Wait call in favor of a await Task.WhenAlll
iR.Complete();
actionList.ForEach(x => x.Completion.Wait());
To:
iR.Complete();
await bufferBlock.Completion.ContinueWith(tsk => actionList.ForEach(x => x.Complete());
await Task.WhenAll(actionList.Select(action => action.Completion).ToList());
3) How do I make the database call async?
I'm going to leave this open because it should be a separate question unrelated to TPL-Dataflow, but in short use an async Api to access your Db and async will naturally grow through your code base. This should get you started.
BufferBlock vs BroadcastBlock
After re-reading your previous question and the answer from #VMAtm. It seems you want each item sent to All five servers, in that case you will need a BroadcastBlock. You would use a BufferBlock to distribute the messages relatively evenly to a flexible pool of servers that each could handle a message. None the less, you will still need to take control of propagating completion and faults to all the connected ActionBlocks by awaiting the completion of the BroadcastBlock.
To Prevent BroadcastBlock Dropped Messages
In general you two options, set your ActionBlocks to be unbound, which is their default value:
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 3, BoundedCapacity = Unbounded });
Or broadcast messages your self from any variety of your own construction. Here is an example implementation from #i3arnon. And another from #svick

NetMQ client to client messaging

I'm trying to create an rpc program to communicate hosts located on different networks and chose Router-Dealer configuration of NetMQ provided here: http://netmq.readthedocs.io/en/latest/router-dealer/#router-dealer
But the problem is that router always selects a random dealer when routing a message to backend.
Code which I used :
using (var frontend = new RouterSocket(string.Format("#tcp://{0}:{1}", "127.0.0.1", "5556")))//"#tcp://10.0.2.218:5559"
using (var backend = new DealerSocket(string.Format("#tcp://{0}:{1}", "127.0.0.1", "5557")))//"#tcp://10.0.2.218:5560"
{
// Handler for messages coming in to the frontend
frontend.ReceiveReady += (s, e) =>
{
Console.WriteLine("message arrived on frontEnd");
NetMQMessage msg = e.Socket.ReceiveMultipartMessage();
string clientAddress = msg[0].ConvertToString();
Console.WriteLine("Sending to :" + clientAddress);
//TODO: Make routing here
backend.SendMultipartMessage(msg); // Relay this message to the backend };
// Handler for messages coming in to the backend
backend.ReceiveReady += (s, e) =>
{
Console.WriteLine("message arrived on backend");
var msg = e.Socket.ReceiveMultipartMessage();
frontend.SendMultipartMessage(msg); // Relay this message to the frontend
};
using (var poller = new NetMQPoller { backend, frontend })
{
// Listen out for events on both sockets and raise events when messages come in
poller.Run();
}
}
Code for Client:
using (var client = new RequestSocket(">tcp://" + "127.0.0.1" + ":5556"))
{
var messageBytes = UTF8Encoding.UTF8.GetBytes("Hello");
var messageToServer = new NetMQMessage();
//messageToServer.AppendEmptyFrame();
messageToServer.Append("Server2");
messageToServer.Append(messageBytes);
WriteToConsoleVoid("======================================");
WriteToConsoleVoid(" OUTGOING MESSAGE TO SERVER ");
WriteToConsoleVoid("======================================");
//PrintFrames("Client Sending", messageToServer);
client.SendMultipartMessage(messageToServer);
NetMQMessage serverMessage = client.ReceiveMultipartMessage();
WriteToConsoleVoid("======================================");
WriteToConsoleVoid(" INCOMING MESSAGE FROM SERVER");
WriteToConsoleVoid("======================================");
//PrintFrames("Server receiving", clientMessage);
byte[] rpcByteArray = null;
if (serverMessage.FrameCount == 3)
{
var clientAddress = serverMessage[0];
rpcByteArray = serverMessage[2].ToByteArray();
}
WriteToConsoleVoid("======================================");
Console.ReadLine();
}
Code for Dealer:
using (var server = new ResponseSocket())
{
server.Options.Identity = UTF8Encoding.UTF8.GetBytes(confItem.ResponseServerID);
Console.WriteLine("Server ID:" + confItem.ResponseServerID);
server.Connect(string.Format("tcp://{0}:{1}", "127.0.0.1", "5557"));
using (var poller = new NetMQPoller { server })
{
server.ReceiveReady += (s, a) =>
{
byte[] response = null;
NetMQMessage serverMessage = null;
try
{
serverMessage = a.Socket.ReceiveMultipartMessage();
}
catch (Exception ex)
{
Console.WriteLine("Exception on ReceiveMultipartMessage : " + ex.ToString());
//continue;
}
byte[] eaBody = null;
string clientAddress = "";
if (serverMessage.FrameCount == 2)
{
clientAddress = serverMessage[0].ConvertToString();
Console.WriteLine("ClientAddress:" + clientAddress);
eaBody = serverMessage[1].ToByteArray();
Console.WriteLine("Received message from remote computer: {0} bytes , CurrentID : {1}", eaBody.Length, confItem.ResponseServerID);
}
else
{
Console.WriteLine("Received message from remote computer: CurrentID : {0}", confItem.ResponseServerID);
}
};
poller.Run();
}
}
Is it possible to choose a specific backend on frontend.ReceiveReady?
Thanks!
Your backend should be router as well. You need the worker to register or you need to know all the available workers and their identity. When send on the backend push the worker identity at the beginning of the server.
Take a look at the Majordomo example in the zeromq guide:
http://zguide.zeromq.org/page:all#toc72
http://zguide.zeromq.org/page:all#toc98

How to use ZMQ when Server on IP other than the client

I learned how to use ZeroMQ on a localhost, but I failed to do it on a remote IP.
Q1: Do I need a broker?If so,Q2: which broker and how to do it.?
Update:
OK. I'm using the ZMQ Weather Update example but with a remote IP ( not the localhost ). Here is what I do using C# ZMQ bindings ( however, I'm OK to use any other language ):
ZMQ Server:
using (var context = new ZContext())
using (var publisher = new ZSocket(context, ZSocketType.PUB))
{
string address = "tcp://*:5001";
publisher.Bind(address);
publisher.Send("msg")
}
Proxy:
using (var context = new ZContext())
using (var frontend = new ZSocket(context, ZSocketType.XSUB))
using (var backend = new ZSocket(context, ZSocketType.XPUB))
{
// Frontend is where the weather server sits
string localhost = "tcp://127.0.0.1:5001";
Console.WriteLine("I: Connecting to {0}", localhost);
frontend.Connect(localhost);
// Backend is our public endpoint for subscribers
string remoteIP = "216.123.23.98"; // For example
var tcpAddress = string.Format("tcp://{0}:8100", remoteIP); // I also tried localhost address here
Console.WriteLine("I: Binding on {0}", tcpAddress);
backend.Bind(tcpAddress);
var epgmAddress = string.Format("epgm://localhost;{0}:8100", remoteIP);
Console.WriteLine("I: Binding on {0}", epgmAddress);
backend.Bind(epgmAddress);
using (var subscription = ZFrame.Create(1))
{
subscription.Write(new byte[] { 0x1 }, 0, 1);
backend.Send(subscription);
}
// Run the proxy until the user interrupts us
ZContext.Proxy(frontend, backend);
}
Client:
using (var context = new ZContext())
using (var subscriber = new ZSocket(context, ZSocketType.SUB))
{
string remoteIP = "tcp://216.123.23.98"; //For example
Console.WriteLine("I: Connecting to {0}…", remoteIP);
subscriber.Connect(connect_to);
// Subscribe to zipcode
string zipCode = args[0];
Console.WriteLine("I: Subscribing to zip code {0}…", zipCode);
subscriber.Subscribe(zipCode);
// Process 10 updates
int i = 0;
long total_temperature = 0;
for (; i < 20; ++i)
{
ZError err;
using (var replyFrame = subscriber.ReceiveFrame(out err))
{
string reply = replyFrame.ReadString(Encoding.ASCII);
Console.WriteLine(reply);
total_temperature += Convert.ToInt64(reply.Split(' ')[1]);
}
}
Console.WriteLine("Average temperature for zipcode '{0}' was {1}", zipCode, (total_temperature / i));
}
When I run this I get error in Server and error in proxy - server gets
Invalid end point
and proxy gets EINVAL(22):
Invalid argument at ZeroMQ.ZSocket.Bind(String endpoint)
A1: No, ZeroMQ is a Broker-less messaging framework.
A2: N/A
How to repair the code?
All the services need to obey respective transport-class addressing rules, for the TCP/IP case - both the .bind() / .connect() methods have to state both parts of the IP:PORT# specification ( with some aids from DNS-resolution for the IP-part, but the :PORT#-part is still mandatory )
( which the source-code does not meet in client, ref.:
subscriber.Connect(connect_to);
whereas there ought be also a Proxy-side matching :PORT#, i.e.:8100, specified, for a correct .connect() ).
For the clarity and for avoiding a port#-collision, remove the epgm transport class from the code.

WCF authentication issue on a particular machine

I have an issue with my WCF service/application. I am using TCP binding with Windows Authentication (dynamically setting the credentials).
The service is hosted on machine A. With my test client application on machine B, C, D, and E, it works fine. When the test client application is run on machine A (on itself), it is telling me that I have invalid credentials when in fact they are correct.
When hosting the service on machine B, the test client on machine B itself authenticates fine. Even if the test client is on A, it authenticates itself successfully on B.
I would say there is something wrong with machine A but I cannot figure out what.
Any help?
Thanks in advance.
K
Client Config:
public static EndpointAddress BuildServiceEndpointAddress()
{
var host = Settings.Default.Host;
var port = Settings.Default.Port;
string endpointAddress;
if (port == 0)
endpointAddress = string.Format("net.tcp://{0}/Service", host);
else
endpointAddress = string.Format("net.tcp://{0}:{1}/Service", host, port);
Console.WriteLine("Endpoint: {0}", endpointAddress);
//var endpointAddressObj = new EndpointAddress(endpointAddress );
return new EndpointAddress(new Uri(endpointAddress), new DnsEndpointIdentity("MyCert"));
}
public static IChannelFactory<IService> BuildChannelFactory(EndpointAddress endpointAddress)
{
var netTcpBinding = new NetTcpBinding()
{
MaxReceivedMessageSize = int.MaxValue,
Security = new NetTcpSecurity()
{
Mode = SecurityMode.Transport,
Transport = new TcpTransportSecurity()
{
ClientCredentialType = TcpClientCredentialType.Windows,
}
},
TransferMode = TransferMode.Streamed
};
var factory = new ChannelFactory<IService>(netTcpBinding);
factory.Endpoint.Address = endpointAddress;
factory.Credentials.Windows.ClientCredential.Domain = Settings.Default.Domain;
factory.Credentials.Windows.ClientCredential.UserName = Settings.Default.Username;
factory.Credentials.Windows.ClientCredential.Password = Settings.Default.Password;
Console.WriteLine("Opening Channel Factory ... ");
factory.Open();
Console.WriteLine("Opening Channel Factory ... done");
return factory;
}
Service Config:
var netTcpBinding = new NetTcpBinding()
{
MaxReceivedMessageSize = int.MaxValue,
Security = new NetTcpSecurity()
{
Mode = SecurityMode.Transport,
Transport = new TcpTransportSecurity()
{
ClientCredentialType = TcpClientCredentialType.Windows,
}
},
TransferMode = TransferMode.Streamed
};
var endpoint = serviceHost.AddServiceEndpoint(
typeof(IService),
netTcpBinding,
new Uri(string.Format("net.tcp://0:{0}/Service", tcpPort)));
ServiceSecurityAuditBehavior newAudit = new ServiceSecurityAuditBehavior();
newAudit.AuditLogLocation = AuditLogLocation.Application;
newAudit.MessageAuthenticationAuditLevel = AuditLevel.SuccessOrFailure;
newAudit.ServiceAuthorizationAuditLevel = AuditLevel.SuccessOrFailure;
newAudit.SuppressAuditFailure = false;
serviceHost.AddServiceEndpoint(ServiceMetadataBehavior.MexContractName,
MetadataExchangeBindings.CreateMexTcpBinding(),
string.Format("net.tcp://localhost:{0}/Service/mex", tcpPort));
serviceHost.Description.Behaviors.Remove<ServiceSecurityAuditBehavior>();
serviceHost.Description.Behaviors.Add(newAudit);
serviceHost.Opening += (sender, eventArgs) => LogDebug("Opening Connection ...");
serviceHost.Opened += (sender, eventArgs) => LogDebug("Opened Connection ...");
serviceHost.Closing += (sender, eventArgs) => LogDebug("Closing connection ...");
serviceHost.Closed += (sender, eventArgs) => LogDebug("Closed connection ...");
serviceHost.Faulted += (sender, eventArgs) => LogError("Fault detected on WCF host");
I just fixed the issue!
I replaced:
return new EndpointAddress(new Uri(endpointAddress), new
DnsEndpointIdentity("MyCert"));
with
return new EndpointAddress(new Uri(endpointAddress));
since the certificate is not needed for windows authentication.
Although I still am confused why it worked on other machines.

Resources