I have some endpoint configuration code in an Autofac module that's registering consumers based on conventions that I'd like to unit test. I'm not trying to verify any behaviour of any consumers I just want to check that my setup code is doing what I need it to do. I'm using InMemoryTestHarness but consuming doesn't seem to be working and I'm not sure about the correlation between configuring the bus and registering consumer test harnesses.
To allow the host to be swapped between Rabbit for prod and in memory for tests I have this in my module:
Func<Action<IReceiveConfigurator>, IBusControl> BusFactory = receiveConfig => Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Host(rabbitMqUrl, hostCfg =>
{
hostCfg.Username(rabbitMqUsername);
hostCfg.Password(rabbitMqPassword);
});
receiveConfig(cfg);
});
For the actual consumer registration in my module I have:
// code to scan assembly and build a list of queue definitions with consumers
...
// consumer registration
builder.AddMassTransit(x =>
{
foreach(var consumerType in consumerTypes)
x.AddConsumer(consumerType);
x.AddBus(context => BusFactory(cfg =>
{
foreach(var queueDef in queueDefs)
cfg.ReceiveEndpoint(queueDef.QueueName, e =>
{
foreach(var consumerDef in queueDef.ConsumerDefs)
e.ConfigureConsumer(context, consumerDef.ConsumerType);
});
});
});
For the unit test setup I am doing:
harness = new InMemoryTestHarness();
var module = new MassTransitModule(typeof(TestMessageConsumer).Assembly)
{
BusFactory = (receiveConfig) =>
{
harness.OnConfigureBus += cfg => receiveConfig(cfg);
Task.WaitAll(harness.Start());
return harness.BusControl;
}
};
var builder = new ContainerBuilder();
builder.RegisterModule(module);
container = builder.Build();
// ensure bus initialisation runs
container.Resolve<IBusControl>();
I've verified in the unit test that Autofac can resolve IBus, IBusControl and concrete consumer classes, as well as, given a message type T an IConsumer<T>.
In my tests, if I do:
await harness.InputQueueSendEndpoint.Send(new TestMessage());
harness.Consumed.Select<TestMessage>().Any().ShouldBeTrue();
then first the test waits on the harness.Consumed line for 30 seconds then the test fails (Any() returns false). I get the same behaviour if I register a consumer harness - plus I'm worried that registering a consumer harness doesn't actually verify my registration.
Have I misunderstood something with the test harness? How would I verify that my consumer config is correct? Is the harness.Consume line taking 30 seconds an indication that I've completely misused the test harness? So many questions...
Thanks,
Daniel
EDIT
Based on the comment from Chris Patterson I've updated my registration to use the MassTransit Autofac integration methods (code updated above) but still getting the same problem.
The test harness creates its own bus instance, and the Consumer, Saga, etc. methods add additional harnesses to that same test harness. If you're resolving a bus from the container as part of your test, you're stuck using that bus. The one in the harness is of no use to you, as are the methods in that harness.
You should separate the testing of your consumers from testing the container registration. And while you're at it, why not use the built-in container support for configuring endpoints, etc. instead of writing it yourself? I believe there is an extension method for .AddMassTransit to AddConsumersFromContainer where you specify the container. This makes it usable with previously loaded modules that added consumers to the container, where the bus is in its own module.
Related
I'm working on a POC for using MassTransit sagas to handle state changes in a system for grant applications. I'm using MassTransit 8.0.0-develop.394, .Net 6, EF Core 6.0.2 and ActiveMQ Artemis 1.19.0.
In the final solution the applicants can register their application and prepare the data for several weeks. A few days before the deadline another external system will be populated with data that will be used to validate the application data. Application data entered before the validation data is populated should just be scheduled for later validation, but data entered after should be validated immediately. I think MassTransit sagas with scheduled events looks like a good fit for this.
In the POC I just schedule the validation start time for some 10 seconds into the future from the program starts, and uses a shorter and shorter delay in the schedule until I just schedule it with a delay of TimeSpan.Zero.
From looking in the database I noticed that some of the schedule events somehow get lost when I run the POC with an empty saga repository, but everything works fine when I rerun the the program with existing sagas in the database. I use the same scheduling code in Initially and in DuringAny, which make me think that there might be some limitations on how short delay its safe to use when scheduling saga events?
Note 1: I've switched to not schedule the event in the saga when its less than 1 second to the valdation can be started, then I just publish the validation message directly, so this issue is not blocking me at the moment.
Note 2: I noticed this when running the POC from the command line and checking the database manually. I've tried to reproduce it in a test using the TestHarness, and also using ActiveMQ Artemis and InMemoryRepository, but with no luck. I've been able to reproduce it (more or less consistently) with a test using Artemis and EF Core Repository. I must admit that the test got quite complex with a lot of Task.Delay and other stuff, so it might be hard to follow the logic, but I can post it here if anyone think it's of any help.
Update 2 using Chris Pattersons recommendation about cfg.UseMessageRetry and cfg.UseInMemoryOutbox in the SagaDefinition and not on the bus.
Here is the updated code where MassTransit is configured
private static ServiceProvider BuildServiceProvider()
{
return new ServiceCollection()
.AddDbContext<MySagaDbContext>(builder =>
{
MySagaDbContextFactory.Apply(builder);
})
.AddMassTransit(cfg =>
{
cfg.AddDelayedMessageScheduler();
cfg.UsingActiveMq((context, config) =>
{
config.Host("artemis", 61616, configureHost =>
{
configureHost.Username("admin");
configureHost.Password("admin");
});
config.EnableArtemisCompatibility();
config.UseDelayedMessageScheduler();
config.ConfigureEndpoints(context);
});
cfg.AddSagaStateMachine<MyStateMachine, MySaga, MySagaDefinition<MySaga>>()
.EntityFrameworkRepository(x =>
{
x.ConcurrencyMode = ConcurrencyMode.Optimistic;
x.ExistingDbContext<MySagaDbContext>();
});
})
.AddLogging(configure =>
{
configure.AddFilter("MassTransit", LogLevel.Error); // Filter out all retry warnings
configure.AddFilter("Microsoft", LogLevel.None);
configure.AddSimpleConsole(options =>
{
options.UseUtcTimestamp = true;
options.TimestampFormat = "HH:mm:ss.fff ";
});
})
.BuildServiceProvider(true);
}
Here is the updated saga definition code
public class MySagaDefinition<TSaga> : SagaDefinition<TSaga> where TSaga : class, ISaga
{
protected override void ConfigureSaga(IReceiveEndpointConfigurator endpointConfigurator, ISagaConfigurator<TSaga> consumerConfigurator)
{
endpointConfigurator.UseMessageRetry(r => r.Intervals(10, 50, 100, 500, 1000));
endpointConfigurator.UseInMemoryOutbox();
}
}
If you are scheduling messages from a saga, or really producing any messages from a saga, you should always have the following middleware components configured:
cfg.UseMessageRetry(r => r.Intervals(50,100,1000));
cfg.UseInMemoryOutbox();
That will ensure that messages produced by the saga are:
Only produced if the saga is successfully saved to the repository
Produced after the saga has been saved to the repository
More details are available in the documentation.
The reason being, a short delay is likely delivering the message before it has been saved, and the scheduled event isn't correlating to an existing saga instance because it hasn't saved yet.
We have UserCreated event that gets published from UserManagement.Api. I have two other Apis, Payments.Api and Notification.Api that should react to that event.
In both Apis I have public class UserCreatedConsumer : IConsumer<UserCreated> (so different namespaces) but only one queue (on SQS) gets created for both consumers.
What is the best way to deal with this situation?
You didn't share your configuration, but if you're using:
x.AddConsumer<UserCreatedConsumer>();
As part of your MassTransit configuration, you can specify an InstanceId for that consumer to generate a unique endpoint address.
x.AddConsumer<UserCreatedConsumer>()
.Endpoint(x => x.InstanceId = "unique-value");
Every separate service (not an instance of the same service) needs to have a different queue name of the receiving endpoint, as described in the docs:
cfg.ReceiveEndpoint("queue-name-per-service-type", e =>
{
// rest of the configuration
});
It's also mentioned in the common mistakes article.
We have the following use case:
We have two busses (internal and external). The idea is that our own services use the internal bus and all third-party services use the external bus. We have a created a service that acts as a message router (aptly named MessageRouter).
When a message is published on the internal bus, the MessageRouter can pick up that message and place it on the external bus, and vice-versa. Through configuration we can tell which messages are allowed to pass from internal to external of from external to internal. This in itself works fine.
We have grouped our messages, we have Events, Commands and Requests. Each service has three ReceiveEnpoints, one for each message type. This means that all events are queued on the Events queue etc. Each message to which a service 'subscribes' gets it's own consumer.
var queues = new Dictionary<string, IEnumerable<Type>>
{
// ReSharper disable once PossibleMultipleEnumeration
{ "Events", consumerTypes.Where(ct => ct.GetGenericArguments().Any(ga => typeof(IEvent).IsAssignableFrom(ga))) },
// ReSharper disable once PossibleMultipleEnumeration
{ "Commands", consumerTypes.Where(ct => ct.GetGenericArguments().Any(ga => typeof(ICommand).IsAssignableFrom(ga))) },
// ReSharper disable once PossibleMultipleEnumeration
{ "Queries", consumerTypes.Where(ct => ct.GetGenericArguments().Any(ga => typeof(IQuery).IsAssignableFrom(ga))) }
};
foreach (var queue in queues)
{
config.ReceiveEndpoint(GetConsumerQueueName(queue.Key), cfg =>
{
foreach (var consumerType in queue.Value)
{
cfg.Consumer(consumerType, consumerContainer.Resolve);
}
configurator?.Invoke((T)cfg);
});
}
Where config is a IBusFactoryConfigurator. This code called when the service starts.
What we would like to be able to do in our MessageRouter is to 'dynamically' add and, more importantly, remove consumers from the ReceiveEndpoint.
So far, we haven't had any luck. We have tried to add the Consumer through the use of the ConnectConsumer method on the BusControl instance. This gives us a ConnectHandle which has a Disconnect method. However, when using this approach, messages are not picked up by our consumers. Looking at the handle shows us that it is a MultipleConnectHandle, however, the handle has no 'internal' handles.
Is there any way to use the Consumer method to register the different consumers and to get their ConnectHandles, So that we can Disconnect the consumer if needed?
As stated, ideally we would like to be able to dynamically add and remove consumers to a ReceiveEndpoint.
You can't add/remove consumers on a receive endpoint while the bus is started, that isn't supported in any way, shape, or form.
You can, however, connect new receive endpoints on separate queues with one or more consumers. In your case above, it seems like you are not getting your consumers registered before connecting the receive endpoint, which is why you aren't seeing anything in the handle collection.
Does the Azure Service Bus Subscription client support the ability to use OnMessage Action when the subscription requires a session?
I have a subscription, called "TestSubscription". It requires a sessionId and contains multipart data that is tied together by a SessionId.
if (!namespaceManager.SubscriptionExists("TestTopic", "Export"))
{
var testRule = new RuleDescription
{
Filter = new SqlFilter(#"(Action='Export')"),
Name = "Export"
};
var subDesc = new SubscriptionDescription("DataCollectionTopic", "Export")
{
RequiresSession = true
};
namespaceManager.CreateSubscription(sub`enter code here`Desc, testRule);
}
In a seperate project, I have a Service Bus Monitor and WorkerRole, and in the Worker Role, I have a SubscriptionClient, called "testSubscriptionClient":
testSubscriptionClient = SubscriptionClient.CreateFromConnectionString(connectionString, _topicName, CloudConfigurationManager.GetSetting("testSubscription"), ReceiveMode.PeekLock);
I would then like to have OnMessage triggered when new items are placed in the service bus queue:
testSubscriptionClient.OnMessage(PersistData);
However I get the following message when I run the code:
InvalidOperationException: It is not possible for an entity that requires sessions to create a non-sessionful message receiver
I am using Azure SDK v2.8.
Is what I am looking to do possible? Are there specific settings that I need to make in my service bus monitor, subscription client, or elsewhere that would let me retrieve messages from the subscription in this manner. As a side note, this approach works perfectly in other cases that I have in which I am not using sessioned data.
Can you try this code:
var messageSession=testSubscriptionClient.AcceptMessageSession();
messageSession.OnMessage(PersistData);
beside of this:
testSubscriptionClient.OnMessage(PersistData);
Edit:
Also, you can register your handler to handle sessions (RegisterSessionHandler). It will fire your handle every new action.
I think this is more suitable for your problem.
He shows both way, in this article. It's for queue, but I think you can apply this to topic also.
hi i'm a noob in this but i want to learn how to use the hosebird client, i downloaded it but from the readme don't understand how to use that. I installed eclipse Java EE and maven in my pc but from the README file in hbc don't see how to connect that to my eclipse. can anyone help me with a list of the thing that i have to do?
this is the readme, i have never used maven before.
thanks
## Getting Started
The Hosebird client is broken down into two modules: hbc-core and hbc-twitter4j. The hbc-core module uses a message queue, which the consumer can poll for the raw String messages, while the hbc-twitter4j module uses the [twitter4j](http://twitter4j.org) listeners and data model on top of the message queue to provide a parsing layer.
The latest hbc artifacts are published to maven central. Bringing hbc into your project should be as simple as adding the following to your maven pom.xml file:
```xml
<dependencies>
<dependency>
<groupId>com.twitter</groupId>
<artifactId>hbc-core</artifactId> <!-- or hbc-twitter4j -->
<version>2.2.0</version> <!-- or whatever the latest version is -->
</dependency>
</dependencies>
```
### Quickstart
Declaring the connection information:
```java
/** Set up your blocking queues: Be sure to size these properly based on expected TPS of your stream */
BlockingQueue<String> msgQueue = new LinkedBlockingQueue<String>(100000);
BlockingQueue<Event> eventQueue = new LinkedBlockingQueue<Event>(1000);
/** Declare the host you want to connect to, the endpoint, and authentication (basic auth or oauth) */
Hosts hosebirdHosts = new HttpHosts(Constants.STREAM_HOST);
StatusesFilterEndpoint hosebirdEndpoint = new StatusesFilterEndpoint();
// Optional: set up some followings and track terms
List<Long> followings = Lists.newArrayList(1234L, 566788L);
List<String> terms = Lists.newArrayList("twitter", "api");
hosebirdEndpoint.followings(followings);
hosebirdEndpoint.trackTerms(terms);
// These secrets should be read from a config file
Authentication hosebirdAuth = new OAuth1("consumerKey", "consumerSecret", "token", "secret");
```
Creating a client:
```java
ClientBuilder builder = new ClientBuilder()
.name("Hosebird-Client-01") // optional: mainly for the logs
.hosts(hosebirdHosts)
.authentication(hosebirdAuth)
.endpoint(hosebirdEndpoint)
.processor(new StringDelimitedProcessor(msgQueue))
.eventMessageQueue(eventQueue); // optional: use this if you want to process client events
Client hosebirdClient = builder.build();
// Attempts to establish a connection.
hosebirdClient.connect();
```
Now, msgQueue and eventQueue will now start being filled with messages/events. Read from these queues however you like.
```java
// on a different thread, or multiple different threads....
while (!hosebirdClient.isDone()) {
String msg = msgQueue.take();
something(msg);
profit();
}
```
You can close a connection with
```java
hosebirdClient.shutdown();
```
### Quick Start Example
To run the sample stream example:
```
mvn install && mvn exec:java -pl hbc-example -Dconsumer.key=XYZ -Dconsumer.secret=SECRET -Daccess.token=ABC -Daccess.token.secret=ABCSECRET
```
You can find these values on http://dev.twitter.com and navigating to one of your applications then to the API Keys tab.
The API key and secrets values on that page correspond to hbc's `-Dconsumer.*` properties.
Alternatively you can set those properties in hbc-examples/pom.xml
## The Details
### Authentication:
Declaring OAuth1 credentials in the client (preferred):
```java
new OAuth1("consumerKey", "consumerSecret", "token", "tokenSecret")
```
Declaring basic auth credentials in the client:
```java
new BasicAuth("username", "password")
```
Be sure not to pass your tokens/passwords as strings directly into the initializers. They should be read from a configuration file that isn't checked in with your code or something similar. Safety first.
### Specifying an endpoint
Declare a StreamingEndpoint to connect to. These classes reside in the package com.twitter.hbc.core.endpoint, and correspond to all of our endpoints. By default, the HTTP parameter "delimited=length" is set for all of our StreamingEndpoints for compatibility with our processor (next section). If you are using our StringDelimitedProcessor this parameter must be set. For a list of available public endpoints and the http parameters we support, see [Twitter's Streaming API docs](https://dev.twitter.com/docs/streaming-apis/streams/public).
#### Filter streams:
```java
StatusesFilterEndpoint endpoint = new StatusesFilterEndpoint();
// Optional: set up some followings and track terms
List<Long> followings = Lists.newArrayList(1234L, 566788L);
List<String> terms = Lists.newArrayList("twitter", "api");
endpoint.followings(followings);
endpoint.trackTerms(terms);
```
#### Firehose streams:
```java
StreamingEndpoint endpoint = new StatusesFirehoseEndpoint();
// Optional: set up the partitions you want to connect to
List<Integer> partitions = Lists.newArrayList(0,1,2,3);
endpoint.partitions(partitions);
// By default, delimited=length is already set for use by our StringDelimitedProcessor
// Do this to unset it (Be sure you really want to do this)
// endpoint.delimited(false);
```
#### Setting up a Processor:
The hosebird client uses the notion of a "processor" which processes the stream and put individual messages into the provided BlockingQueue. We provide a StringDelimitedProcessor class which should be used in conjunction with the StreamingEndpoints provided. The processor takes as its parameter a BlockingQueue, which the client will put String messages into as it streams them.
Setting up a StringDelimitedProcessor is as easy as:
```java
new StringDelimitedProcessor(msgQueue);
```
### Control streams for Sitestream connections
Hosebird provides [control stream support for sitestreams](https://dev.twitter.com/docs/streaming-apis/streams/site/control).
To make control stream calls with the hosebird client, first create a client. When calling connect() to create a connection to a stream with control stream support, the first message you receive will be the streamId. You'll want to hold on to that when processing the messages if you plan on using control streams, so after calling connect(), be sure to keep track of the streamId of this connection. Note that due to reconnections, the streamId could change, so always use the latest one. If you're using our twitter4j layer, keeping track of the control messages/streamIds will be taken care of for you.
```java
SitestreamController controlStreams = client.getSitestreamController();
// When making a connection to the stream with control stream support one of the response messages will include the streamId.
// You'll want to hold on to that when processing the messages if you plan on using control streams
// add userId to our control stream
controlStreams.addUser(streamId, userId);
// remove userId to our control stream
controlStreams.removeUser(streamId, userId);
```
### The hbc-twitter4j module
The hbc-twitter4j module uses the twitter4j listeners and models. To use it, create a normal Client object like before using the ClientBuilder, then depending on which type of stream you are reading from, create an appropriate Twitter4jClient. The Twitter4jClient wraps around the Client it is passed, and calls the callback methods in the twitter4j listeners whenever it retrieves a message from the message queue. The actual work of polling from the message queue, parsing, and executing the callback method is done by forking threads from an executor service that the client is passed.
If connecting to a status stream (filter, firehose, sample), use Twitter4jStatusClient:
```java
// client is our Client object
// msgQueue is our BlockingQueue<String> of messages that the handlers will receive from
// listeners is a List<StatusListener> of the t4j StatusListeners
// executorService
Twitter4jClient t4jClient = new Twitter4jStatusClient(client, msgQueue, listeners, executorService);
t4jClient.connect();
// Call this once for every thread you want to spin off for processing the raw messages.
// This should be called at least once.
t4jClient.process(); // required to start processing the messages
t4jClient.process(); // optional: another Runnable is submitted to the executorService to process the msgQueue
t4jClient.process(); // optional
```
If connecting to a userstream, use Twitter4jUserstreamClient. If making a sitestream connection, use Twitter4jSitestreamClient.
#### Using Handlers, a Twitter4j listener add-on
All Twitter4jClients support Handlers, which extend their respective Twitter4j listeners: StatusStreamHandler extends StatusesListener, UserstreamHandler extends UserstreamListener, SitestreamHandler extends SitestreamHandler. These handlers have extra callback menthods that may be helpful for parsing messages that the Twitter4j listeners do not yet support
```java
UserstreamListener listener = new UserstreamHandler() {
/**
* <UserstreamListener methods here>
*/
#Override
public void onDisconnectMessage(DisconnectMessage disconnectMessage) {
// this method is called when a disconnect message is received
}
#Override
public void onUnfollow(User source, User target) {
// do something
}
#Override
public void onRetweet(User source, User target, Status retweetedStatus) {
// do your thing
}
#Override
public void onUnknownMessageType(String msg) {
// msg is any message that isn't handled by any of our other callbacks
}
}
listeners.append(listener);
Twitter4jClient t4jClient = new Twitter4jUserstreamClient(client, msgQueue, listeners, executorService);
```
## Building / Testing
To build locally (you must use java 1.7 for compiling, though we produce 1.6 compatible classes):
```
mvn compile
```
To run tests:
```
mvn test
```
You can install the Eclipse Maven plugin then compile from Eclipse, or you can do something simple and straightforward like me. Start a command prompt then from the folder of unzipped hbc-master, e.g. c:\downloads\hbc-master, run the following command:
c:\downloads\hbc-master>c:\downloads\apache-maven-3.2.3-bin\apache-maven
-3.2.3\bin\mvn compile
This will compile all the HBC classes. Then you can put all the generated class files including hbc-master\hbc-core\target\classes\ and hbc-master\hbc-twitter4j\target\classes\ into one folder then use the Jar command to zip them together as one Jar file that you can add to your Eclipse project, e.g.
c:\downloads\hbc-master\mybuild>jar cvf Twitter-api.jar com twitter4j
This will provide all the classes that allow you to run the simple examples like example\FilterStreamExample. However, for more complex examples like EnterpriseStreamExample and Twitter4jSampleStreamExample, you will need more to grab more libraries including twitter4j-core, twitter4j-stream, and guava.jar (Google collections).
Hope this helps.