Spring boot rabbitmq message not getting requeued - spring

Hi I am trying to requeue certain messages if a specific exception is thrown, but for any validation failures I want them to go straight to the dead letter queue. I have the the relevant queues and dead letter queues enabled. I am finding hat my validation failures are got to the dlq, but the other failures are constantly in an unack state and getting constantly retried, beyond the max-attempts and multiplier I had set up, any ideas why this is? code below I am using Spring boot 2.0.4 release
#RabbitListener(queues = "${queuename}")
public void consume(final #Valid #Payload MyRequest myRequest) {
if (method.fail()) {
throw new RuntimeException("");
}
}
#Bean
public DefaultMessageHandlerMethodFactory myHandlerMethodFactory() {
DefaultMessageHandlerMethodFactory factory = new DefaultMessageHandlerMethodFactory();
factory.setMessageConverter(jackson2Converter());
factory.setValidator(amqpValidator());
return factory;
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
registrar.setMessageHandlerMethodFactory(myHandlerMethodFactory());
}
#Bean
public Validator amqpValidator() {
return new OptionalValidatorFactoryBean();
}
#Bean
public Jackson2JsonMessageConverter jackson2JsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory listenerContainerFactory =
new SimpleRabbitListenerContainerFactory();
listenerContainerFactory.setConnectionFactory(connectionFactory());
listenerContainerFactory.setErrorHandler(new ConditionalRejectingErrorHandler(
new MyErrorPayload()));
listenerContainerFactory.setMessageConverter(jackson2JsonMessageConverter());
return listenerContainerFactory;
}
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory(rabbitQueueHost);
connectionFactory.setUsername(rabbitQueueUsername);
connectionFactory.setPassword(rabbitQueuePassword);
connectionFactory.setVirtualHost(rabbitQueueVirtualHost);
return connectionFactory;
}
public class MyErrorPayload implements FatalExceptionStrategy {
#Override
public boolean isFatal(Throwable t) {
if (t instanceof ListenerExecutionFailedException &&
(t.getCause() instanceof MessageConversionException ||
t.getCause() instanceof MethodArgumentNotValidException )
) {
return true;
}
return false;
}
}
application.yml ( properties)
spring:
rabbitmq:
host: localhost
username: uu
password: pp
virtual-host: /
listener:
simple:
default-requeue-rejected: false
retry:
enabled: true
initial-interval: 2000
multiplier: 1.5
max-interval: 10000
max-attempts: 3

It's because you are not using Boot's auto configuration for the container factory. So the retry configuration is ignored.
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory listenerContainerFactory =
new SimpleRabbitListenerContainerFactory();
listenerContainerFactory.setConnectionFactory(connectionFactory());
listenerContainerFactory.setErrorHandler(new ConditionalRejectingErrorHandler(
new MyErrorPayload()));
listenerContainerFactory.setMessageConverter(jackson2JsonMessageConverter());
return listenerContainerFactory;
}
The same was true for the sample that #Barath references in his comment.
Inject the configurer into your factory method and invoke it; for example, for that sample...
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory,
SimpleRabbitListenerContainerFactoryConfigurer configurer) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setErrorHandler(errorHandler());
return factory;
}
If there is only one message converter Bean, the configurer will add that too.
I have updated the sample.
EDIT
Custom retry policy for selective exceptions; the following disables retry for ValidationException but retries all others. (Again, for the sample app)...
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory,
SimpleRabbitListenerContainerFactoryConfigurer configurer, RabbitProperties properties) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setErrorHandler(errorHandler());
ListenerRetry retryConfig = properties.getListener().getSimple().getRetry();
if (retryConfig.isEnabled()) {
RetryInterceptorBuilder<?> builder = (retryConfig.isStateless()
? RetryInterceptorBuilder.stateless()
: RetryInterceptorBuilder.stateful());
RetryTemplate retryTemplate = new RetryTemplate();
Map<Class<? extends Throwable>, Boolean> retryableExceptions = Collections
.singletonMap(ValidationException.class, false);
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy(retryConfig.getMaxAttempts(),
retryableExceptions, true, true); // retry all exceptions except Validation
retryTemplate.setRetryPolicy(retryPolicy);
ExponentialBackOffPolicy backOffPolicy = new ExponentialBackOffPolicy();
backOffPolicy.setInitialInterval(retryConfig.getInitialInterval().toMillis());
backOffPolicy.setMaxInterval(retryConfig.getMaxInterval().toMillis());
backOffPolicy.setMultiplier(retryConfig.getMultiplier());
retryTemplate.setBackOffPolicy(backOffPolicy);
builder.retryOperations(retryTemplate);
builder.recoverer(new RejectAndDontRequeueRecoverer());
factory.setAdviceChain(builder.build());
}
return factory;
}
No messages are ever requeued since you have default-requeue-rejected: false.

Related

How to prevent a specific exception from being retried in RabbitMQ

In my Spring Boot project I'm using RabbitMQ for messaging and have two exceptions CustomExceptionA and CustomExceptionB. I want my CustomExceptionA to be retried n times and CustomExceptionB not to retry but directly sent to DLQ.
Below are my configurations:-
yaml file
spring:
rabbitmq:
listener:
simple:
default-requeue-rejected: false
retry:
enabled: true
initial-interval: 2s
max-attempts: 3
max-interval: 2s
multiplier: 1
Configuration file
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory,
SimpleRabbitListenerContainerFactoryConfigurer configurer) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setErrorHandler(errorHandler());
return factory;
}
#Bean
public ErrorHandler errorHandler() {
return new ConditionalRejectingErrorHandler(customExceptionStrategy());
}
#Bean
FatalExceptionStrategy customExceptionStrategy() {
return new CustomFatalExceptionStrategy();
}
-----------------------------------------------------------------------------------------------------------------------------
#Component
public class CustomFatalExceptionStrategy extends ConditionalRejectingErrorHandler.DefaultExceptionStrategy {
#Override
public boolean isFatal(Throwable throwable) {
return (throwable.getCause() instanceof CustomExceptionB);
}
}
According to the blog : https://www.baeldung.com/spring-amqp-error-handling mechanism should work but for some reason it is not working for me.
Someone please look at the issue.

Spring Boot RabbitMQ ConnectException:

I have an application that should work with rabbitmq. I have RabbitMQConfig, which tries to connect with the rabbit, but if it fails to connect, the application does not start at all.
What I want to achieve is that after launching the application, it will try to connect to the rabbit and if it manages to connect, I will start functionality to create a queue and listen to it accordingly. Currently at startup if the rabbit is available and then disappears it starts throwing "java.net.ConnectException: Connection refused". Can I catch this error and how, both at startup and during the operation of the application.
This is config file:
public class RabbitMQConfig implements RabbitListenerConfigurer {
private ConnectionFactory connectionFactory;
#Autowired
public RabbitMQConfig(ConnectionFactory connectionFactory) {
this.connectionFactory = connectionFactory;
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
registrar.setMessageHandlerMethodFactory(messageHandlerMethodFactory());
}
#Bean
MessageHandlerMethodFactory messageHandlerMethodFactory() {
DefaultMessageHandlerMethodFactory messageHandlerMethodFactory = new DefaultMessageHandlerMethodFactory();
messageHandlerMethodFactory.setMessageConverter(consumerJackson2MessageConverter());
return messageHandlerMethodFactory;
}
#Bean
public MappingJackson2MessageConverter consumerJackson2MessageConverter() {
return new MappingJackson2MessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(new Jackson2JsonMessageConverter());
return rabbitTemplate;
}
#Bean
public AmqpAdmin amqpAdmin() {
return new RabbitAdmin(connectionFactory);
}
#Bean(name = "rabbitListenerContainerFactory")
public SimpleRabbitListenerContainerFactory simpleRabbitListenerContainerFactory(
SimpleRabbitListenerContainerFactoryConfigurer configurer,
ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
FixedBackOff recoveryBackOff = new FixedBackOff(10000,FixedBackOff.UNLIMITED_ATTEMPTS);
factory.setRecoveryBackOff(recoveryBackOff);
configurer.configure(factory, connectionFactory);
return factory;
}
Whit this methods i create and start listener:
#Service
public class RabbitMQService {
#RabbitListener(queues = "${queueName}", autoStartup = "false", id = "commandQueue")
public void receive(CommandDataDTO commandDataDTO) {
public void receive(Object rMessage) {
doSomething(rMessage);
}
public void createQueue() {
Queue queue = new Queue("queueName"), true, false, false);
Binding binding = new Binding("queueName"),
Binding.DestinationType.QUEUE, env.getProperty("spring.rabbitmq.exchange"),"rKey",
null);
admin.declareQueue(queue);
admin.declareBinding(binding);
startListener();
}
//When queue is active we start Listener
public void startListener() {
boolean isQueuqReady = false;
while (!isQueuqReady) {
Properties p = admin.getQueueProperties(env.getProperty("management.registry.info.device-type") + "_"
+ env.getProperty("management.registry.info.device-specific-type"));
if (p != null) {
log.info("Rabbit queue is up. Start listener.");
isQueuqReady = true;
registry.getListenerContainer("commandQueue").start();
}
}
}
Problem is that I want, regardless of whether there is a connection or not with the rabbit, the application to be able to work and, accordingly, to intercept when there is a connection and when not to do different actions.

Redelivery Policy Is Not Working in Activemq Spring Boot

I have used below configuration
#SpringBootApplication
#EnableScheduling
public class NotificationApplication {
#Value("${jms.broker.endpoint}")
private String brokerUrl;
#Autowired
private Environment env;
public static void main(String[] args) {
ApplicationContext ctx = SpringApplication.run(NotificationApplication.class, args);
RecordReader abc = ctx.getBean(RecordReader.class);
abc.readNotifications();
}
#Bean
public ActiveMQConnectionFactory activeMQConnectionFactory() {
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(brokerUrl);
RedeliveryPolicy redeliveryPolicy = new RedeliveryPolicy();
redeliveryPolicy.setMaximumRedeliveries(env.getProperty("jms.redelivery.maximum", Integer.class));
factory.setRedeliveryPolicy(redeliveryPolicy);
factory.setTrustedPackages(Arrays.asList("com.lms.notification"));
return factory;
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() throws Throwable {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(activeMQConnectionFactory());
factory.setMessageConverter(jacksonJmsMessageConverter());
factory.setConcurrency(env.getProperty("jms.connections.concurrent"));
return factory;
}
#Bean
public JmsTemplate jmsTemplate() {
JmsTemplate template = new JmsTemplate();
template.setConnectionFactory(activeMQConnectionFactory());
template.setMessageConverter(jacksonJmsMessageConverter());
return template;
}
#Bean
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
}
The issue is the Redelivery policy is not working as I have defined in activeMQConnectionFactory bean. Means I have set maximum redelivery 1, but is not being redelivered in case of exception in listener. Also in case of exception in listener it should go to the default DLQ, which is also not happening. But if I comment the jmsListenerContainerFactory bean all works fine.
I am not able to identify why this is happening. Can any one look into this what wrong I am doing?
I am using Activemq 5.16.1
Thanks

How to configure RabbitMQ (in Spring Boot 2.x) so manual acknowledge works

So I read all the examples, I have the com.rabbitmq.client.Channel and the #Header(AmqpHeaders.DELIVERY_TAG), but when I try to call Channel.basicNack(long deliveryTag, boolean multiple, boolean requeue) the result is "java.lang.IllegalStateException: Channel closed; cannot ack/nack". I can see that the CachingConnectionFactory does not support any of the acknowledge methods. So my question is what ConnectionFactory do I have to use and howto configure it, so that basicAck/basicNack works?
Spring Boot Version 2.1.0.RELEASE
application.yaml:
spring:
rabbitmq:
host: ${RABBITMQ_HOST:localhost}
port: ${RABBITMQ_PORT:5672}
username: ${RABBITMQ_USERNAME:guest}
password: ${RABBITMQ_PASSWORD:guest}
listener:
type: simple
simple:
acknowledge-mode: manual
Config class:
#EnableRabbit
#Configuration
public class RabbitMqConfig implements RabbitListenerConfigurer {
#Value("${app.rabbitmq.incoming-queue}")
private String incomingQueue;
private AmqpAdmin amqpAdmin;
#Autowired
public RabbitMqConfig(AmqpAdmin amqpAdmin) {
this.amqpAdmin = amqpAdmin;
}
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setChannelTransacted(true);
rabbitTemplate.setMessageConverter(jsonMessageConverter());
return rabbitTemplate;
}
#Bean
MessageConverter jsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registar) {
registar.setMessageHandlerMethodFactory(createDefaultMessageHandlerMethodFactory());
}
#Bean
public DefaultMessageHandlerMethodFactory createDefaultMessageHandlerMethodFactory() {
DefaultMessageHandlerMethodFactory factory = new DefaultMessageHandlerMethodFactory();
factory.setMessageConverter(new MappingJackson2MessageConverter());
return factory;
}
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(
ConnectionFactory connectionFactory, CommonAmqpErrorHandler commonAmqpErrorHandler) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMessageConverter(jsonMessageConverter());
factory.setErrorHandler(commonAmqpErrorHandler);
return factory;
}
#PostConstruct
public void afterInit() {
amqpAdmin.declareQueue(new Queue(getDeadLetterQueueName(incomingQueue), true));
amqpAdmin.declareQueue(
QueueBuilder.durable(incomingQueue).withArgument("x-dead-letter-exchange", "")
.withArgument("x-dead-letter-routing-key",
getDeadLetterQueueName(incomingQueue)).build());
}
private String getDeadLetterQueueName(String queueName) {
return queueName + ".dead-letter.queue";
}
}
Listener code:
#Transactional(rollbackOn = Exception.class)
#RabbitListener(queues = "${app.rabbitmq.incoming-queue}", errorHandler = "notificationListenerErrorHandler")
public void onMessage(#Valid #Payload NotificationDto notification, Message message,
Channel channel, #Header(AmqpHeaders.DELIVERY_TAG) long tag) throws IOException {
System.out.println("00000 > " + tag);
System.out.println("11111");
channel.basicNack(tag, false, true);
System.out.println("222222");
}
After starting it from scratch, it turns out that the
#Transactional(rollbackOn = Exception.class)
is causing the problem. If I remove it, it's working

Spring Boot RabbitMQ Publisher and Receiver On the Same Project

I have an application (Microservice like) which should send and receives messages from other applications (Microservices). The application has several publishers with every publisher publishing to a specific queue as well as several subscriber classes with each subscriber subscribing to only one queue. Unfortunately, my subscriber classes are consuming the same messages I publish. How should I go about it?
Here is my code:
a) Publisher 1 - does not have a listener method since it only publishes to my.queues.queue1
#Configuration
public class RabbitQueue1Publisher{
private static final String QUEUE_NAME = "my.queues.queue1";
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory("http://127.0.0.1:1675");
connectionFactory.setUsername("guest");
connectionFactory.setPassword("guest");
return connectionFactory;
}
#Bean
public Queue simpleQueue() {
return new Queue(QUEUE_NAME);
}
#Bean
public MessageConverter jsonMessageConverter(){
return new Jackson2JsonMessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
template.setRoutingKey(QUEUE_NAME);
template.setMessageConverter(jsonMessageConverter());
return template;
}
}
b) Publisher 2 - also does not have a listener method since it only publishes to my.queues.queue2
#Configuration
public class RabbitQueue2Publisher{
private static final String QUEUE_NAME = "my.queues.queue2";
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory("http://127.0.0.1:1675");
connectionFactory.setUsername("guest");
connectionFactory.setPassword("guest");
return connectionFactory;
}
#Bean
public Queue simpleQueue() {
return new Queue(QUEUE_NAME);
}
#Bean
public MessageConverter jsonMessageConverter(){
return new Jackson2JsonMessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
template.setRoutingKey(QUEUE_NAME);
template.setMessageConverter(jsonMessageConverter());
return template;
}
}
c) Consumer 1 - consumes from queue3. Has a listener method
#Configuration
public class RabbitQueue3Subscriber{
private static final String QUEUE_NAME = "my.queue.queue3";
#Autowired
private Queue3Listener Queue3Listener;
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory("http://127.0.0.1:15672");
connectionFactory.setUsername("guest");
connectionFactory.setPassword("guest");
return connectionFactory;
}
#Bean
public Queue simpleQueue() {
return new Queue(QUEUE_NAME);
}
#Bean
public MessageConverter jsonMessageConverter(){
return new Jackson2JsonMessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
template.setRoutingKey(QUEUE_NAME);
template.setMessageConverter(jsonMessageConverter());
return template;
}
#Bean
public SimpleMessageListenerContainer userListenerContainer() {
SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer();
listenerContainer.setConnectionFactory(connectionFactory());
listenerContainer.setQueues(simpleQueue());
listenerContainer.setMessageConverter(jsonMessageConverter());
listenerContainer.setMessageListener(Queue3Listener);
listenerContainer.setAcknowledgeMode(AcknowledgeMode.AUTO);
return listenerContainer;
}
}
d) Consumer 2 - consumes from queue4. Has a listener method
#Configuration
public class RabbitQueue4Subscriber{
private static final String QUEUE_NAME = "my.queue.queue4";
#Autowired
private Queue4Listener Queue4Listener;
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory("http://127.0.0.1:15672");
connectionFactory.setUsername("guest");
connectionFactory.setPassword("guest");
return connectionFactory;
}
#Bean
public Queue simpleQueue() {
return new Queue(QUEUE_NAME);
}
#Bean
public MessageConverter jsonMessageConverter(){
return new Jackson2JsonMessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
template.setRoutingKey(QUEUE_NAME);
template.setMessageConverter(jsonMessageConverter());
return template;
}
#Bean
public SimpleMessageListenerContainer userListenerContainer() {
SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer();
listenerContainer.setConnectionFactory(connectionFactory());
listenerContainer.setQueues(simpleQueue());
listenerContainer.setMessageConverter(jsonMessageConverter());
listenerContainer.setMessageListener(Queue4Listener);
listenerContainer.setAcknowledgeMode(AcknowledgeMode.AUTO);
return listenerContainer;
}
}
Though I am publishing and consuming to/from different queues, I end up consuming the same messages I produce. Can someone point out what I am doing wrong or suggest the way to do it?
Here is how it works for me. I have publisher and a consumer of Rabbitmq. Doesn't mater if they are part of the same project or different.
Publisher:
Publisher Configuration
#Configuration
class PublisherConfig{
String queueName = "com.queueName";
String routingKey = "com.routingKey";
String exchange = "com.exchangeName";
#Bean
Queue queue() {
return new Queue(queueName, false);
}
#Bean
TopicExchange exchange() {
return new TopicExchange(exchange);
}
#Bean
Binding binding(Queue queueFoo, TopicExchange exchange) {
return BindingBuilder.bind(queueFoo).to(exchange).with(routingKey);
}
//Required only if you want to pass custom object as part of payload
#Bean
public MappingJackson2MessageConverter jackson2Converter() {
return new MappingJackson2MessageConverter();
}
}
Publish Message
#Autowired private RabbitMessagingTemplate rabbitMessagingTemplate;
#Autowired private MappingJackson2MessageConverter mappingJackson2MessageConverter;
rabbitMessagingTemplate.setMessageConverter(this.mappingJackson2MessageConverter);
rabbitMessagingTemplate.convertAndSend(exchange, routingKey, employObj)
Consumer
Consumer Configuration
#Configuration
public class RabbitMQConfiguration implements RabbitListenerConfigurer {
public MappingJackson2MessageConverter jackson2Converter() {
return new MappingJackson2MessageConverter();
}
#Bean
public DefaultMessageHandlerMethodFactory handlerMethodFactory() {
DefaultMessageHandlerMethodFactory factory = new DefaultMessageHandlerMethodFactory();
factory.setMessageConverter(jackson2Converter());
return factory;
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
registrar.setMessageHandlerMethodFactory(handlerMethodFactory());
}
}
Listen a Message
#RabbitListener(queues = "com.queueName")
public void receiveMessage(Employee employee) {
// More code
}
You can encapsulate Publisher and Listener configurations in two different #configuration files.
Hope this helps you
P.S.
OP asked for explanation. Here it is:
Exchange and Routing Key
Publisher publishes a message to an exchange with a particular routing key. Routing key helps to differentiate the type of message it is.
Suppose:
Send all user logged in messages with routing key of 'user_logged_in'.
Send all email sent messages with 'email_sent'.
Queue:
Once the routing key is attached with the exchange there comes a queue.
Queue is attached a exchange and routing key and all the published messages will sit in this queue.
Now consumer explicitly, connects to such queues and listen messages.
So queue name in publisher config and consumer config has to be the same.
Once your publisher is up you can actually visit RabbitMq dashboard
and see the exchange, routing key and queue to see how it works.

Resources