Spring STOMP Incomplete Frame - spring

I have created a websocket using STOMP inside Spring. The endpoint works like a charm when used with javascript libraries however when I use any of the simple websocket google chrome extensions (i.e. Simple WebSocket Client, Smart Websocket Client, Web Socket Client), spring throws the "Incomplete STOMP frame content message. Diving into the code, I've been able to see that the cause of this is I cannot insert the null character /u0000 with any of these tools. I assume all the java script frameworks do this by default. Has someone found a workaround for this so that I can use any websocket client with Spring STOMP?
The stomp code is located here: https://github.com/spring-projects/spring-framework/blob/master/spring-messaging/src/main/java/org/springframework/messaging/simp/stomp/StompDecoder.java
On [currently] lines 308-320 the below code exists. This method returns null because byteBuffer.remaining is not greater than content length (both are 0) .There is a StompSubProtocolHandler exception that fires afterrwards. I tried looking into all the handlers and interceptors but there doesn't seem to be a way to intercept things at this level without rewriting almost everything. I wanted to just inject "\0" into the payload...
if (contentLength != null && contentLength >= 0) {
if (byteBuffer.remaining() > contentLength) {
byte[] payload = new byte[contentLength];
byteBuffer.get(payload);
if (byteBuffer.get() != 0) {
throw new StompConversionException("Frame must be terminated with a null octet");
}
return payload;
}
else {
return null;
}
}

I had exactly the same problem, I tested with Web socket client.
In order to be able to test STOMP manually on my local environment, I have configured my Spring context. That way I don't need to add the null character on the client side. It's automatically added if it does not exist.
For that, in the class AbstractWebSocketMessageBrokerConfigurer I added:
#Override
public void configureWebSocketTransport(WebSocketTransportRegistration registration) {
registration.addDecoratorFactory(new WebSocketHandlerDecoratorFactory() {
#Override
public WebSocketHandler decorate(WebSocketHandler webSocketHandler) {
return new EmaWebSocketHandlerDecorator(webSocketHandler);
}
});
}
The decorator add automatically carriage returns when there is no request body (ex: connect command).
/**
* Extension of the {#link WebSocketHandlerDecorator websocket handler decorator} that allows to manually test the
* STOMP protocol.
*
* #author Sebastien Gerard
*/
public class EmaWebSocketHandlerDecorator extends WebSocketHandlerDecorator {
private static final Logger logger = LoggerFactory.getLogger(EmaWebSocketHandlerDecorator.class);
public EmaWebSocketHandlerDecorator(WebSocketHandler webSocketHandler) {
super(webSocketHandler);
}
#Override
public void handleMessage(WebSocketSession session, WebSocketMessage<?> message) throws Exception {
super.handleMessage(session, updateBodyIfNeeded(message));
}
/**
* Updates the content of the specified message. The message is updated only if it is
* a {#link TextMessage text message} and if does not contain the <tt>null</tt> character at the end. If
* carriage returns are missing (when the command does not need a body) there are also added.
*/
private WebSocketMessage<?> updateBodyIfNeeded(WebSocketMessage<?> message) {
if (!(message instanceof TextMessage) || ((TextMessage) message).getPayload().endsWith("\u0000")) {
return message;
}
String payload = ((TextMessage) message).getPayload();
final Optional<StompCommand> stompCommand = getStompCommand(payload);
if (!stompCommand.isPresent()) {
return message;
}
if (!stompCommand.get().isBodyAllowed() && !payload.endsWith("\n\n")) {
if (payload.endsWith("\n")) {
payload += "\n";
} else {
payload += "\n\n";
}
}
payload += "\u0000";
return new TextMessage(payload);
}
/**
* Returns the {#link StompCommand STOMP command} associated to the specified payload.
*/
private Optional<StompCommand> getStompCommand(String payload) {
final int firstCarriageReturn = payload.indexOf('\n');
if (firstCarriageReturn < 0) {
return Optional.empty();
}
try {
return Optional.of(
StompCommand.valueOf(payload.substring(0, firstCarriageReturn))
);
} catch (IllegalArgumentException e) {
logger.trace("Error while parsing STOMP command.", e);
return Optional.empty();
}
}
}
Now I can do requests like:
CONNECT
accept-version:1.2
host:localhost
content-length:0
SEND
destination:/queue/com.X.notification-subscription
content-type:text/plain
reply-to:/temp-queue/notification
hello world :)
Hope this helps.
S.

Related

I want to connect the Spring application with the external bukkit server through the REST API method

I want to control the bukkit server through the spring web application.
For example, send a command to the console, receive his response, etc
I'm trying to figure out a way, but I can't find a good one.
How shall I do it?
Even if third-party plugins are imported through the database, I want to find a way to do basic bukkit control.
First, you need to decide how to send the request to the server. It seems to me that in your case, the easiest is run the built-in java web server (HttpServer) to receive commands, and then process them.
If you need synchronous actions, then you can always do callSyncMethod
To receive command output, simply create your own implementation of CommandSender with overridden sendMessage methods
For example, how do command execution endpoint
JavaPlugin plugin = /** get plugin **/;
HttpServer server = HttpServer.create(new InetSocketAddress("localhost", 8001), 0);
server.createContext("/executeCommand", exchange -> {
if (!exchange.getRequestMethod().equals("POST")) {
exchange.getResponseBody().write("Method not supported".getBytes(StandardCharsets.UTF_8));
return;
}
// In this example body is command
String body = new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8);
StringBuilder builder = new StringBuilder();
// You also need override many another methods to compile code,but just leave it empty
CommandSender sender = new CommandSender() {
#Override
public void sendMessage(#NotNull String message) {
builder.append(message);
}
#Override
public void sendMessage(#NotNull String... messages) {
for (String message : messages) {
builder.append(message + "\n");
}
}
#Override
public boolean isOp() {
return true;
}
#Override
public boolean hasPermission(#NotNull String name) {
return true;
}
#Override
public #NotNull String getName() {
return "WebServerExecutor";
}
};
// Waiting command execute finish
Bukkit.getScheduler().callSyncMethod(plugin, () -> Bukkit.dispatchCommand(sender, body)).get();
byte[] response = builder.toString().getBytes(StandardCharsets.UTF_8);
exchange.getResponseBody().write(response);
});
server.start()

creating Opentelemetry Context using trace-id and span-id of remote parent

I have micro service which support open tracing and that injecting trace-id and span-id in to header. Other micro service support open telemetry. how can I create parent span using trace-id and span-id in second micro service?
Thanks,
You can use W3C Trace Context specifications to achieve this. We need to send traceparent(Ex: 00-8652a752089f33e2659dff28d683a18f-7359b90f4355cfd9-01) from producer via HTTP headres ( or you can create it using the trace-id and span-id in the consumer). Then we can extract the remote context and create the span with traceparent.
This is the consumer controller. TextMapGetter used to map that traceparent data to the Context. ExtractModel is just a custom class.
#GetMapping(value = "/second")
public String sencondTest(#RequestHeader(value = "traceparent") String traceparent){
try {
Tracer tracer = openTelemetry.getTracer("cloud.events.second");
TextMapGetter<ExtractModel> getter = new TextMapGetter<>() {
#Override
public String get(ExtractModel carrier, String key) {
if (carrier.getHeaders().containsKey(key)) {
return carrier.getHeaders().get(key);
}
return null;
}
#Override
public Iterable<String> keys(ExtractModel carrier) {
return carrier.getHeaders().keySet();
}
};
ExtractModel model = new ExtractModel();
model.addHeader("traceparent", traceparent);
Context extractedContext = openTelemetry.getPropagators().getTextMapPropagator()
.extract(Context.current(), model, getter);
try (Scope scope = extractedContext.makeCurrent()) {
// Automatically use the extracted SpanContext as parent.
Span serverSpan = tracer.spanBuilder("CloudEvents Server")
.setSpanKind(SpanKind.SERVER)
.startSpan();
try {
Thread.sleep(150);
} finally {
serverSpan.end();
}
}
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
return "Server Received!";
}
Then when we configuring the OpenTelemetrySdk need to set W3CTraceContextPropagator in Context Propagators.
// Use W3C Propagator(to extract span from HTTP headers) since we use the W3C specifications
TextMapPropagator textMapPropagator = W3CTraceContextPropagator.getInstance();
OpenTelemetrySdk openTelemetrySdk = OpenTelemetrySdk.builder()
.setTracerProvider(tracerProvider)
.setPropagators(ContextPropagators.create(textMapPropagator))
.buildAndRegisterGlobal();
Here is my customer ExtractModel class
public class ExtractModel {
private Map<String, String> headers;
public void addHeader(String key, String value) {
if (this.headers == null){
headers = new HashMap<>();
}
headers.put(key, value);
}
public Map<String, String> getHeaders() {
return headers;
}
public void setHeaders(Map<String, String> headers) {
this.headers = headers;
}
}
You can find more details in the official documentation for manual instrumentation.
Generally you have to propogate the span-id and trace-id if it is available in header. Any request you get in your microservice, check if the headers have span-id and trace-id in them. If yes,extract them and use them in your service.
If it is not present then you create a new one and use it in your service and also add it to requests that go out of your microservice.

Is there a way to record response times of feign client

#FeignClient(...)
public interface SomeClient {
#RequestMapping(value = "/someUrl", method = POST, consumes = "application/json")
ResponseEntity<String> createItem(...);
}
Is there a way to find the response times for createItem api call?
We are using spring boot, actuator, prometheus.
We have straight forward as well as a customized way for logging the feign clients request and response (including the response time). We have to inject the feign.Logger.Level bean, that's it.
THE DEFAULT/ STRAIGHT FORWARD WAY
#Bean
Logger.Level feignLoggerLevel() {
return Logger.Level.BASIC;
}
there are BASIC,FULL,HEADERS,NONE(default) logging levels are available for more details
The above bean injection will give you the logging of feign request and response in the below format:
REQUEST:
refer
log(configKey, "---> %s %s HTTP/1.1", request.httpMethod().name(), request.url());
ex:2019-09-26 12:50:12.163 [DEBUG] [http-nio-4200-exec-5] [com.sample.FeignClient:72] [FeignClient#getUser] ---> END HTTP (0-byte body)
where the configkey means FeignClientClassName#FeignClientCallingMethodName ex: ApiClient#apiMethod.
RESPONSE
refer
log(configKey, "<--- HTTP/1.1 %s%s (%sms)", status, reason, elapsedTime);
ex:2019-09-26 12:50:12.163 [DEBUG] [http-nio-4200-exec-5] [com.sample.FeignClient:72] [FeignClient#getUser] <--- HTTP/1.1 200 OK (341ms)
the elapsedTime is what the response time taken for the API call.
NOTE: If you prefer the default way of the feign client logging then we have to consider the underlying application logging level as well because the feign.Slf4jLogger class logging with the feign request and response details with the DEBUG level (refer). If the underlying logging level above DEBUG then you may need to specify the explicit logger for the feign logging package/class otherwise it will not work.
THE CUSTOMIZED WAY
If you prefer logging with your customized format then you can extend the feign.Logger class and customize your logging. For a typical example if I want to log the header details of request and response in a single line as a list(by default Logger.Level.HEADERS prints the header in multiple lines):
package com.test.logging.feign;
import feign.Logger;
import feign.Request;
import feign.Response;
import lombok.extern.slf4j.Slf4j;
import java.io.IOException;
import static feign.Logger.Level.HEADERS;
#Slf4j
public class customFeignLogger extends Logger {
#Override
protected void logRequest(String configKey, Level logLevel, Request request) {
if (logLevel.ordinal() >= HEADERS.ordinal()) {
super.logRequest(configKey, logLevel, request);
} else {
int bodyLength = 0;
if (request.requestBody().asBytes() != null) {
bodyLength = request.requestBody().asBytes().length;
}
log(configKey, "---> %s %s HTTP/1.1 (%s-byte body) %s", request.httpMethod().name(), request.url(), bodyLength, request.headers());
}
}
#Override
protected Response logAndRebufferResponse(String configKey, Level logLevel, Response response, long elapsedTime)
throws IOException {
if (logLevel.ordinal() >= HEADERS.ordinal()) {
super.logAndRebufferResponse(configKey, logLevel, response, elapsedTime);
} else {
int status = response.status();
Request request = response.request();
log(configKey, "<--- %s %s HTTP/1.1 %s (%sms) %s", request.httpMethod().name(), request.url(), status, elapsedTime, response.headers());
}
return response;
}
#Override
protected void log(String configKey, String format, Object... args) {
log.debug(format(configKey, format, args));
}
protected String format(String configKey, String format, Object... args) {
return String.format(methodTag(configKey) + format, args);
}
}
also we have to inject the customFeignLogger class bean
#Bean
public customFeignLogger customFeignLogging() {
return new customFeignLogger();
}
If you are building FeignClient by yourself then you can build it with the customized logger:
Feign.builder().logger(new customFeignLogger()).logLevel(Level.BASIC).target(SomeFeignClient.class,"http://localhost:8080");
Add the following annotation to your project.
package com.example.annotation
#Target(ElementType.METHOD)
#Retention(RetentionPolicy.RUNTIME)
public #interface DebugTracking {
#Aspect
#Component
public static class DebugTrackingAspect {
#Around("#annotation(com.example.annotation.DebugTracking)")
public Object trackExecutionTime(ProceedingJoinPoint joinPoint) throws Throwable {
StopWatch stopWatch = new StopWatch();
stopWatch.start(joinPoint.toShortString());
Exception exceptionThrown = null;
try {
// Execute the joint point as usual
return joinPoint.proceed();
} catch (Exception ex) {
exceptionThrown = ex;
throw ex;
} finally {
stopWatch.stop();
System.out.println(String.format("%s took %dms.", stopWatch.getLastTaskName(), stopWatch.getLastTaskTimeMillis()));
if (exceptionThrown != null) {
System.out.println(String.format("Exception thrown: %s", exceptionThrown.getMessage()));
exceptionThrown.printStackTrace();
}
}
}
}
}
Then annotate the methods you want to track in your #FeignClient with #DebugTracking.
I'm using the following (with Spring and Lombok) :
#Configuration // from Spring
#Slf4j // from Lombok
public class MyFeignConfiguration {
#Bean // from Spring
public MyFeignClient myFeignClient() {
return Feign.builder()
.logger(new Logger() {
#Override
protected void log(String configKey, String format, Object... args) {
LOG.info( String.format(methodTag(configKey) + format, args)); // LOG is the Lombok Slf4j object
}
})
.logLevel(Logger.Level.BASIC) // see https://cloud.spring.io/spring-cloud-netflix/multi/multi_spring-cloud-feign.html#_feign_logging
.target(MyFeignClient.class,"http://localhost:8080");
}
}
correct way doing this is using custom logger as pointed above. Using #Aspect is wrong. With that you create additional wrapper around the service. Feign already records this metric. Get that metric from feign.

Why spring-cloud-stream not populating JMSMessageID while publishing messages to Solace topics?

Summarization of the problem
In my project, we are trying to use Spring-Cloud-Stream (SCS) to connect to Solace. Eventually we plan to move to Kafka. So using SCS will help us move over to Kafka quite easily without any code changes and very minimal configuration & dependency changes.
We had been using Solace for a while using JMS. Now when we tried to publish messages to Solace using SCS, we observed that in the message, some crucial JMS Headers (JMSMessageID, JMSType, JMSPriority,JMSCorrelationID, JMSExpiration) are blank.
Do we need to configure the JMS headers separately ? If yes, how ?
What I've already tried
I tried to set headers like this, but this is just resulting in duplicate headers with the same name.
#Output(SendReport.TO_NMR)
public void sendMessage(String request) {
log.info("****************** Got this Report Request: " + request);
MessageBuilder<String> builder = MessageBuilder.withPayload(request);
builder.setHeader("JMSType","report-request");
builder.setHeader("JMSMessageId","1");
builder.setHeader("JMSCorrelationId","11");
builder.setHeader("JMSMessageID","4");
builder.setHeader("JMSCorrelationID","114");
builder.setHeader("ApplicationMessageId","111");
builder.setHeader("ApplicationMessageID","112");
builder.setCorrelationId("23434");
Message message = builder.build();
sendReport.output().send(message);
}
JMS Header of the message in Solace looks like this
JMSMessageID
JMSDestination TOPIC_NAME
JMSTimestamp Wed Dec 31 18:00:00 CST 1969
JMSType
JMSReplyTo
JMSCorrelationID
JMSExpiration 0
JMSPriority 0
JMSType nmr-report-request
JMSMessageId 1
JMSMessageID 4
_isJavaSerializedObject-contentType true
_isJavaSerializedObject-id true
solaceSpringCloudStreamBinderVersion 0.1.0
ApplicationMessageId 111
ApplicationMessageID 112
JMSCorrelationId 11
JMSCorrelationID 114
correlationId 23434
id [-84,-19,0,5,115,114,0,14,106,97,118,97,46,117,116,105,108,46,85,85,73,68,-68,-103,3,-9,-104,109,-123,47,2,0,2,74,0,12,108,101,97,115,116,83,105,103,66,105,116,115,74,0,11,109,111,115,116,83,105,103,66,105,116,115,120,112,13,-26,2,-51,111,-17,73,73,-18,-32,-26,-11,-46,-89,50,-37] (offset=377, length=80)
contentType [-84,-19,0,5,115,114,0,33,111,114,103,46,115,112,114,105,110,103,102,114,97,109,101,119,111,114,107,46,117,116,105,108,46,77,105,109,101,84,121,112,101,56,-76,29,-63,64,96,-36,-81,2,0,3,76,0,10,112,97,114,97,109,101,116,101,114,115,116,0,15,76,106,97,118,97,47,117,116,105,108,47,77,97,112,59,76,0,7,115,117,98,116,121,112,101,116,0,18,76,106,97,118] (offset=473, length=190)
timestamp 1555707627482
Code used to connect to Solace
Spring Boot Main Class
#SpringBootApplication
#EnableDiscoveryClient
#Slf4j
#EnableBinding({SendReport.class})
public class ReportServerApplication {
public static void main(final String[] args) {
ApplicationContext ctx = new ClassPathXmlApplicationContext("applicationContext-server.xml");
new SpringApplicationBuilder(ReportServerApplication.class).listeners(new EnvironmentPreparedListener()) .run(args);
}
Class to connect channel to topic:
public interface SendReport {
String TO_NMR = "solace-poc-outbound";
#Output(SendReport.TO_NMR)
MessageChannel output();
}
Message Handler:
#Slf4j
#Component
#EnableBinding({SendReport.class})
public class MessageHandler {
private SendReport sendReport;
public MessageHandler(SendReport sendReport){
this.sendReport = sendReport;
}
#Output(SendReport.TO_NMR)
public void sendMessage(String request) {
log.info("****************** Got this Report Request: " + request);
var message = MessageBuilder.withPayload(request).build();
sendReport.output().send(message);
}
}
Properties used for configuration : application.yml
spring:
cloud:
# spring cloud stream binding
stream:
bindings:
solace-poc-outbound:
destination: TOPIC_NAME
contentType: text/plain
solace:
java:
host: tcp://xyz.abc.com
#port: xxx
msgVpn: yyy
clientUsername: aaa
Dependencies used:
'org.springframework.cloud:spring-cloud-stream',
'com.solace.spring.cloud:spring-cloud-starter-stream-solace:1.1.+'
Observation
Expected result : All JMS headers should get populated by SCS.
Actual result : Some JMS headers are not getting populated.
See JMS Message JavaDocs:
/** Sets the message ID.
*
* <P>This method is for use by JMS providers only to set this field
* when a message is sent. This message cannot be used by clients
* to configure the message ID. This method is public
* to allow a JMS provider to set this field when sending a message
* whose implementation is not its own.
*
* #param id the ID of the message
*
* #exception JMSException if the JMS provider fails to set the message ID
* due to some internal error.
*
* #see javax.jms.Message#getJMSMessageID()
*/
void
setJMSMessageID(String id) throws JMSException;
So, this property cannot be populated from the application level.
In the ActiveMQ I see the code like this:
msg.setMessageId(new MessageId(producer.getProducerInfo().getProducerId(), sequenceNumber));
// Set the message id.
if (msg != message) {
message.setJMSMessageID(msg.getMessageId().toString());
But still: it is not what we can control from the application level.
The priority, deliveryMode and timeToLive ca be populated from the JmsSendingMessageHandler:
if (this.jmsTemplate instanceof DynamicJmsTemplate && this.jmsTemplate.isExplicitQosEnabled()) {
Integer priority = StaticMessageHeaderAccessor.getPriority(message);
if (priority != null) {
DynamicJmsTemplateProperties.setPriority(priority);
}
if (this.deliveryModeExpression != null) {
Integer deliveryMode =
this.deliveryModeExpression.getValue(this.evaluationContext, message, Integer.class);
if (deliveryMode != null) {
DynamicJmsTemplateProperties.setDeliveryMode(deliveryMode);
}
}
if (this.timeToLiveExpression != null) {
Long timeToLive = this.timeToLiveExpression.getValue(this.evaluationContext, message, Long.class);
if (timeToLive != null) {
DynamicJmsTemplateProperties.setTimeToLive(timeToLive);
}
}
}
The JmsCorrelationID must be populated by the JmsHeaders.CORRELATION_ID. The JmsType by the JmsHeaders.TYPE, respectively:
public void fromHeaders(MessageHeaders headers, javax.jms.Message jmsMessage) {
try {
Object jmsCorrelationId = headers.get(JmsHeaders.CORRELATION_ID);
if (jmsCorrelationId instanceof Number) {
jmsCorrelationId = jmsCorrelationId.toString();
}
if (jmsCorrelationId instanceof String) {
try {
jmsMessage.setJMSCorrelationID((String) jmsCorrelationId);
}
catch (Exception e) {
this.logger.info("failed to set JMSCorrelationID, skipping", e);
}
}
Object jmsReplyTo = headers.get(JmsHeaders.REPLY_TO);
if (jmsReplyTo instanceof Destination) {
try {
jmsMessage.setJMSReplyTo((Destination) jmsReplyTo);
}
catch (Exception e) {
this.logger.info("failed to set JMSReplyTo, skipping", e);
}
}
Object jmsType = headers.get(JmsHeaders.TYPE);
if (jmsType instanceof String) {
try {
jmsMessage.setJMSType((String) jmsType);
}
catch (Exception e) {
this.logger.info("failed to set JMSType, skipping", e);
}
}
See DefaultJmsHeaderMapper for more info.

#RabittListener ignores routingKey

On the producer side i'm sending multiple object types serialized as json strings. For distinguishing purposes, i'm sending the class name as a routing key.
On the consuming side i try to use multiple #RabittListener configurations each with a different routing key, so that only the corresponding class is processed.
But it does not work, messages with other routing keys are routed to that method.
What am i doing wrong here?
/**
* Sends an arbitrary object via the configured queue to a amqp instance. To distinguish the message types on the consumer side, we send the message with
* the objects class name as a routing key.
*
* The object is sent as serialized json object
*
* #param obj
* #throws TurbineException
*/
public void send(Object obj) throws TurbineException {
if (enabled) {
String routingKey = obj.getClass().getName();
String json = serializer.toJson(obj);
byte[] payload = json.getBytes();
if (connection == null) {
throw new TurbineException("Trying to send message to RabbitMQ but connection is null");
}
try (Channel channel = connection.createChannel();) {
log.info(String.format("Sending data of type %s to queue %s", routingKey, queueName));
log.info(String.format("Data sent: %s", json));
channel.exchangeDeclare(EMS_META_EXCHANGE_NAME, BuiltinExchangeType.DIRECT);
channel.queueDeclare(queueName, true, false, false, null);
channel.queueBind(queueName, EMS_META_EXCHANGE_NAME, routingKey);
channel.basicPublish(EMS_META_EXCHANGE_NAME, routingKey, null, payload);
}
catch (IOException | TimeoutException e) {
throw new TurbineException(e);
}
}
}
#Autowired
private ProfileMetaService profileMetaService;
#RabbitListener(bindings = #QueueBinding(value = #Queue(value = MetaServiceApplication.EMS_META_QUEUE, durable = "true"), exchange = #Exchange(value = MetaServiceApplication.EMS_META_EXCHANGE_NAME), key = "com.xaxis.janus.turbine.modeling.profile.ProfileMeta"))
#Transactional
public void processMessage(Message message) {
try {
String msg = new String(message.getBody());
if (LOG.isDebugEnabled()) {
LOG.debug("ProfileMeta received: {}", msg);
}
ProfileMeta profileMeta = fromJson(msg, ProfileMeta.class);
profileMetaService.save(profileMeta);
}
catch (Exception e) {
throw new RuntimeException(e);
}
}
RabbitMQ/AMQP doesn't work that way; you need a different queue for each routing key.
Consumer's can't "select" messages from a queue based on the key that was used to route them.
The routing is done in the exchange; each consumer needs its own queue.

Resources