I'm trying to create a new queue, but when using
cf create-service aws-sqs standard my-q
the name of the queue in AWS is automatically assigned and is just an id composed of random letters and numbers.
This is fine when using the normal java client. However, we want to use spring-cloud-aws-messaging (#SqsListener annotation), because it offers us deletion policies out of the box, and a way to extend visibility, so that we can implement retries easily.
#SqsListener(value = "my-q", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void listen(TestItem item, Visibility visibility) {
log.info("received message: " + item);
//do business logic
//if call fails
visibility.extend(1000);
//throw exception
//if no failure, message will be dropped
}
The queue name on the annotation is declared, so we can't change it dynamically after reading the VCAP_SERVICE environment variable injected by PCF on the application.
The only alternative we can think of is use reflection to set accessibility on value of the annotation, and set the value to the name on the VCAP_SERVICE, but that's just nasty, and we'd like to avoid it if possible.
Is there any way to change the name of the queue to something specific on creation? This suggests that it's possible, as seen below:
cf create-service aws-sqs standard my-q -c '{ "CreateQueue": { "QueueName": “my-q”, "Attributes": { "MaximumMessageSize": "1024"} } }'
However, this doesn't work. It returns:
Incorrect Usage: Invalid configuration provided for -c flag. Please
provide a valid JSON object or path to a file containing a valid JSON
object.
How do I set the name on creation of the queue? Or the only way to achieve my end goal is to use reflection?
EDIT: As pointed out by Daniel Mikusa, the double quotes were not real double quotes, and that was causing the error. The command is successful now, however it doesn't create the queue with the intended name. I'm now wondering if this name needs to be set on bind-service instead. The command has a -c option too but I cannot find any documentation to support which parameters are available for a aws-sqs service.
Related
I want to create API, which takes part of its input from environment (urls etc..). Basically a Jar file.
I also wan't the values being auto updated when application.properties changes. When there is change, this is called:
org.springframework.cloud.endpoint.RefreshEndpoint#refresh
However, I consider it bad practice to have magic env variable keys like 'server.x.url' in the Api contract between client application and the jar. (Problem A)
That's why I'd like to use Api like this. But there's problem of old values.
public class MyC {
TheAPI theApi=null;
void MyC(){
theApi = new TheApi();
theApi.setUrl( env.get("server.x.url") );
}
doStuff() {
theApi.doStuff(); // fails as theApi has obsolete value of server.x.url, Problem B
}
So either I have ugly API contract or I get obsolete values in the API calls.
I'm sure there must be Spring way of solving this, but I cant get it to my head just now.
I've a requirement to download a file from S3 based on a message content. In other words, the file to download is previously unknown, I've to search and find it at runtime. S3StreamingMessageSource doesn't seem to be a good fit because:
It relies on polling where as I need to wait for the message.
I can't find any way to create a S3StreamingMessageSource dynamically in the middle of a flow. gateway(IntegrationFlow) looks interesting but what I need is a gateway(Function<Message<?>, IntegrationFlow>) that doesn't exist.
Another candidate is S3MessageHandler but it has no support for listing files which I need for finding the desired file.
I can implement my own message handler using AWS API directly, just wondering if I'm missing something, because this doesn't seem like an unusual requirement. After all, not every app just sits there and keeps polling S3 for new files.
There is S3RemoteFileTemplate with the list() function which you can use in the handle(). Then split() result and call S3MessageHandler for each remote file to download.
Although the last one has functionality to download the whole remote dir.
For anyone coming across this question, this is what I did. The trick is to:
Set filters later, not at construction time. Note that there is no addFilters or getFilters method, so filters can only be set once, and can't be added later. #artem-bilan, this is inconvenient.
Call S3StreamingMessageSource.receive manually.
.handle(String.class, (fileName, h) -> {
if (messageSource instanceof S3StreamingMessageSource) {
S3StreamingMessageSource s3StreamingMessageSource = (S3StreamingMessageSource) messageSource;
ChainFileListFilter<S3ObjectSummary> chainFileListFilter = new ChainFileListFilter<>();
chainFileListFilter.addFilters(
new S3SimplePatternFileListFilter("**/*/*.json.gz"),
new S3PersistentAcceptOnceFileListFilter(metadataStore, ""),
new S3FileListFilter(fileName)
);
s3StreamingMessageSource.setFilter(chainFileListFilter);
return s3StreamingMessageSource.receive();
}
log.warn("Expected: {} but got: {}.",
S3StreamingMessageSource.class.getName(), messageSource.getClass().getName());
return messageSource.receive();
}, spec -> spec
.requiresReply(false) // in case all messages got filtered out
)
Is it possible to extend the visibility time out of a message that is in flight.
See:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html.
Section: Changing a Message's Visibility Timeout.
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQSClient.html#changeMessageVisibility-com.amazonaws.services.sqs.model.ChangeMessageVisibilityRequest-
In summary I want to be able to extend the first set visibility timeout for a given message that is in flight.
Example if 15secs have passed I then want to extend the timeout by another 20secs. Better example in java docs above.
From my understanding in the links above you can do this on the amazon side.
Below are my current settings;
SqsMessageDrivenChannelAdapter adapter =
new SqsMessageDrivenChannelAdapter(queue);
adapter.setMessageDeletionPolicy(SqsMessageDeletionPolicy.ON_SUCCESS);
adapter.setMaxNumberOfMessages(1);
adapter.setSendTimeout(2000);
adapter.setVisibilityTimeout(200);
adapter.setWaitTimeOut(20);
Is it possible to extend this timeout?
Spring Cloud AWS supports this starting with Version 2.0. Injecting a Visiblity parameter in your SQS listener method does the trick:
#SqsListener(value = "my-sqs-queue")
void onMessageReceived(#Payload String payload, Visibility visibility) {
...
var extension = visibility.extend(20);
...
}
Note, that extend will work asynchronously and will return a Future. So if you want to be sure further down the processing, that the visibility of the message is really extended at the AWS side of things, either block on the Future using extension.get() or query the Future with extension.isDone()
OK. Looks like I see your point.
We can change visibility for particular message using API:
AmazonSQS.changeMessageVisibility(String queueUrl, String receiptHandle, Integer visibilityTimeout)
For this purpose in downstream flow you have to get access to (inject) AmazonSQS bean and extract special headers from the Message:
#Autowired
AmazonSQS amazonSqs;
#Autowired
ResourceIdResolver resourceIdResolver;
...
MessageHeaders headers = message.getHeaders();
DestinationResolver destinationResolver = new DynamicQueueUrlDestinationResolver(this.amazonSqs, this.resourceIdResolver);
String queueUrl = destinationResolver.resolveDestination(headers.get(AwsHeaders.QUEUE));
String receiptHandle = headers.get(AwsHeaders.RECEIPT_HANDLE);
amazonSqs.changeMessageVisibility(queueUrl, receiptHandle, YOUR_DESIRED_VISIBILITY_TIMEOUT);
But eh, I agree that we should provide something on the matter as out-of-the-box feature. That may be even something similar to QueueMessageAcknowledgment as a new header. Or even just one more changeMessageVisibility method to this one.
Please, raise a GH issue for Spring Cloud AWS project on the matter with link to this SO topic.
I am successfully exposing a method using the Spring annotations:
#Override
#ManagedOperation(description = "synchronize To Local Directory")
#ManagedOperationParameters({ #ManagedOperationParameter(name = "localDirectory", description = "The Local Directory") }
public void synchronizeToLocalDirectory(File localDirectory) {
super.synchronizeToLocalDirectory(localDirectory);
}
By successful I mean that it can be seen in jConsole.
However, the operation cannot be invoked, obviously because the parameter; localDirectory, needs to be specified.
localDirectory is of type File.
The problem is that the button/option to set the localDirectory in the invocation of the operation is not active.
If I change the type from File to String it works, but I do not want to do that - I preferably want the user to choose a directory via a file chooser dialog.
Questions:
Does JMX cater for a File type parameter in a operation that can be specified in something like jConsole? Or does it have to be setup as a Composite type?
Thanks
No java.io.File is not a directly supported JMX type. As you are correctly suggesting you need to either use a composite type or adjust the management interface to accept eg. String value of the file path.
I'm trying to send a message to a queue using a Message object and am getting the error
The specified format name does not support the requested operation. For example, a direct queue format name cannot be deleted.
Here is the code.
Order ord = new Order(new Guid(), "Smith & Smith");
Message orderMessage = new Message(ord);
orderMessage.UseEncryption = true;
orderMessage.EncryptionAlgorithm = EncryptionAlgorithm.Rc2;
orderMessage.Recoverable = true;
orderMessage.Priority = MessagePriority.VeryHigh;
orderMessage.TimeToBeReceived = TimeSpan.FromHours(1);
orderMessage.UseJournalQueue = true;
orderMessage.Body = "Test Encryption";
queue.Send(orderMessage, "Encrypted Order");
Any help with this is appreciated.
Tom
Did you ever solve this? I came across this problem myself and found out I needed to use (just like the error says) a different format name.
The strange thing was that if I set UseAuthentication property using the MQ certificate, then it worked. But if I also wanted to set UseEncryption, then it did not work.
You do not specify your queue/server setup/formats, but I suspect you're trying to send from one machine to another machine's public queue within the same domain, using DIRECT formatname? As the MQ Manager will use the domain AD to lookup the certificate and queue details, it raises an exception as the format name is invalid (not the same as specified in the AD). So instead of using the direct format, use the queue ID to define the formatname. I switched this:
"FormatName:Direct=TCP:111.222.1.22\your_public_queue"
with this:
"FormatName:PUBLIC=7EB2A53C-7593-462C-A568-5A0EFA26D91D"
Now it worked. You can find your queue ID by right-clicking your queue on the receiver machine and then go to Properties->General and see the value specified in field "ID".
I have found that getting the FormatName correct whether public or private in nature will save hours of work. It's incredibly important to understand the setup of each (Public requiring AD and private does not when access remotely). This is a great summary of FormatName.
https://blogs.msdn.microsoft.com/johnbreakwell/2009/02/26/difference-between-path-name-and-format-name-when-accessing-msmq-queues/
One note on this issue, if your queue format name starts this way: "FormatName:Direct=" then you will receive the error "The specified format name does not support the requested operation. For example, a direct queue format name cannot be deleted" if you try to access the queue's QueueName property. Use the queue's FormatName property instead.