I'm using ElasticMQ (via docker image v1.3.3) and can't get the DLQ to work.
This is my elasticmq.conf:
include classpath("application.conf")
node-address {
protocol = http
host = localhost
port = 9324
context-path = ""
}
rest-sqs {
enabled = true
bind-port = 9324
bind-hostname = "0.0.0.0"
// Possible values: relaxed, strict
sqs-limits = strict
}
rest-stats {
enabled = true
bind-port = 9325
bind-hostname = "0.0.0.0"
}
queues {
main {
defaultVisibilityTimeout = 10 seconds
delay = 2 seconds
receiveMessageWait = 0 seconds
deadLettersQueue {
name = "retry"
maxReceiveCount = 1
}
}
retry {
defaultVisibilityTimeout = 10 seconds
delay = 2 seconds
receiveMessageWait = 0 seconds
}
deadletter {
defaultVisibilityTimeout = 10 seconds
delay = 2 seconds
receiveMessageWait = 0 seconds
}
}
I'm sending a message like so (using the AWS CLI):
aws --endpoint-url http://localhost:9324 sqs send-message --queue-url http://localhost:9324/queue/main --message-body "Hello, queue"
And receiving it like so:
aws --endpoint-url http://localhost:9324 sqs receive-message --queue-url http://localhost:9324/queue/main --wait-time-seconds 10
I'm not deleting the message from the queue but the message is being deleted and not being moved to the DLQ (i.e., the retry queue). I'm also trying to receive the message with Java code and getting the same result.
Why is that?
Related
I am running a terraform through Jenkins which starts up an ec2 then runs a shell script on it using user_data. I run this job 23 times in parallel, and for some reason each time only a few of them (anywhere from 1 to 8 and always different indices) will hang on "aws_instance.genomic-etl-ec2: Still creating..." until the connection times out after approximately an hour and throws a RequestExpired error, with no further details on why. The other instances start fine within around 2-3 minutes each.
My resource:
data "template_file" "my-user_data" {
template = file("scripts/my_script.sh")
}
data "template_cloudinit_config" "my-user-data" {
gzip = true
base64_encode = true
# user_data
part {
content_type = "text/x-shellscript"
content = data.template_file.my-user_data.rendered
}
}
resource "aws_instance" "genomic-etl-ec2" {
ami = var.ami-id
instance_type = "m5.12xlarge"
associate_public_ip_address = true
subnet_id = var.my-subnet-us-east-id
iam_instance_profile = "my-deployment-profile"
user_data = data.template_cloudinit_config.my-user-data.rendered
vpc_security_group_ids = [
aws_security_group.my-sg1.id,
aws_security_group.my-sg2.id
]
root_block_device {
delete_on_termination = true
encrypted = true
volume_size = 1000
}
provisioner "local-exec" {
command = "sleep 40"
}
tags = {
Owner = "Me"
Environment = "development"
Name = "My EC2 - ${id}"
automaticPatches = "1"
}
}
Sometimes AWS instances take a long time to become fully available. It's not uncommon for those to take longer than Terraform's default timeout, causing Terraform to fail.
As per the official documentation on the Terraform aws_instance resource, the create timeout defaults to 10 minutes. If a particular instance type is taking longer than 10 minutes to become available, then you need to increase the create timeout setting:
resource "aws_instance" "genomic-etl-ec2" {
# ...
timeouts {
create = "20m"
}
}
I'm exposing a simple SSE endpoint using the SseEmitter Spring API, persisting all the emitters in a ConcurrentHashMap. The timeout for each emitter is set to 24 hours. Every 10 seconds I'm sending a message to all the clients. Clients are subscribed with native EventSource implementation, listening for events of particular name.
Unfortunately, I've noticed that every 5 minutes the connection is lost and reestablished again - even though timeout of emitter was explicitly set to 24 hours. Client's part does log it as an error, however on server side there's nothing. The issue occurs on both Tomcat and Jetty. I'd like to keep the session open without any interruptions, so resetting the connection every 5 minutes is unacceptable. Any ideas why this could be happening?
#RestController
#RequestMapping("api/v1/sse")
class SseController {
private val emitters = ConcurrentHashMap<String, SseEmitter>()
#GetMapping
fun initConnection(#RequestParam token: String): SseEmitter {
logger.info { "Init connection from $token" }
val emitter = SseEmitter(24 * 60 * 60 * 1000)
emitter.onCompletion {
logger.info { "Completion" }
emitters.remove(token)
}
emitter.onTimeout { logger.info { "Timeout " } }
emitter.onError { logger.error(it) { "Error" } }
emitters[token] = emitter
return emitter
}
#Scheduled(fixedRate = 10000)
fun send() {
emitters.forEach { (k, v) ->
logger.info { "Sending message to $k" }
v.send(
SseEmitter.event()
.id(UUID.randomUUID().toString())
.name("randomEvent")
.data("some data")
)
}
}
}
const eventSource = new EventSource(url);
eventSource.addEventListener('randomEvent', (e) =>
console.log(e.data)
);
eventSource.onerror = (e) => console.log(e);
Alright, seems it was an issue with Stackblitz's service worker. I've just implemented the same client-side solution in Chrome's plain console and the disconnecting is no longer happening.
I'm learning TF and trying to apply an infrastructure that creates:
a simple lambda function
an SNS topic
get that lambda to subscribe the SNS topic
a Cloud Watch Event that publishes a message to the topic at some interval
a Cloud Watch Log Group to check if the lambda gets notified by the SNS
The lambda permission to allow calls from SNS
I'm able to apply that successfully. The infrastructure seems perfectly fine (it has the same aspect when I create that myself through the visual aws console)
But the cloud watch Event doesn't get triggered (when built from TF), so no message is published to SNS and lambda doesn't get called. I don't know why
Anyone know how can I accomplish that? Bellow my .tf script:
provider "aws" {
region = "us-east-1"
}
//lambda function handler & code file
resource "aws_lambda_function" "lambda-function" {
function_name = "Function01"
handler = "com.rafael.lambda.Function01"
role = "arn:aws:iam::12345:role/LambdaRoleTest"
runtime = "java8"
s3_bucket = aws_s3_bucket.sns-test.id
s3_key = aws_s3_bucket_object.file_upload.id
source_code_hash = filebase64sha256("../target/sns-cw-lambda-poc.jar")
}
//allow sns to call lambda
resource "aws_lambda_permission" "allow-sns-to-lambda" {
function_name = aws_lambda_function.lambda-function.function_name
action = "lambda:InvokeFunction"
principal = "sns.amazonaws.com"
source_arn = aws_sns_topic.call-lambdas-topic.arn
statement_id = "AllowExecutionFromSNS"
}
//app s3 repository
resource "aws_s3_bucket" "sns-test" {
bucket = "app-bucket-12345"
region = "us-east-1"
}
//app jar file
resource "aws_s3_bucket_object" "file_upload" {
depends_on = [
aws_s3_bucket.sns-test
]
bucket = aws_s3_bucket.sns-test.id
key = "sns-cw-lambda-poc.jar"
source = "../target/sns-cw-lambda-poc.jar"
server_side_encryption = "AES256"
etag = filebase64sha256("../target/sns-cw-lambda-poc.jar")
}
//to check lambda exec logs
resource "aws_cloudwatch_log_group" "lambda-cloudwatch-logs" {
name = "/aws/lambda/${aws_lambda_function.lambda-function.function_name}"
retention_in_days = 1
}
//rule to trigger SNS
resource "aws_cloudwatch_event_rule" "publish-sns-rule" {
name = "publish-sns-rule"
schedule_expression = "rate(1 minute)"
}
//cloud watch event targets SNS
resource "aws_cloudwatch_event_target" "sns-publish" {
count = "1"
rule = aws_cloudwatch_event_rule.publish-sns-rule.name
target_id = aws_sns_topic.call-lambdas-topic.name
arn = aws_sns_topic.call-lambdas-topic.arn
}
//SNS topic to subscribe
resource "aws_sns_topic" "call-lambdas-topic" {
name = "call-lambdas-topic"
}
//lambda subscribes the topic, so it should be nofied when other resource publishes to the topic
resource "aws_sns_topic_subscription" "sns-lambda-subscritption" {
topic_arn = aws_sns_topic.call-lambdas-topic.arn
protocol = "lambda"
endpoint = aws_lambda_function.lambda-function.arn
}
I figured it out, I forgot to add the SNS policies that allow CloudWatch to publish to SNS topic. To get the above script to work, just add this:
resource "aws_sns_topic_policy" "default" {
count = 1
arn = aws_sns_topic.call-lambdas-topic.arn
policy = "${data.aws_iam_policy_document.sns_topic_policy.0.json}"
}
data "aws_iam_policy_document" "sns_topic_policy" {
count = "1"
statement {
sid = "Allow CloudwatchEvents"
actions = ["sns:Publish"]
resources = [aws_sns_topic.call-lambdas-topic.arn]
principals {
type = "Service"
identifiers = ["events.amazonaws.com"]
}
}
}
I have set up icinga2 to monitor a few services with different intervals, so one service might be checked every 10 seconds. If it gives a critical error I will receive a notification, but I will receive it every 10 seconds if the error persists, or until I acknowledge it. I just want to receive it once for each state change. Then maybe after a specified time again, but it is not that important.
Here is my config:
This is more or less the standard template.conf, but I have added the "interval=0s", because I read that it should prevent notifications from being sent multiple times.
template Notification "mail-service-notification" {
command = "mail-service-notification"
interval = 0s
states = [ OK, Critical ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
vars += {
notification_logtosyslog = false
}
period = "24x7"
}
And here is the part of the notification.conf that includes the template:
object NotificationCommand "telegram-service-notification" {
import "plugin-notification-command"
command = [ SysconfDir + "/icinga2/scripts/telegram-service-notification.sh" ]
env = {
NOTIFICATIONTYPE = "$notification.type$"
SERVICEDESC = "$service.name$"
HOSTNAME = "$host.name$"
HOSTALIAS = "$host.display_name$"
HOSTADDRESS = "$address$"
SERVICESTATE = "$service.state$"
LONGDATETIME = "$icinga.long_date_time$"
SERVICEOUTPUT = "$service.output$"
NOTIFICATIONAUTHORNAME = "$notification.author$"
NOTIFICATIONCOMMENT = "$notification.comment$"
HOSTDISPLAYNAME = "$host.display_name$"
SERVICEDISPLAYNAME = "$service.display_name$"
TELEGRAM_BOT_TOKEN = TelegramBotToken
TELEGRAM_CHAT_ID = "$user.vars.telegram_chat_id$"
}
}
apply Notification "telegram-icingaadmin" to Service {
import "mail-service-notification"
command = "telegram-service-notification"
user_groups = [ "icingaadmins" ]
assign where host.name
}
I think you had a typo.
It should work if you set interval = 0 (not "interval = 0s")
After that change you must restart the icinga service.
I try to do a simple cluster using akka.net.
The goal is to have a server receiving request and akka.net processing it through it cluster.
For testing and learning I create a simple WCF service that receive a math equation and I want to send this equation to be solved.
I have one project server and another client.
The configuration on the server side is :
<![CDATA[
akka {
actor {
provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
debug {
receive = on
autoreceive = on
lifecycle = on
event-stream = on
unhandled = on
}
deployment {
/math {
router = consistent-hashing-group #round-robin-pool # routing strategy
routees.paths = [ "/user/math" ]
virtual-nodes-factor = 8
#nr-of-instances = 10 # max number of total routees
cluster {
enabled = on
max-nr-of-instances-per-node = 2
allow-local-routees = off
use-role = math
}
}
}
}
remote {
helios.tcp {
transport-class = "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote"
applied-adapters = []
transport-protocol = tcp
port = 8081
hostname = "127.0.0.1"
}
}
cluster {
seed-nodes = ["akka.tcp://ClusterSystem#127.0.0.1:8081"] # address of seed node
}
}
]]>
On the Client side the configuration is like this :
<![CDATA[
akka {
actor.provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
remote {
log-remote-lifecycle-events = DEBUG
log-received-messages = on
helios.tcp {
transport-class = "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote"
applied-adapters = []
transport-protocol = tcp
port = 0
hostname = 127.0.0.1
}
}
cluster {
seed-nodes = ["akka.tcp://ClusterSystem#127.0.0.1:8081"] # address of the seed node
roles = ["math"] # roles this member is in
}
actor.deployment {
/math {
router = round-robin-pool # routing strategy
routees.paths = ["/user/math"]
nr-of-instances = 10 # max number of total routees
cluster {
enabled = on
allow-local-routees = on
use-role = math
max-nr-of-instances-per-node = 10
}
}
}
}
]]>
The cluster connection seems to correctly be made. I see the status [UP] and the association with the role "math" that appeared on the server side.
Event follwing the example on the WebCramler, I don't achieved to make a message to be delivered. I always get a deadletters.
I try like this :
actor = sys.ActorOf(Props.Empty.WithRouter(FromConfig.Instance), "math");
or
var actor = sys.ActorSelection("/user/math");
Does someone know a good tutorial or could help me ?
Thanks
Some remarks:
First: assuming your sending work from the server to the client. Then you are effectively remote deploying actors on your client.
Which means only the server node needs the actor.deployment config section.
The client only needs the default cluster config (and your role setting ofcourse).
Second: Try to make it simpler first. Use a round-robin-pool instead. Its much simpler. Try to get that working. And work your way up from there.
This way its easier to eliminate configuration/network/other issues.
Your usage: actor = sys.ActorOf(Props.Empty.WithRouter(FromConfig.Instance), "math"); is correct.
A sample of how your round-robin-pool config could look:
deployment {
/math {
router = round-robin-pool # routing strategy
nr-of-instances = 10 # max number of total routees
cluster {
enabled = on
max-nr-of-instances-per-node = 2
allow-local-routees = off
use-role = math
}
}
}
Try this out. And let me know if that helps.
Edit:
Ok after looking at your sample. Some things i changed
ActorManager->Process: Your creating a new router actor per request. Don't do that. Create the router actor once, and reuse the IActorRef.
Got rid of the minimal cluster size settings in the MathAgentWorker project
Since your not using remote actor deployment. I changed the round-robin-pool to a round-robin-group.
After that it worked.
Also remember that if your using the consistent-hashing-group router you need to specify the hashing key. There are various ways to do that, in your sample i think the easiest way would be to wrap the message your sending to your router in a ConsistentHashableEnvelope. Check the docs for more information.
Finally the akka deployment sections looked like this:
deployment {
/math {
router = round-robin-group # routing strategy
routees.paths = ["/user/math"]
cluster {
enabled = on
allow-local-routees = off
use-role = math
}
}
}
on the MathAgentWorker i only changed the cluster section which now looks like this:
cluster {
seed-nodes = ["akka.tcp://ClusterSystem#127.0.0.1:8081"] # address of the seed node
roles = ["math"] # roles this member is in
}
And the only thing that the ActorManager.Process does is:
return await Program.Instance.RouterInstance.Ask<TResult>(msg, TimeSpan.FromSeconds(10));