Acknowledged messages getting retried - ruby

Question

I am using Google Pub/Sub ruby client to process messages sent to multiple topics. For each message received, I enqueue it using ActiveJob and acknowledge it to mark it as processed.
subscription.listen do |msg|
Rails.logger.debug("Processing message with id #{msg.message_id}")
MyJob.perform_later(JSON.parse(msg.data))
msg.acknowledge!
Rails.logger.debug("ACKed message with id #{msg.message_id}")
end
However, by checking the logs I see that messages that had already been acknowledged are being processed once and again (see extract from logs):
2021-05-04 02:15:17.089 EDT "Processing message with id 2260372604401883"
2021-05-04 02:15:17.180 EDT "ACKed message with id 2260372604401883"
2021-05-04 02:17:58.121 EDT "Processing message with id 2260372604401883"
2021-05-04 02:17:58.186 EDT "ACKed message with id 2260372604401883"
2021-05-04 02:20:59.899 EDT "Processing message with id 2260372604401883"
2021-05-04 02:20:59.985 EDT "ACKed message with id 2260372604401883"
2021-05-04 02:22:21.083 EDT "Processing message with id 2260372604401883"
2021-05-04 02:22:21.394 EDT "ACKed message with id 2260372604401883"
2021-05-04 02:24:18.389 EDT "Processing message with id 2260372604401883"
2021-05-04 02:24:18.485 EDT "ACKed message with id 2260372604401883"
2021-05-04 02:25:54.274 EDT "Processing message with id 2260372604401883"
2021-05-04 02:25:54.385 EDT "ACKed message with id 2260372604401883"
2021-05-04 02:26:59.087 EDT "Processing message with id 2260372604401883"
2021-05-04 02:26:59.184 EDT "ACKed message with id 2260372604401883"
The number of unacked messages reported by Google is dramatically increasing, so I suspect that the acknowledge! method is not behaving as expected.
Ruby version: 2.6.6
google-cloud-pubsub version: 2.6.1 (latest)

Answer 1

I have never seen this problem before, but I can suggest a few things to try:
Log the subscriber ack deadline to make sure it allows enough time for MyJob.perform_later to return. This is really just a sanity check since it's most likely the default 60 seconds, but you might as well check.
Configure an error handler on the subscriber:
subscriber = subscription.listen do |msg|
# process message
msg.acknowledge!
end
Rails.logger.debug("subscriber ack deadline: #{subscriber.deadline}")
# Register to be notified when unhandled errors occur.
subscriber.on_error do |error|
Rails.logger.error(error.message)
end
Configure gRPC debug logging. The generic logger configuration instructions are in LOGGING.md. To adapt them to Rails you can place the following in config/initializers/grpc_logging.rb:
module MyLogger
def logger
Rails.logger
end
end
# Define a gRPC module-level logger method before grpc/logconfig.rb loads.
module GRPC
extend MyLogger
end