For an unknown reason, MassTransit keeps moving messages of a specific type to the error queue. I believe that a deserialization error is causing this.
What's the best way to debug deserialization errors?
The easiest way to figure out what's going on is enable one of the logging providers and look at the logs for a given consumer that's kicking stuff over. You might need to configure specific loggers, anything under MassTransit would be helpful.
In addition, I believe Chris added the some information to the envelop in a recent version of MT. You might be able to identify the error just by looking at the JSON or XML payload envelop in the message queue.
Related
I have a sink application that fails to write to db, but am having trouble debugging. Note that I also asked a more specific question here, but this question in SO is more general: How should I go about debuggging an SCDF stream pipeline when no errors come up?
What I'm trying to do
I am trying to follow a tutorial (specifically, this tutorial) which uses some prebuilt applications. Everything is up and running with no error messages, and the source application is correctly writing to Kafka. However, the sink seems to be failing to write anything.
Note that I do see the debugging guide here:
https://dataflow.spring.io/docs/stream-developer-guides/troubleshooting/debugging-stream-apps/#sinks
However, this seems to only be relevant when you are writing your own sink.
I am not asking about how to solve my issue per se, but rather about debugging protocol for SCDF apps in general. What is the best way to go about debugging in these kinds of situations where no errors come up but the core functionality isn't working?
Assuming you know how to view the logs and there are no error messages, the next step is to turn on DEBUG logging for spring-integration. You can set a property on the sink logging.level.org.springframework.integration=DEBUG that will log any messages coming into the sink.
We have multiple team nifi applications running in same nifi machine... Is there any way to log the logs specific to my application? Also by default nifi-app.log file is difficult to track the issues and bulletin board shows the error msg for only 5 mins... How to get the errors captured and send an mail alert in Nifi?
Please help me to get through this. Thanks in advance!
There are a couple ways to approach this. One is to route failure relationships from processors to a PutEmail processor which can send an alert on errors. Another is to use a custom reporting task to alert a monitoring service when a certain number of flowfiles are in an error queue.
Finally, we have heard that in multitenant environments, log parsing is difficult. While NiFi aims to reduce or completely eliminate the need to visually inspect logs by providing the data provenance feature, in the event you do need to inspect the logs, we recommend searching the log by processor ID to isolate relevant messages. You can also use NiFi itself to ingest those same logs and perform parsing and filtering activities if desired. Future versions may improve this experience.
By parsing the nifi log, you can separate the logs which is specific to your team applications, by using the processor group id and using Nifi Rest API. Check the below link for the nifi template and python codes to solve this issue:
https://link.medium.com/L6IY1wTimV
You can send all the errors in a Processor Group to the same processor, it could be a regular UpdateAttribute or a custom processor, this processor is going to add the path and all the relevant information and then send this to a general error/logs flow that is going to check the information inside the flowfile regarding to the error, and will make the decision of send or not an email, to whom and this kind of things.
Using this approach, the system keeps simple and inside NiFi, so you don't add more layers of complexity, and you are going to have only one processor to manage the errors per Process Group.
This is the way we are managing errors in my company.
I saw a question & its answer below;
https://stackoverflow.com/a/46128844/7419921
Although I understood that I cannot do anything for the error queue via MassTransit, what should I handle the error queue?
Error messages would be accumlating. It's pressing storage capacity.
It seems that I have nothing to do for the error queue. Is there no choice but to remove them?
If so, I cannot imagine a meaning of the error queue.
The meaning of error queues is very simple. Messages come to error queues because, well, of errors! When you fix issues in your application, you can move messages from the error queue back to the regular queue using Shovel plugin, and voila - you recovered lost data. We do this very often.
If you cannot move them back because these messages aren't actual anymore or they contain wrong data - this is also very valuable since using these messages you can reproduce the issue and see if you can fix the sender.
I've found references in a few places to some internal logging capabilities of ZMQ. The functionality that I think might exist is the ability to connect to either or both of a inproc or ipc SUB socket and listen to messages that give information about the internal state of ZMQ. This would be quite useful when debugging a distributed application. For instance, if messages are missing/being dropped, it might shed some light on why they're being dropped.
The most obvious mention of this is here: http://lists.zeromq.org/pipermail/zeromq-dev/2010-September/005724.html, but it's also referred to here: http://lists.zeromq.org/pipermail/zeromq-dev/2011-April/010830.html. However, I haven't found any documentation of this feature.
Is some sort of logging functionality truly available? If so, how is it used?
Some grepping through the git history eventually answered my question. The short answer is that a way for ZMQ to transmit logging messages to the outside world was implemented, but it was never used to actually send logging messages by the rest of the code base. After a while it was removed since nothing used it.
The commit that originally added it making use of an inproc socket:
https://github.com/zeromq/libzmq/commit/ce0972dca3982538fd123b61fbae3928fad6d1e7
The commit that added a new "sys" socket type specifically to support the logging:
https://github.com/zeromq/libzmq/commit/651c1adc80ddc724877f2ebedf07d18e21e363f6
JIRA issue, pull request, and commit to remove the functionality:
https://zeromq.jira.com/browse/LIBZMQ-336
https://github.com/zeromq/libzmq/pull/277
https://github.com/zeromq/libzmq/commit/5973da486696aca389dab0f558c5ef514470bcd2
I have this question, i am just throwing it out there. I am implementing a small logging feature for my spring based REST API server for logging all requests coming in.
I am expecting 1000s of users to use this API so with a blocking i/o logger, it's going to slow down everything. I have two approaches to solve the problem:
1. Have a async logger using an in-memory arrylist. then use the spring scheduler to flush this out to a log file periodically.
2. Use JMS and send the logs to the queue. Let the queue handle the logging asynchronously.
Has anyone done this before with spring. Though i am for option 2, are there better ways of doing this? Need some expert advice. Thanks everyone !
More info - I think synchronous logging will be a bottle neck because this REST API is consumed by a front end RoR app. So one session of the user will definitely result in 100s of API calls occuring very frequently. I am logging the actual request along with the JSON sent in the POSTs.
Has anyone done this before with spring.
Not so strangely, yes - Asynchronous Logging Using Spring
The article mentions that if you don't want any log events to be lost, JMS would be the way to go - otherwise sticking to Async makes sense for high volume logging.
If you really want to build your own logger, I suggest you to take a look at akka, it is much easier than JMS to set up.
You can use it locally (use of all CPU cores of your local machine), or even with remote agents.