Is there any way to put the std logs provided by the application and the errors to a span?
I know that I can send some logs with span.LogKV() or span.LogFields() but it makes code look bad while there are same logs with both application logger and span logger.
I'm looking for an automated way to put all logs to the corresponding span.
If you are using spring boot with auto configuration, the logs printed using log4j will be instrumented and sent automatically in the span.
In Go this is not very straightforward, and largely depends on the logging library you use and the interface it provides. One example is implemented in the HotROD demo in the Jaeger repository and it is described in the blog post accompanying the demo. It uses go.uber.org/zap logging library underneath and allows to write log statements accepting the Context argument:
logger.For(ctx).Info(
"Searching for nearby drivers",
zap.String("location", location),
)
Behind the scenes logger.For(ctx) captures the current tracing spans and adds all log statements to that span, in addition to sending then to stdout as usual.
To my knowledge, go.uber.org/zap does not yet support this mode of logging natively, and therefore requires a wrapper.
If you use another logging library, than on the high level this is what needs to happen:
there must be a way to pass the Context to the logging methods, which could be captured internally by the logging framework in the log message data structure
the "appender" (or whatever it's called, I am using log4j terminology) that's responsible for taking log messages and writing them to some output stream can be extended to retrieve the Span from the Context and write the log message to the Span using span.LogKV() or similar API.
Related
I have a sink application that fails to write to db, but am having trouble debugging. Note that I also asked a more specific question here, but this question in SO is more general: How should I go about debuggging an SCDF stream pipeline when no errors come up?
What I'm trying to do
I am trying to follow a tutorial (specifically, this tutorial) which uses some prebuilt applications. Everything is up and running with no error messages, and the source application is correctly writing to Kafka. However, the sink seems to be failing to write anything.
Note that I do see the debugging guide here:
https://dataflow.spring.io/docs/stream-developer-guides/troubleshooting/debugging-stream-apps/#sinks
However, this seems to only be relevant when you are writing your own sink.
I am not asking about how to solve my issue per se, but rather about debugging protocol for SCDF apps in general. What is the best way to go about debugging in these kinds of situations where no errors come up but the core functionality isn't working?
Assuming you know how to view the logs and there are no error messages, the next step is to turn on DEBUG logging for spring-integration. You can set a property on the sink logging.level.org.springframework.integration=DEBUG that will log any messages coming into the sink.
I'm trying out the .Net agent in Elastic APM and I'm using a C# application which is created using a framework called ASP.net Boilerplate. I've added the core libraries as mentioned in the documentation and added the settings in appsettings.json. This enables the default instrumentation and I got traces in the APM visualized through Kibana.
Currently I've got a node.js application running and I publish a message to a RabbitMQ queue with the traceparent in the message payload. The C# app reads the published message. I need to create a transaction or span using this traceparent / trace id so that Kibana would show the trace among the distributed systems.
I want to know if there is a way to create a transaction (or span) using a traceparent that is being sent from another system not using a HTTP protocol. I've checked the Elastic APM agent documentation -> Public API for information but couldnt find any information on this. Is there a way? Thanks.
I want to know if there is a way to create a transaction (or span) using a traceparent that is being sent from another system not using a HTTP protocol.
Yes, this is possible and there is an API for it. This part of the documentation explains it.
So you'll need to do this when you start your transaction - I imagine in your scenario this will be when you read a message from RabbitMQ.
When you start the transaction there is an optional parameter called distributedTracingData - if you pass it, then the transaction will reuse the traceid which you passed through RabbitMQ, and this way the new transaction will be part of the whole trace. If you don't pass this parameter, a new traceid will be generated and a new trace will be started.
Another comment that may help: you pass the trace id into the method where you start the transaction and each span will inherit this trace id within a transaction - so you control this on the transaction level and accordingly you don't pass it into a span.
Here is a small code snippet on how this would look:
serializedDistributedTracingData = //read this from the message which you get RabbitMq
var transaction2 = Agent.Tracer.StartTransaction("RadFromQueue", "RabbitMQRead",
DistributedTracingData.TryDeserializeFromString(serializedDistributedTracingData));
#gregkalapos, again thank you for the information. I checked how to acquire the neccessary trace information as in node.js agent documentation and when I debugged noticed that it was the trace id. Next in the C# consumer end I placed a code snippet as mentioned in the .Net agent and gave it a run. Kibana displayed the transactions from two different services in a single trace as I hoped it would.
I have 5 web applications which I developed using Springboot(A,B,C,D and E).
Below are the flow of 2 calls
First Flow:
A's Controllers --> A's Service --> B's Controller --> B'Service --> C's Controller --> C's Service --> C's Dao --> DB
SecondFlow:
A's Controllers --> A's Service --> D's Controller --> D'Service --> B's Controller --> B'Service --> C's Controller --> C's Service --> C's Dao --> DB
Once fetch/push data from/into the DB then the corresponding methods are returning some value. For each and every method logging the status (input details and returning status). I am able to see logs in each service separately. But I want to see complete one request-response(A's controller request to A's controller response) cycle logs in one file.
How Can I achieve it?
This is a ver bad idea, but let's take a step back and look at the problem instead of guessing a solution.
You have multiple applications that collaborate together to execute (distributed) transactions. You need to trace those interactions to see your dataflow. This is very useful for many reasons so it's correct that you care about it. It is also correct to collect all you log entries in a single sink, even if it won't be a file because it is not well suited to manage prodcution workloads. A typical scenario that many organization implements is the following
Each application send logs to files or standard output
For each node of you infrastructure there is an agent that reads that streams, does some basic conversion (eg. translates log entries in a common format) and send data to a certain sink
The sink is a database, the best technology option is a DBMS without strict requirements about data schema (you are storing everything in a single huge table after all) and transactional properites (if the data are logs, you are fine with an optimistic concurrency control). You also want some tool that is more good at reads than writes and have good performance in complex searches to drill down a large amount of structured data
a Dashboard to read logs, make searches and even create dashboard with synthetic stats about events
BONUS: use a buffer to manage load spikes
There are precise tools to do the job and they are
logstash/beats/fluentd
Elasticsearch....what else? ;)
Kibana, the favourite Elasticsearch client
BONUS: rabbimq/kafka/otherMessageBroker or Redis
but you still miss a step
Suppose you call a REST API, something simple like a POST /users/:userId/cart
API Gateway receives your request with a JWT
API Gateway calls Authentication-service to validate and decode the JWT
API Gateway calls Authorization-service to check if the client as right to perform the request
API Gateway calls User-service to find :userId
User-Service calls Cart-service to add the product on :userId cart
Cart-Service calls Notification-Service to decide whether is needed to send a notification for the completed task
Notification-Service calls Push-Gateway to invoke an external push notification service
....and back
to not get lost in this labyrinth you NEED just one thing: the correlation ID
Correlation IDs attach a unique ID to all interaction beetween these microservices (headers in HTTP calls or AMQP messages, for instance) and your custom log library (because you've already built a custom logging library and shared it among all the teams) capture this ID to include it in every log entry wrote in the context of the single request processed from each of those microservices. You can even add that correlation ID in the client response, catch it if the respone carry out an error code and perform a query on your logs DB to find all the entries with the given correlation ID. If the system clock works, they will be retrieved in the correct time order and you will be able to reconstruct the dataflow
Distributed Systems make everything more complicate and add a lot of overhead on things that we ever done before, but if you put in your pocket the right tools to manage the complexity, you can see the benefits
you can implement central logging system like:
The ELK stack (Elastic Search, Logstash and Kibana) for Centralized Logging.
We are currently evaluating the latest WSO2 BPS 3.0 as an open source replacement for Oracle BPEL. So far I was able to create and deploy a workflow on the BPS server. I was also able to test it and everything seem to work fine.
The problem however is looking at the EXECUTION TRACE like we can on the Oracle BPEL console.
I succesfuuly enabled SOAP TRACING only to see SOAP messages incoming and outgoing from the BPEL process. I however would like to see the output at each interim step of the workflow. Oracle does a wonderful job wherein I can just click on individual steps in the execution trace and view the output after each step. This is a very very important functionality and am surprised is not enabled OUT OF THE BOX.
I also tried the steps at BPEL Designer for Eclipse: how to debug a BPEL process but still cannot get it to work.
Can somebody list the exact steps so I can visualize the output of every step in the workflow.
You can find the input and output messages corresponding to each activity from the instance view. From the management console, go-to instance view and click on a given process instance id. Then you would get to the instance view.
Also by enabling soap tracer from the management console, you can view the incoming and outgoing soap messages.
Additionally, you can enabling message tracing at log level to log all messages coming in and going out.
I have this question, i am just throwing it out there. I am implementing a small logging feature for my spring based REST API server for logging all requests coming in.
I am expecting 1000s of users to use this API so with a blocking i/o logger, it's going to slow down everything. I have two approaches to solve the problem:
1. Have a async logger using an in-memory arrylist. then use the spring scheduler to flush this out to a log file periodically.
2. Use JMS and send the logs to the queue. Let the queue handle the logging asynchronously.
Has anyone done this before with spring. Though i am for option 2, are there better ways of doing this? Need some expert advice. Thanks everyone !
More info - I think synchronous logging will be a bottle neck because this REST API is consumed by a front end RoR app. So one session of the user will definitely result in 100s of API calls occuring very frequently. I am logging the actual request along with the JSON sent in the POSTs.
Has anyone done this before with spring.
Not so strangely, yes - Asynchronous Logging Using Spring
The article mentions that if you don't want any log events to be lost, JMS would be the way to go - otherwise sticking to Async makes sense for high volume logging.
If you really want to build your own logger, I suggest you to take a look at akka, it is much easier than JMS to set up.
You can use it locally (use of all CPU cores of your local machine), or even with remote agents.