We are currently evaluating the latest WSO2 BPS 3.0 as an open source replacement for Oracle BPEL. So far I was able to create and deploy a workflow on the BPS server. I was also able to test it and everything seem to work fine.
The problem however is looking at the EXECUTION TRACE like we can on the Oracle BPEL console.
I succesfuuly enabled SOAP TRACING only to see SOAP messages incoming and outgoing from the BPEL process. I however would like to see the output at each interim step of the workflow. Oracle does a wonderful job wherein I can just click on individual steps in the execution trace and view the output after each step. This is a very very important functionality and am surprised is not enabled OUT OF THE BOX.
I also tried the steps at BPEL Designer for Eclipse: how to debug a BPEL process but still cannot get it to work.
Can somebody list the exact steps so I can visualize the output of every step in the workflow.
You can find the input and output messages corresponding to each activity from the instance view. From the management console, go-to instance view and click on a given process instance id. Then you would get to the instance view.
Also by enabling soap tracer from the management console, you can view the incoming and outgoing soap messages.
Additionally, you can enabling message tracing at log level to log all messages coming in and going out.
Related
I have a sink application that fails to write to db, but am having trouble debugging. Note that I also asked a more specific question here, but this question in SO is more general: How should I go about debuggging an SCDF stream pipeline when no errors come up?
What I'm trying to do
I am trying to follow a tutorial (specifically, this tutorial) which uses some prebuilt applications. Everything is up and running with no error messages, and the source application is correctly writing to Kafka. However, the sink seems to be failing to write anything.
Note that I do see the debugging guide here:
https://dataflow.spring.io/docs/stream-developer-guides/troubleshooting/debugging-stream-apps/#sinks
However, this seems to only be relevant when you are writing your own sink.
I am not asking about how to solve my issue per se, but rather about debugging protocol for SCDF apps in general. What is the best way to go about debugging in these kinds of situations where no errors come up but the core functionality isn't working?
Assuming you know how to view the logs and there are no error messages, the next step is to turn on DEBUG logging for spring-integration. You can set a property on the sink logging.level.org.springframework.integration=DEBUG that will log any messages coming into the sink.
I'm trying out the .Net agent in Elastic APM and I'm using a C# application which is created using a framework called ASP.net Boilerplate. I've added the core libraries as mentioned in the documentation and added the settings in appsettings.json. This enables the default instrumentation and I got traces in the APM visualized through Kibana.
Currently I've got a node.js application running and I publish a message to a RabbitMQ queue with the traceparent in the message payload. The C# app reads the published message. I need to create a transaction or span using this traceparent / trace id so that Kibana would show the trace among the distributed systems.
I want to know if there is a way to create a transaction (or span) using a traceparent that is being sent from another system not using a HTTP protocol. I've checked the Elastic APM agent documentation -> Public API for information but couldnt find any information on this. Is there a way? Thanks.
I want to know if there is a way to create a transaction (or span) using a traceparent that is being sent from another system not using a HTTP protocol.
Yes, this is possible and there is an API for it. This part of the documentation explains it.
So you'll need to do this when you start your transaction - I imagine in your scenario this will be when you read a message from RabbitMQ.
When you start the transaction there is an optional parameter called distributedTracingData - if you pass it, then the transaction will reuse the traceid which you passed through RabbitMQ, and this way the new transaction will be part of the whole trace. If you don't pass this parameter, a new traceid will be generated and a new trace will be started.
Another comment that may help: you pass the trace id into the method where you start the transaction and each span will inherit this trace id within a transaction - so you control this on the transaction level and accordingly you don't pass it into a span.
Here is a small code snippet on how this would look:
serializedDistributedTracingData = //read this from the message which you get RabbitMq
var transaction2 = Agent.Tracer.StartTransaction("RadFromQueue", "RabbitMQRead",
DistributedTracingData.TryDeserializeFromString(serializedDistributedTracingData));
#gregkalapos, again thank you for the information. I checked how to acquire the neccessary trace information as in node.js agent documentation and when I debugged noticed that it was the trace id. Next in the C# consumer end I placed a code snippet as mentioned in the .Net agent and gave it a run. Kibana displayed the transactions from two different services in a single trace as I hoped it would.
We have requirements for getting data from a SOAP web service, where same records are going to be exposed. Then the record is transformed and written do the DB.
We are the acitve side and at the certain intervals we are going to check if a new record has appeared.
Our main goal are:
to have a scheduler for setting intervals
to have a mechanizm to retry if something goes wrong (eg. lost connection)
to have a visual control of the process - check the places where something stuck (like dashboard in SCDF)
Since there is no sample wsdl source app, I guess the Task (or Stream ?) should be written by ourself. But what to use for repeating and scheduling...
I Need your advice in choosing the right approach.
I'm not tied to the SCDF solution if any other are more suitable.
If you intend to consume directly as SOAP messages from external services, you could either build a custom Spring Cloud Stream source or a simple Spring Batch/Spring Cloud Task application. Both the options provide the resiliency patterns, including retries.
However, if the upstream data is not real-time, you would choose the Task path because the streams are long-running and they never terminate. Tasks, on the other hand, run for a finite period of time, terminate, and free-up resources. There's also the option to use the platform-specific scheduler implementation to trigger to launch the Task on a recurring window periodically.
From the SCDF dashboard, you can design/build Composed Tasks, including the state transitions and the desired downstream operation.
We have multiple team nifi applications running in same nifi machine... Is there any way to log the logs specific to my application? Also by default nifi-app.log file is difficult to track the issues and bulletin board shows the error msg for only 5 mins... How to get the errors captured and send an mail alert in Nifi?
Please help me to get through this. Thanks in advance!
There are a couple ways to approach this. One is to route failure relationships from processors to a PutEmail processor which can send an alert on errors. Another is to use a custom reporting task to alert a monitoring service when a certain number of flowfiles are in an error queue.
Finally, we have heard that in multitenant environments, log parsing is difficult. While NiFi aims to reduce or completely eliminate the need to visually inspect logs by providing the data provenance feature, in the event you do need to inspect the logs, we recommend searching the log by processor ID to isolate relevant messages. You can also use NiFi itself to ingest those same logs and perform parsing and filtering activities if desired. Future versions may improve this experience.
By parsing the nifi log, you can separate the logs which is specific to your team applications, by using the processor group id and using Nifi Rest API. Check the below link for the nifi template and python codes to solve this issue:
https://link.medium.com/L6IY1wTimV
You can send all the errors in a Processor Group to the same processor, it could be a regular UpdateAttribute or a custom processor, this processor is going to add the path and all the relevant information and then send this to a general error/logs flow that is going to check the information inside the flowfile regarding to the error, and will make the decision of send or not an email, to whom and this kind of things.
Using this approach, the system keeps simple and inside NiFi, so you don't add more layers of complexity, and you are going to have only one processor to manage the errors per Process Group.
This is the way we are managing errors in my company.
I do not have much experiences with Oracle Service Bus, I am trying to design a logging solution with BigData.
As I read, the default log and report activity in OSB will put the data into the domain's server log file or into the database where we setup the server domain. If I want to put all the logs into a separate BigData database. I will need to either of these approaches:
Java callout, use JMS or some other technology to send data to the bigdata server.
Web service callout, create a separate web service to handle the logging.
Create custom report provider to replace the default one in OSB Reporting.
Something else
Please tell give me some ideas about what method I should be using, and please provide your reasons if you can, thank you so much.
Isn't the logging framework in weblogic based on Log4j? That means you can use a JMSAppender (probably prudent to wrap in an Async log4j appender if you can) and handle it however you want.
Or, if you're talking about the OSB Reporting framework, there's a few options:
Configure the default JMS reporting provider (which uses the underlying SOAINFRA database which hopefully is set up to be something better than the default Derby instance), then write a MDB that pulls reports off the queue and inserts it into SAS BigData
Turn the JMS provider off and use a custom provider, which can do anything you want. If you want, you can still do a two-step process, where the reporting provider itself puts reports on a JMS queue so it returns quickly, and a different MDB pulls messages off and persists them at its own pace.
I do not recommend a web service or database callout without an async step in the middle, because you need logging and reporting to be very quick and use as little resources for as short a period as possible.
You don't want logging to hog threads while you're experiencing load. I have seen entire buses brought down because of one hiccup, because the logging database suffered a performance blip, which caused a bunch of open threads trying to log to it, which caused thread starvation or timeouts, which caused more error logging...
if you have a buffer like a JMS queue, then you can handle peaks by planning ahead. You can say "actually I want a JMS queue of 10,000 messages, and if that overflows due to whatever reason, I want to (push the overflow to a separate queue over on this other box) or (filter out all the non-essential messages) or (throw new messages away) or (action of your choice). Oh yeah, and if the logging database fails then I will try 3 times to commit and if not, move it to this other queue". Or whatever you want.
There are multiple ways to achieve this. You could use the report activity to push to JMS or use the log activity.
You can also write a small routine such as this (either on OSB or outside it), that can read anything that you are logging (such as via the log activity but also additional metadata that is logged when you turn on monitoring of OSB components) and do with it whatever is needed (such as pushing it to a database or BigData store).
The key is to avoid writing an explicit service call in each pipeline/flow and the above approach(es) use standard OSB/ODL* loggers
*Oracle Diagnostic Logging