JSON ordering and logging response/request body in envoy - proxy

I am using envoy proxy in my application and I am trying to print logs in these three ways:
In fixed ORDERING JSON format
Need to add request and response body of the request to log
Can we add a route level logging (enable/disabling), not with Lua?
All below scenarios log level is added in listener filter in my application.
#1:
I am able to add stdout access logging and able to print logs in JSON format but not able to order them in a fixed format like first key should be response code and so on. Each time JSON log prints order of keys changes.
#2: In my logs how can I add my request and response body? I tried to find formats supported by the envoy but had no luck.
What extra parameters do I need to in json_format to get the body?
json_format": {"protocol": "%PROTOCOL%","duration": "%DURATION%","my_custom_header": "%REQ(MY_CUSTOM_HEADER)%"}
#3: Can we add route level logging in envoy and also enable/disable on route level? I read the documentation but was not able to find anything.
In Lua we can add like below on route level can we do without Lua/another plugin?
envoy.filters.http.lua:
"#type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.LuaPerRoute
# name: bye.lua
disabled : true

For #1, I have same question with you on #1. It's terrible that we can't control JSON order for the access log.
For #2, I don't think print request body is a good idea. You can try to log request body in your application and track envoy log and your application log with the same x-request-id.
For #3, I don't have further knowledge for lua filter. But enovy access log itself has many fields for UPSTEAM and DOWNSTREAM.
Hope you have already resolved the issue and share back : )

Related

PHPMailer DEBUG OUTPUT without Attachment data

I am using PHPMailer (v 6.6.0) and would like to record meaningful debug information. As noted in other threads, there appears to be no means to exclude message content/body and attachments from the Debug output. I've looked at the feasibility of using the $mail->Debugoutput = function($str,$level){} callback to filter the data, but don't see a way to identify the message content or attachment data from the header info that I'm really interested in - SMTP connection info and message header.
Does anyone have a solution or clue as-to how to parse out the message content (ideal) and attachment (MOST important) data from the Debug output?
My suggestion would be to remove the message and attachment content from the LEVEL 1 and 2 logging and provide them only as an option with another LEVEL.
Thanks in-advance.
-jmd

Which http status codes to use when processing http post?

I have a HTML form, which I submit via http post.
There are two cases:
Case 1: The data is valid and data on the server will be updated accordingly
Case 2: The data is invalid and the http response contains an error message for the user.
Which http status codes should be used for each case?
I use htmx to submit the form. This means I don't need to use the POST/Redirect/GET pattern.
This question is not about JSON-APIs.
The complete list of HTTP response codes published by the Mozilla Foundation is pretty comprehensive and easy-to-read, so I'd recommend always consulting it as a guide. For the generic use-cases mentioned by you, there are a couple of different codes you can return - depending on what happens with the data on the server, and what you want to happen in the user's browser.
CASE 1: data is valid, data on server is updated
Base on your short description, the different status codes that might be applicable are:
200 (OK): you are updating an existing record on your own server - eg., the user is submitting a form which updates their existing contact information on your website - and information was received, and the record updated successfully. The response would usually contain a copy of the updated record.
201 (Created): you are not updating an existing record, but rather, creating a new record on your server - eg., your user is adding a new phone number to their contact details, which is saved in your database as a separate 'phone' record. The response should contain a copy of the newly created record.
205 (Reset Content): the same as 200, but implies that the browser view needs to be refreshed. This is useful when the record that is being updated has values that are dynamically calculated by the server, and which might change automatically depending on the values you're submitting. For example, if you have a user adding extra information to their online profile, which might grant them special status, badges and privileges on the website. This means, that if the user is viewing their profile information, that information will need to be updated with the new 'status' automatically granted by the server. The 205 response body will normally be empty, which means that to update the browser view your response-handling code will need to:
do further ajax requests and update the relevant part(s) of your
interface with new information from the server, or
redirect the user to a new URL, or
reload the entire page.
If working with HTMX, a 200 or 201 response would include the actual html snippet of the record that you want updated on the page - and HTMX will replace it automatically for you when it receives the response. With a 205 response, you could send an HX-Trigger response header that would call a custom event on the interface elements that need to update themselves - see the examples in the docs.
CASE 2: data is invalid, data on server is not updated
The status code that needs to be returned in case of an error varies depending on what caused the error. Errors that the server believes are the responsibility of the client - such as 'sending invalid data' - have a status code in the 4XX range. Some of the common errors in that range include 404 ('Not Found'), 403 ('Forbidden'), and 'Unauthorised' (401).
In the case of a client sending data that the server cannot process because it is 'not valid' - either because the request itself is malformed, or because the data doesn't pass some business validation logic - the current advice is to return status 400 (Bad Request).
Many years ago, some people believed that the status code 400 should only be used to indicate a malformed request (syntactical error) - not to indicate a failure in business validation logic (semantic error). There was a lot of debate, and temporarily a new status code (422) was created, that was supposed to cover semantic errors, exclusively. In 2014, however, the official definition of the status 400 code was changed to allow for the inclusion of both syntactical and semantical errors - which rendered status 422 essentially unnecessary.
You can find lots of discussions and explanations online about the differences between 400 and 422, and some people still argue passionately about this to this day. In practice, however, the 400 code is all you'll need - and you can include a response body in it that explains in detail, if needed, the cause of the error.
Note that when working with HTMX, a response with a 400 code should trigger an htmx:responseError event automatically. You can trap that event, for example, to update your form interface elements, in case of data validation errors caught by the server.
Well, 200 OK and 201 Created are the best for successful result.
For invalid data I would return 422 Unprocessable Entity, because the headers are correct, but body is not (though parseable by the server). The caveat is some HTTP clients won't handle 422 properly and in this case you have to use 400 Bad Request, however, the most of the modern clients will be fine.
You have said it is not about JSON APIs, but how will you meet this type of requirement - it is not clear whether this is relevant for your scenario???
SERVER DRIVEN BEHAVIOUR
I cannot see how a client could ever decide an HTTP status code based on input data. How would the client deal with these examples?
The call is not authenticated (cookie or token) - an API would return 401 - this tells the UI to perform a retry action.
The call is not authorized - an API would return 403 or 404 and the UI would present an error display.
The data is malformed or invalid according to domain specific checks - an API would return 400 and tell the UI what is wrong so that it can perform actions.
Something went wrong in server processing, eg data cannot be saved because database is down.
MY THOUGHTS
htmx looks interesting but a key requirement before using it would be ensuring that htmx can read server side error responses and use values returned. Maybe there is an elegant way to do this ...
Maybe I am just paranoid :). But it is worth being careful when choosing technologies that there are no blocking issues. Lack of error handlng control would be a blocking issue in most systems.
I'm using htmx 1.8 with asp.net core 6.0.
This works for me.
controller:
//server side validation failed
Response.StatusCode = 422;
return PartialView("Core", product);
client side javascript:
document.body.addEventListener('htmx:beforeOnLoad', function (evt) {
if (evt.detail.xhr.status === 422) {
//
// allow 422 responses to swap as we are using this as a signal that
// a form was submitted with bad data and want to rerender with the
// error messages
//
evt.detail.shouldSwap = true;
evt.detail.isError = false;
}
});
200 OK or 201 Created are the best choice for a successful POST request.
However, for invald data, you can pass 415 Unsupported Media Type

Extracting response and using While Controller to wait for response in Jmeter

All, I need additional assistance. I have tried researching and was using this following resource guide. https://www.blazemeter.com/blog/using-while-controller-jmeter
Problem
I need to Implement a way to wait for a specific Response from the server and record the time it takes from the start of the request to the end where I get a response of Complete in Jmeter. I have been looking at while controller and researching it.
Problem#1
If I add a while controller, my request failed because its adding multiple token. If I leave it without the while controller then it only gets 1 bearer token and it is able to submit a get request and get the response appropriately. I am suspecting there is a pre-procession somewhere else in my script that doing this...
Uses beanshell preprocesser to grab the token
sampler.getHeaderManager().add(new Header("Authorization","Bearer " + vars.get("BEARER")));
Problem #2
I have a GET Request to check the status of the request. Response in the BODY comes back as this json
"{"Status": "RECEIVED", "DllUrl": "", "Message": " "}"
I need it to continue checking for the status of COMPLETE……..also may need to check for failed as well
Was looking at regular Expression extractor to get the status and only stop until it finds COMPLETE status. That way I can measure time start and end time for completion. Any help and guidance will be helpful.
You're adding a header and not removing the previous one therefore it adds another header on each iteration.
In fact you don't need to do any scripting for this, you can just define the header in the HTTP Header Manager:
With regards to extracting this Status value, it's better to do it using JSON Extractor configured like:
Once done you can use the following __jexl3() function to "wait" till the operation finishes:
${__jexl3("${status}" != "COMPLETE",)}

How to log performance for each node in a route in camel in the correct order of invocation and not in the order of completion?

I have a simple route like this
from("file:data/inbox?noop=true").transform().body().to("file:data/outbox").bean(UpdateInventory.class);
from("direct:update").to("file:data/anotherbox").to("direct:newupdate");
from("direct:newupdate").to("file:data/newbox");
And the output i am expecting is
-file://data/inbox
--transform[simple{body}] 15 ms
--file:data/outbox 5
---bean[com.classico.sample.UpdateInventory#3a469fea 19
---file:data/anotherbox 6
----direct:newupdate 5
-----file:data/newbox 4
I tried using a EventNotifier and when the ExchangeCompletedEvent is received i fetched the message History.But since the second exchange is completed first message history is showing up in the reverse order of invocation.Is it possible to store all the messgae histories in a collection and print them in reverse order or Is there any event that is suitable for this.?
if (event instanceof ExchangeCompletedEvent) {
ExchangeCompletedEvent exchangeCompletedEvent = (ExchangeCompletedEvent) event;
Exchange exchange = exchangeCompletedEvent.getExchange();
String routeId = exchange.getFromRouteId();
List<MessageHistory> list = exchange.getProperty(Exchange.MESSAGE_HISTORY, List.class);
for (MessageHistory history : list) {
String id = history.getNode().getId();
String label = URISupport.sanitizeUri(history.getNode().getLabel());
log.info(String.format(MESSAGE_HISTORY_OUTPUT, routeId, id, label, history.getElapsed()));
}
}
You can use JMX to get all that details for each processor.
There is also a dumpRouteStatsAsXml operation on each route / camelContext that can output a xml file of the route(s) with all performance stats.
We use this in the hawtio web console to list this kind of information.
http://hawt.io/
Also the Camel Karaf / Jolokia Commands uses this as well
https://github.com/apache/camel/blob/master/platforms/commands/commands-core/src/main/java/org/apache/camel/commands/ContextInfoCommand.java#L175
And in next release of Camel you can also easier get the various processor mbeans, from CamelContext if you know their id, using
https://github.com/apache/camel/blob/master/camel-core/src/main/java/org/apache/camel/CamelContext.java#L545
Then you can use the getters on the mbean to get the performance stats.
The event notifer which was suggested is also great, but the events are on a higher level, although you get an event for sending to an endpoint, such as to some external system, which often is enough to capture details about. For low level details as asked here, then you need to use the JMX stats.
Ohh I forgot to tell about the message history EIP which also has a trace of how the message was routed with time taken stats as well.
http://camel.apache.org/message-history.html
That is maybe also just what you need, then you can get that information from the exchange as shown on that link.
I would suggest using the camel EventNotifier. You can find documentation on how to use it here:
http://camel.apache.org/eventnotifier-to-log-details-about-all-sent-exchanges.html

No user defined transport headers in pulled messages from queue OSB JMS WLS

im facing a problem with my jms messeages.
Case is:
in my mainPS i give an employeeId on request which routes to BS to get employees with emploeeId >= than given. Response is employees collecion.
On response action in mainPS I loop through whole collection and id like to add to JMS queue one by one employee as simple message. In every loop iteration im seting user-defined transport header in if statement as msgName 'even' or 'odd' depends on employeeId on Outbound Request. On loged result i can see that headers have added properly.
Then (still for every employee) I publish my JMS-BS which adds message to queue.
When I want to pull my messages in JMSConsumerPS there are no any transport headers which i have added. Consumer has Get All Headers property set as 'YES'.
Im logging in consumer my whole
$inbound/ctx:transport
and what i get is:
<con:transport>
<con:uri>myURI</con:uri>
<con:mode>request</con:mode>
<con:qualityOfService>best-effort</con:qualityOfService>
<con:request xsi:type="jms:JmsRequestMetaData" xmlns:jms="http://www.bea.com/wli/sb/transports/jms" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<tran:headers xsi:type="jms:JmsRequestHeaders" xmlns:tran="http://www.bea.com/wli/sb/transports">
<jms:JMSDeliveryMode>2</jms:JMSDeliveryMode>
<jms:JMSExpiration>0</jms:JMSExpiration>
<jms:JMSMessageID>ID:<834866.1398327222060.0></jms:JMSMessageID>
<jms:JMSPriority>4</jms:JMSPriority>
<jms:JMSRedelivered>false</jms:JMSRedelivered>
<jms:JMSTimestamp>1398327222060</jms:JMSTimestamp>
<jms:JMSXDeliveryCount>1</jms:JMSXDeliveryCount>
</tran:headers>
<tran:encoding xmlns:tran="http://www.bea.com/wli/sb/transports">UTF-8</tran:encoding>
<jms:message-type>Text</jms:message-type>
</con:request>
Thanks for help.
Just a guess.
I was setting transport header in in-correct place. I had a "publish"
action in the proxy and I was setting transport header just before the
publish action. I moved the "transport header setting" to be done
inside the "publish" action/task . Now it works as expected.
Taken from https://community.oracle.com/thread/2155298?tstart=165, link most likely will be dead soon enough - Oracle community ...
Thank you very much for that. That was almost same issue. What I did was movig "transport header setting" into request action of published BS. Thing is i have tried it before asking but had no idea why it didnt work. Probably the reason could be that i turned on pass all headers through pipeline that time [?]. Had no idea if that can make such a mess. Thanks for replaying.

Resources