Spring Boot: getting query string parameters and request body in AuditApplicationEvent listener - spring-boot

Spring Boot REST app here. I'm trying to configure Spring Boot request auditing to log each and every HTTP request that any resources/controllers receive with the following info:
I need to see in the logs the exact HTTP URL (path) that was requested by the client, including the HTTP method and any query string parameters; and
If there is a request body (such as with a POST or PUT) I need to see the contents of that body in the logs as well
My best attempt so far:
#Component
public class MyAppAuditor {
private Logger logger;
#EventListener
public void handleAuditEvent(AuditApplicationEvent auditApplicationEvent) {
logger.info(auditApplicationEvent.auditEvent);
}
}
public class AuditingTraceRepository implements TraceRepository {
#Autowired
private ApplicationEventPublisher applicationEventPublisher
#Override
List<Trace> findAll() {
throw new UnsupportedOperationException("We don't expose trace information via /trace!");
}
#Override
void add(Map<String, Object> traceInfo) {
Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
AuditEvent traceRequestEvent = new AuditEvent(new Date(), "SomeUser", 'http.request.trace', traceInfo);
AuditApplicationEvent traceRequestAppEvent = new AuditApplicationEvent(traceRequestEvent);
applicationEventPublisher.publishEvent(traceRequestAppEvent);
}
}
However at runtime if I use the following curl command:
curl -i -H "Content-Type: application/json" -X GET 'http://localhost:9200/v1/data/profiles?continent=NA&country=US&isMale=0&height=1.5&dob=range('1980-01-01','1990-01-01')'
Then I only see the following log messages (where MyAppAuditor send audit events):
{ "timestamp" : "14:09:50.516", "thread" : "qtp1293252487-17", "level" : "INFO", "logger" : "com.myapp.ws.shared.auditing.MyAppAuditor", "message" : {"timestamp":"2018-06-29T18:09:50+0000","principal":"SomeUser","type":"http.request.trace","data":{"method":"GET","path":"/v1/data/profiles","headers":{"request":{"User-Agent":"curl/7.54.0","Host":"localhost:9200","Accept":"*/*","Content-Type":"application/json"},"response":{"X-Frame-Options":"DENY","Cache-Control":"no-cache, no-store, max-age=0, must-revalidate","X-Content-Type-Options":"nosniff","Pragma":"no-cache","Expires":"0","X-XSS-Protection":"1; mode=block","X-Application-Context":"application:9200","Date":"Fri, 29 Jun 2018 18:09:50 GMT","Content-Type":"application/json;charset=utf-8","status":"200"}},"timeTaken":"89"}} }
So as you can see, the auditor is picking up the base path (/v1/data/profiles) but is not logging any of the query string parameters. I also see a similar absence of request body info when I hit POST or PUT endpoints that do require a request body (JSON).
What do I need to do to configure these classes (or other Spring classes/configs) so that I get the level of request auditing that I'm looking for?

Fortunately, Actuator makes it very easy to configure those Trace events.
Adding parameters to Trace info
You can take a look at all of the options. You'll notice the defaults (line 42) are:
Include.REQUEST_HEADERS,
Include.RESPONSE_HEADERS,
Include.COOKIES,
Include.ERRORS,
Include.TIME_TAKEN
So you'll need to also add Include.PARAMETERS and anything else you'd like to have in the trace. To configure that, there's a configuration property for that management.trace.include.
So to get what you want (i.e. parameters), plus the defaults, you'd have:
management.trace.include = parameters, request-headers, response-headers, cookies, errors, time-taken
Adding request body to Trace info
In order to get the body, you're going to have to add in this Bean to your Context:
#Component
public class WebRequestTraceFilterWithPayload extends WebRequestTraceFilter {
public WebRequestTraceFilterWithPayload(TraceRepository repository, TraceProperties properties) {
super(repository, properties);
}
#Override
protected Map<String, Object> getTrace(HttpServletRequest request) {
Map<String, Object> trace = super.getTrace(request);
String body = null;
try {
body = request.getReader().lines().collect(Collectors.joining("\n"));
} catch (IOException e) {
throw new RuntimeException(e);
}
if(body != null) {
trace.put("body", body);
}
return trace;
}
}
The above code will override the AutoConfigure'd WebRequestTraceFilter bean (which, because it is #ConditionalOnMissingBean will give deference to your custom bean), and pull the extra payload property off of the request then add it to to the Map properties that get published to your TraceRepository!
Summary
Request Parameters can be added to TraceRepository trace events by via the management.trace.include property
The Request Body can be added to the TraceRepository trace events by creating an extended Bean to read the body off of the HTTP request and supplementing the trace events

Related

Feign ErrorDecoder is not invoked - how to configure feign to use it?

As i understand the decode() method of the feign ErrorDecoder will be called when a request responds with a status code != 2xx. Through debugging my tests i found out that the decode() method of my CustomErrorDecoder is not invoked on e.g. 504 or 404. I tried two ways to configure it:
Either include it as a Bean in the client configuration:
#Bean
public CustomErrorDecoder customErrorDecoder() {
return new CustomErrorDecoder();
}
or write it into the application configuration :
feign:
client:
config:
myCustomRestClientName:
retryer: com.a.b.some.package.CustomRetryer
errorDecoder: com.a.b.some.package.CustomErrorDecoder
Both ways don't invoke the ErrorDecoder. What am I doing wrong? The Bean is beeing instantiated and my CustomErrorDecoder looks like this:
#Component
public class CustomErrorDecoder implements ErrorDecoder {
private final ErrorDecoder defaultErrorDecoder = new Default();
#Override
public Exception decode(String s, Response response) {
Exception exception = defaultErrorDecoder.decode(s, response);
if (exception instanceof RetryableException) {
return exception;
}
if (response.status() == 504) {
// throwing new RetryableException to retry 504s
}
return exception;
}
}
Update:
I have created a minimal reproducible example in this git repo. Please look at the commit history to find 3 ways that I tried.
The problem is that your feign client uses feign.Response as the return type:
import feign.Param;
import feign.RequestLine;
import feign.Response;
public interface TestEngineRestClient {
#RequestLine(value = "GET /{uuid}")
Response getReport(#Param("uuid") String uuid);
}
In this case, Feign delegates its handling to the developer - e.g., you can retrieve HTTP status and a response body and do some stuff with it.
If interested, you can look at the source code of feign.SynchronousMethodHandler, executeAndDecode section.
To fix this, replace Response.class with the desired type in case of the correct response with status code = 2xx (probably some DTO class). I made a PR where I've changed it to String for simplicity.

how to write WebFlux filter which needs to pass Rest body for authorization service call

I need to implement an authorization filter for WebFlux spring-boot application. To invoke authorization service I need to get url, queryParam, headers and body.
I can get first 3 properties from ServerRequest of the HandlerFilterFunction, however I have problem with body. The authorization call requires all four parameters:
Principal authorization(String method,Sting requestUrl,Map<String, ListValue> headersMap, String body) throws ServiceException
The method throws exception if user is not authorized to execute the method otherwise returns Principal object.
The method is never called when filter is executed because I don't have any subscriber to mono object:
public class WebFluxAuthorizationFilter implements HandlerFilterFunction<ServerResponse, ServerResponse> {
#Override
public Mono<ServerResponse> filter(ServerRequest request, HandlerFunction<ServerResponse> handlerFunction) {
String method = request.methodName();
String requestUrl = request.uri().toString();
HttpHeaders headers = request.headers().asHttpHeaders();
Map<String, ListValue> headersMap = new HashMap<>();
for (Map.Entry<String, List<String>> entry : headers.entrySet()) {
ListValue grpcList = toListValue(entry.getValue());
headersMap.put(entry.getKey(), grpcList);
}
Mono<Principal> mono = request.bodyToMono(String.class)
.flatMap(body -> {
return Mono.just(authorization(method, requestUrl, headersMap, body));
});
return handlerFunction.handle(request);
}
I have the same issue if I use WebFilter instead of HandlerFilterFunction
What should I do to get access to body and run authorization method shown above ?
The current code does not invoke authorization() because lambda body -> ... is never called.
Please let me know how filter should be written to get the body and use during filter execution .

issue with Spring and asynchronous controller + HandlerInterceptor + IE/Edge

I am working on a Spring application that serves up REST endpoints. One of the endpoints essentially acts as a proxy between the HTML client and a third party cloud storage provider. This endpoint retrieves files from the storage provider and proxies them back to the client. Something like the following (note there is a synchronous and asynchronous version of the same endpoint):
#Controller
public class CloudStorageController {
...
#RequestMapping(value = "/fetch-image/{id}", method = RequestMethod.GET, produces = MediaType.IMAGE_JPEG_VALUE)
public ResponseEntity<byte[]> fetchImageSynchronous(#PathVariable final Long id) {
final byte[] imageFileContents = this.fetchImage(id);
return ResponseEntity.ok().body(imageFileContents);
}
#RequestMapping(value = "/fetch-image-async/{id}", method = RequestMethod.GET, produces = MediaType.IMAGE_JPEG_VALUE)
public Callable<ResponseEntity<byte[]>> fetchImageAsynchronous(#PathVariable final Long id) {
return () -> {
final byte[] imageFileContents = this.fetchImage(id);
return ResponseEntity.ok().body(imageFileContents);
};
}
private byte[] fetchImage(final long id) {
// fetch the file from cloud storage and return as byte array
...
}
...
}
Due to the nature of the client app (HTML5 + ajax) and how this endpoint is used, user authentication is supplied to this endpoint differently that the other endpoints. To handle this, a HandlerInterceptor was developed to deal with authentication for this endpoint:
#Component("cloudStorageAuthenticationInterceptor")
public class CloudStorageAuthenticationInterceptor extends HandlerInterceptorAdapter {
#Override
public boolean preHandle(final HttpServletRequest request, final HttpServletResponse response, final Object handler) {
// examine the request for the authentication information and verify it
final Authentication authenticated = ...
if (authenticated == null) {
try {
pResponse.sendError(HttpServletResponse.SC_UNAUTHORIZED);
} catch (IOException e) {
throw new RuntimeException(e);
}
return false;
}
else {
try {
request.login(authenticated.getName(), (String) authenticated.getCredentials());
} catch (final ServletException e) {
throw new BadCredentialsException("Bad credentials");
}
}
return true;
}
}
The interceptor is registered like this:
#Configuration
#EnableWebMvc
public class ApiConfig extends WebMvcConfigurerAdapter {
#Autowired
#Qualifier("cloudStorageAuthenticationInterceptor")
private HandlerInterceptor cloudStorageAuthenticationInterceptor;
#Override
public void addInterceptors(final InterceptorRegistry registry) {
registry.addInterceptor(this.cloudStorageAuthenticationInterceptor)
.addPathPatterns(
"/fetch-image/**",
"/fetch-image-async/**"
);
}
#Override
public void configureAsyncSupport(final AsyncSupportConfigurer configurer) {
final ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(this.asyncThreadPoolCoreSize);
executor.setMaxPoolSize(this.asyncThreadPoolMaxSize);
executor.setQueueCapacity(this.asyncThreadPoolQueueCapacity);
executor.setThreadNamePrefix(this.asyncThreadPoolPrefix);
executor.initialize();
configurer.setTaskExecutor(executor);
super.configureAsyncSupport(configurer);
}
}
Ideally, the image fetching would be done asynchronously (using the /fetch-image-asyc/{id} endpoint) because it has to call a third party web service which could have some latency.
The synchronous endpoint (/fetch-image/{id}) works correctly for all browsers. However, if using the asynchronous endpoint (/fetch-image-async/{id}), Chrome and Firefox work as expect.
However, if the client is Microsoft IE or Microsoft Edge, we seem some strange behavior. The endpoint is called correctly and the response sent successfully (at least from the server's viewpoint). However, it seems that the browser is waiting for something additional. In the IE/Edge DevTools window, the network request for the image shows as pending for 30 seconds, then seems to timeout, updates to successful and the image is successfully display. It also seems the connection to the server is still open, as the server side resources like database connections are not released. In the other browsers, the async response is received and processed in a second or less.
If I remove the HandlerInterceptor and just hard-wire some credentials for debugging, the behavior goes away. So this seems to have something to with the interaction between the HandlerInterceptor and the asynchronous controller method, and is only exhibited for some clients.
Anyone have a suggestion on why the semantics of IE/Edge are causing this behavior?
Based on your description, there are some different behaviors when using IE or Edge
it seems that the browser is waiting for something additional
the connection seems still open
it works fine if remove HandlerInterceptor and use hard code in auth logic
For the first behavior, I would suggest you use fiddler to trace all http requests. It is better if you could compare two different actions via fiddler (1) run on chrome, 2) run on edge ). Check all http headers in requests and responses carefully to see whether there is some different part. For the other behaviors, I would suggest you write logs to find which part spend the most time. It will provide you useful information to troubleshot.
After much tracing on the server and reading through the JavaDocs comments for AsyncHandlerInterceptor, I was able to resolve the issue. For requests to asynchronous controller methods, the preHandle method of any interceptor is called twice. It is called before the request is handed off to the servlet handling the request and again after the servlet has handled the request. In my case, the interceptor was attempting to authenticate the request for both scenarios (pre and post request handling). The application's authentication provider checks credentials in a database. For some reason if the client is IE or Edge, the authentication provider was unable to get a database connection when called from preHandle in the interceptor after the servlet handled the request. The following exception would be thrown:
ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Could not open connection; nested exception is org.hibernate.exception.JDBCConnectionException: Could not open connection] with root cause
java.sql.SQLTransientConnectionException: HikariPool-0 - Connection is not available, request timed out after 30001ms.
So the servlet would successfully handle the request and send a response, but the filter would get hung up for 30 seconds waiting for the database connection to timeout on the post processing called to preHandle.
So for me, the simple solution was to add a check in preHandle if it is being called after the servlet has already handled the request. I updated the preHandle method as follows:
#Override
public boolean preHandle(final HttpServletRequest pRequest, final HttpServletResponse pResponse, final Object pHandler) {
if (pRequest.getDispatcherType().equals(DispatcherType.REQUEST)) {
... perform authentication ...
}
return true;
}
That solved the issue for me. It doesn't explain everything (i.e., why only IE/Edge would cause the issue), but it seems that preHandle should only do work before the servlet handles the request anyways.

Why this externa web service call go into error only when the call is performed using Spring RestTemplate?

I am working on a Spring project implementing a simple console application that have to call an external REST web service passing to it a parameter and obtaining a response from it.
The call to this webservice is:
http://5.249.148.180:8280/GLIS_Registration/6
where 6 is the specified ID. If you open this address in the browser (or by cURL tool) you will obtain the expected error message:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<sampleid>IRGC 100000</sampleid>
<genus>Oryza</genus>
<error>PGRFA sampleid [IRGC 100000], genus [Oryza] already registered for this owner</error>
</response>
This error message is the expected response for this request and I correctly obtain it also using cURL tool to perform the request.
So I have to perform this GET request from my Spring application.
To do it I create this getResponse() method into a RestClient class:
#Service
#Scope(proxyMode = ScopedProxyMode.TARGET_CLASS)
public class RestClient {
RestTemplate restTemplate;
String uriResourceRegistrationApi;
public RestClient() {
super();
restTemplate = new RestTemplate();
uriResourceRegistrationApi = "http://5.249.148.180:8280/GLIS_Registration/7";
}
public ResponseEntity<String> getResponse() {
ResponseEntity<String> response = restTemplate.getForEntity(uriResourceRegistrationApi, String.class);
return response;
}
}
Then I call this method from this test method:
#Test
public void singleResourceRestTest() {
System.out.println("singleResourceRestTest() START");
ResponseEntity<String> result = restClient.getResponse();
System.out.println("singleResourceRestTest() END");
}
But I am experiencing a very strange behavior, what it happens is:
1)The call to my external web service seems that happens (I saw it from the web services log).
2) The web service retrieve the parameter having value 7 but then it seems that can't use it as done without problem performing the request from the browser or by the shell statment:
curl -v http://5.249.148.180:8280/GLIS_Registration/7
But now, calling in this way, my webservice (I can't post the code because it is a WSO2 ESB flow) give me this error message:
<200 OK,<?xml version="1.0" encoding="UTF-8"?>
<response>
<error>Location information not correct</error>
<error>At least one between <genus> and <cropname> is required</error>
<error>Sample ID is required</error>
<error>Date is required</error>
<error>Creation method is required</error>
</response>,{Vary=[Accept-Encoding], Content-Type=[text/html; charset=UTF-8], Date=[Fri, 05 May 2017 14:07:09 GMT], Transfer-Encoding=[chunked], Connection=[keep-alive]}>
Looking the web service log it seems that performing the call using RestTemplate it have some problem to use the retrieved ID=7 to perform a database query.
I know it looks terribly strange and you can see: "The problem is of your web service and not of the Spring RestTemplate". This is only partially true because I implemented this custom method that perform a low level Http GET call, this callWsOldStyle() (putted into the previous RestClient class):
public void callWsOldStyle() {
HttpURLConnection connection = null;
BufferedReader reader = null;
try {
URL restAPIUrl = new URL("http://5.249.148.180:8280/GLIS_Registration/7");
connection = (HttpURLConnection) restAPIUrl.openConnection();
connection.setRequestMethod("GET");
// Read the response
reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuilder jsonData = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
jsonData.append(line);
}
System.out.println(jsonData.toString());
}catch(Exception e) {
e.printStackTrace();
}
finally {
// Clean up
IOUtils.closeQuietly(reader);
if(connection != null)
connection.disconnect();
}
}
Using this method instead the RestTemplate one it works fine and this line:
System.out.println(jsonData.toString());
print the expected result:
<?xml version="1.0" encoding="UTF-8"?><response><sampleid>IRGC 100005</sampleid><genus>Oryza</genus><error>PGRFA sampleid [IRGC 100005], genus [Oryza] already registered for this owner</error></response>
To summarize:
Calling my WS from the browser it works.
Calling my WS using cURL it works.
Calling my WS using my callWsOldStyle() method it works.
Calling my WS using the method that use RestTemplate it go into error when my WS receive and try to handle the request.
So, what can be the cause of this issue? What am I missing? Maybe can depend by some wrong header or something like this?
As Pete said you are receiving an internal server error (status code 500) so you should check the server side of this rest service.
In any case you can do the following for the resttemplate
create an org.springframework.web.client.RequestCallback object if
you need to do something in the request
create an org.springframework.web.client.ResponseExtractor<String>
object in order to extract your data
use the resttemplate
org.springframework.web.client.RequestCallback
public class SampleRequestCallBack implements RequestCallback
{
#Override
public void doWithRequest(ClientHttpRequest request) throws IOException
{
}
}
org.springframework.web.client.ResponseExtractor
public class CustomResponseExtractor implements ResponseExtractor<String>
{
private static final Logger logger = LoggerFactory.getLogger(CustomResponseExtractor.class.getName());
#Override
public String extractData(ClientHttpResponse response) throws IOException
{
try
{
String result = org.apache.commons.io.IOUtils.toString(response.getBody(), Charset.forName("UTF8"));
if( logger.isInfoEnabled() )
{
logger.info("Response received.\nStatus code: {}\n Result: {}",response.getStatusCode().value(), result);
}
return result;
}
catch (Exception e)
{
throw new IOException(e);
}
}
}
REST TEMPLATE CALL
#Test
public void testStack()
{
try
{
String url = "http://5.249.148.180:8280/GLIS_Registration/6";
String response = restTemplate.execute(url, HttpMethod.GET, new SampleRequestCallBack(), new CustomResponseExtractor());;
logger.info(response);
}
catch (Exception e)
{
logger.error("Errore", e);
}
}
Angelo

Response MIME type for Spring Boot actuator endpoints

I have updated a Spring Boot application from 1.4.x to 1.5.1 and the Spring Actuator endpoints return a different MIME type now:
For example, /health is now application/vnd.spring-boot.actuator.v1+json instead simply application/json.
How can I change this back?
The endpoints return a content type that honours what the client's request says it can accept. You will get an application/json response if the client send an Accept header that asks for it:
Accept: application/json
In response to the comment of https://stackoverflow.com/users/2952093/kap (my reputation is to low to create a comment): when using Firefox to check endpoints that return JSON I use the Add-on JSONView. In the settings there is an option to specify alternate JSON content types, just add application/vnd.spring-boot.actuator.v1+jsonand you'll see the returned JSON in pretty print inside your browser.
As you noticed the content type for actuators have changed in 1.5.x.
If you in put "application/json" in the "Accept:" header you should get the usual content-type.
But if you don't have any way of modifying the clients, this snippet returns health (without details) and original content-type (the 1.4.x way).
#RestController
#RequestMapping(value = "/health", produces = MediaType.APPLICATION_JSON_VALUE)
public class HealthController {
#Inject
HealthEndpoint healthEndpoint;
#RequestMapping(method = RequestMethod.GET)
public ResponseEntity<Health > health() throws IOException {
Health health = healthEndpoint.health();
Health nonSensitiveHealthResult = Health.status(health.getStatus()).build();
if (health.getStatus().equals(Status.UP)) {
return ResponseEntity.status(HttpStatus.OK).body(nonSensitiveHealthResult);
} else {
return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(nonSensitiveHealthResult);
}
}
}
Configuration (move away existing health)
endpoints.health.path: internal/health
Based on the code in https://github.com/spring-projects/spring-boot/issues/2449 (which also works fine but completely removes the new type) I came up with
#Component
public class ActuatorCustomizer implements EndpointHandlerMappingCustomizer {
static class Fix extends HandlerInterceptorAdapter {
#Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler)
throws Exception {
Object attribute = request.getAttribute(HandlerMapping.PRODUCIBLE_MEDIA_TYPES_ATTRIBUTE);
if (attribute instanceof LinkedHashSet) {
#SuppressWarnings("unchecked")
LinkedHashSet<MediaType> lhs = (LinkedHashSet<MediaType>) attribute;
if (lhs.remove(ActuatorMediaTypes.APPLICATION_ACTUATOR_V1_JSON)) {
lhs.add(ActuatorMediaTypes.APPLICATION_ACTUATOR_V1_JSON);
}
}
return true;
}
}
#Override
public void customize(EndpointHandlerMapping mapping) {
mapping.setInterceptors(new Object[] {new Fix()});
}
}
which puts the new vendor-mediatype last so that it will use application/json for all actuator endpoints when nothing is specified.
Tested with spring-boot 1.5.3
Since SpringBoot 2.0.x the suggested solution in implementing the EndpointHandlerMappingCustomizer doesn't work any longer.
The good news is, the solution is simpler now.
The Bean EndpointMediaTypes needs to be provided. It is provided by the SpringBoot class WebEndpointAutoConfiguration by default.
Providing your own could look like this:
#Configuration
public class ActuatorEndpointConfig {
private static final List<String> MEDIA_TYPES = Arrays
.asList("application/json", ActuatorMediaType.V2_JSON);
#Bean
public EndpointMediaTypes endpointMediaTypes() {
return new EndpointMediaTypes(MEDIA_TYPES, MEDIA_TYPES);
}
}
To support application/vnd.spring-boot.actuator.v1+json in Firefox's built in JSON viewer, you can install this addon: json-content-type-override. It will convert content types that contain "json" to "application/json".
Update: Firefox 58+ has built-in support for these mime types, and no addon is needed anymore. See https://bugzilla.mozilla.org/show_bug.cgi?id=1388335

Resources