Spring Boot: CRLF - Securely log payload in REST API - spring-boot

I have a Spring Boot app which exposes a REST API. I need to log the payload to be able to find errors in the JSON in the API calls.
I have ran a code analysis tools that reports the following security risk when I log the payload.
https://find-sec-bugs.github.io/bugs.htm#CRLF_INJECTION_LOGS
How can I protect against code injection? I guess removing new lines only protect against fake log entries and will not protect against code injection?
REST API:
#PostMapping("/my/api")
public ResponseEntity<String> handleApi(#RequestBody Body body) {
Payload logging:
#Slf4j
public class CustomRequestLoggingFilter extends AbstractRequestLoggingFilter {
private static final int MAX_PAYLOAD_LENGTH = 64000;
public CustomRequestLoggingFilter() {
this.setIncludeQueryString(true);
this.setIncludePayload(true);
this.setMaxPayloadLength(MAX_PAYLOAD_LENGTH);
}
#Override
public void afterRequest(HttpServletRequest request, String message) {
if (request.getRequestURI().equals("/my/api")) {
log.info(message); //This is the security risk
}
}

You can try to use OWASP Json Sanitizer library (https://owasp.org/www-project-json-sanitizer/migrated_content) to clean and sanitize Json input prior logging it. If you are not concerned about adding additional 3rd party dependency to your project.
NOTE: Last release of this library was in Jan 11, 2021
Example:
#Override
public void afterRequest(HttpServletRequest request, String message) {
if (request.getRequestURI().equals("/my/api")) {
String sanitizedJson = JsonSanitizer.sanitize(message);
log.info(sanitizedJson );
}
}

The linked report is suggesting a possible solution of replacing newlines to remove the risk:
log.info(message.replaceAll("[\r\n]",""));
You can manually sanitize each parameter.
log.info("User " + val.replaceAll("[\r\n]","") + " (" + userAgent.replaceAll("[\r\n]","") + ") was not authenticated");
Or using other solutions which change your logging configuration:
You can also configure your logger service to replace new line for all message events. Here is sample configuration for LogBack using the replace function.
<pattern>%-5level - %replace(%msg){'[\r\n]', ''}%n</pattern>
Finally, you can use a logger implementation that replace new line by spaces. The project OWASP Security Logging has an implementation for Logback and Log4j.

The vulnerability you are mentioning have nothing to do with code injection, only with the possibility of manipulating your logs.
Remediation for that, if you are producing plain text logs, is to sanitize that message (best with the OWASP library that Dmitriy suggested), but if you are managing your logs with some tool (e.g. ELK), probably you should produce logs in JSON format and that would automatically mitigate this issue for you.
Back to the code injection, considering you have set some max payload length, I don't think you can have any code injection in that log statement.

Related

Spring #ControllerAdvice/ #ExceptionHandler not working

My Spring Boot Service will do a job and exit after success with 0 (there is no restcontroller), but i want it aslo to exit on every exception so i added #ControllerAdvice on a class and put this method:
#ControllerAdvice
#RequiredArgsConstructor
#Slf4j
public class ImportInekData {
final InekService inekService;
final ImportDataService dataService;
public void doTheJob(){
log.info("Fetching new list from Inek.");
UpdatedList updatedList = inekService.getUpdatedList();
List<Standort> toBeUpdated = updatedList.getToBeUpdated();
List<String> toBeDeleted = updatedList.getToBeDeleted();
log.info("List fetched with " + toBeUpdated.size() + " valid entries to be updated and " + toBeDeleted.size() + " entries marked for deletion. ");
log.info("Pushing to DB...");
dataService.importAll(toBeUpdated);
}
#EventListener
public void onStart(ContextStartedEvent start){
log.info("Application started.");
doTheJob();
log.info("Import finished.");
SpringApplication.exit(start.getApplicationContext(), () -> 0);
}
#ExceptionHandler(value = Exception.class)
public String outOnException(Exception e){
log.error("Exception occurred see logs. Stopping..");
SpringApplication.exit(context, () -> -1);
return "dying";
}
}
All is working fine but when i throw an IllegalArgumentException the #ExceptionHandler method is not called. First i had a void method with no parameter and then i began trying with String return and at least one parameter - that is not needed.
How get this working? Is there a better way for my case to react on every exception?
Controller Advices in spring is a mechanism intended to properly handle the Exceptions at the level of spring MVC.
Spring MVC in a nutshell is a web framework, and as such, it assumes that you have some kind of web endpoint that is called by the end user or maybe frontend. This endpoint is an "entry-point" to your backend code that can have services, query the database, and so forth. If during this backend flow the exception is thrown in general you don't want that the web endpoint will return 500 internal server error, so spring provides tooling for convenient mapping of these exceptions: translating them to json with a "good-looking" message, with correct HTTP code, and so forth.
If you don't have any controllers, then the whole concept of controller advices is not applicable in your flow, so there is no point in using it...
Now the real question is what exactly do you want to achieve with this exception handling?
If Application context cannot start usually spring boot application will be closed gracefully...
If you want to close the application programmatically, make sure you've read this thread

Create TCP client to send and receive in spring boot application

I am supposed to interact with a legacy system where I have to setup a TCP client using spring-integration in java/kotlin to send a message to a TCP system and receive its response, parse this response and send it to other client via REST. I went through many documentations and blogs which tells how to do via xml. Not able to find corresponding annotations for everything. Any code snippet will be very helpful.
#Service
class MyService{
#Autowired
MyGateway gateway;
public String callTCPClient(String msg){
return gateway.exchange(msg);
}
}
interface MyGateway{
String exchange (String msg)
}
As is shown in that answer, you can do whatever you want after the response is received...
#Bean
public IntegrationFlow client() {
return IntegrationFlows.from(MyGateway.class)
.handle(Tcp.outboundGateway(
Tcp.netClient("localhost", 1234)
.serializer(codec()) // default is CRLF
.deserializer(codec()))) // default is CRLF
.transform(Transformers.objectToString()) // byte[] -> String
.get();
}
In this case, we simply transform the byte array to a String, but you can perform whatever operations you want on it, e.g. JSON to Object. You can add as many steps as you want - add .handle(...) elements to call arbitrary methods. Read the Spring Integration Reference Manual.

Create a Custom Spring Cloud Netflix Ribbon Client

I am using Spring Cloud Netflix Ribbon in combination with Eureka in a Cloud Foundry environment.
The use case I am trying to implement is the following:
I have a running CF application named address-service with several instances spawned.
The instances are registering to Eureka by the service name address-service
I have added custom metadata to service instances using
eureka.instance.metadata-map.applicationId: ${vcap.application.application_id}
I want to use the information in Eureka's InstanceInfo (in particular the metadata and how many service instances are available) for setting a CF HTTP header "X-CF-APP-INSTANCE" as described here.
The idea is to send a Header like "X-CF-APP-INSTANCE":"appIdFromMetadata:instanceIndexCalculatedFromNoOfServiceInstances" and thus "overrule" CF's Go-Router when it comes to load balancing as described at the bottom of this issue.
I believe to set headers, I need to create a custom RibbonClient implementation - i.e. in plain Netflix terms a subclass of AbstractLoadBalancerAwareClient as described here - and override the execute() methods.
However, this does not work, as Spring Cloud Netflix Ribbon won't read the class name of my CustomRibbonClient from application.yml. It also seems Spring Cloud Netflix wraps quite a bit of classes around the plain Netflix stuff.
I tried implementing a subclass of RetryableRibbonLoadBalancingHttpClient and RibbonLoadBalancingHttpClient which are Spring classes. I tried giving their class names in application.yml using ribbon.ClientClassName but that does not work. I tried to override beans defined in Spring Cloud's HttpClientRibbonConfiguration but I cannot get it to work.
So I have two questions:
is my assumption correct that I need to create a custom Ribbon Client and that the beans defined here and here won't do the trick?
How to do it properly?
Any ideas are greatly appreciated, so thanks in advance!
Update-1
I have dug into this some more and found RibbonAutoConfiguration.
This creates a SpringClientFactory which provides a getClient() method that is only used in RibbonClientHttpRequestFactory (also declared in RibbonAutoConfiguration).
Unfortunately, RibbonClientHttpRequestFactory hard-codes the client to Netflix RestClient. And it does not seem possible to override either SpringClientFactory nor RibbonClientHttpRequestFactory beans.
I wonder if this is possible at all.
Ok, I'll answer this question myself, in case someone else may need that in the future.
Actually, I finally managed to implement it.
TLDR - the solution is here: https://github.com/TheFonz2017/Spring-Cloud-Netflix-Ribbon-CF-Routing
The solution:
Allows to use Ribbon on Cloud Foundry, overriding Go-Router's load balancing.
Adds a custom routing header to Ribbon load balancing requests (including retries) to instruct CF's Go-Router to route requests to the service instance selected by Ribbon (rather than by its own load balancer).
Shows how to intercept load balancing requests
The key to understanding this, is that Spring Cloud has its own LoadBalancer framework, for which Ribbon is just one possible implementation. It is also important to understand, that Ribbon is only used as a load balancer not as an HTTP client. In other words, Ribbon's ILoadBalancer instance is only used to select the service instance from the server list. Requests to the selected server instances are done by an implementation of Spring Cloud's AbstractLoadBalancingClient. When using Ribbon, these are sub-classes of RibbonLoadBalancingHttpClient and RetryableRibbonLoadBalancingHttpClient.
So, my initial approach to add an HTTP header to the requests sent by Ribbon's HTTP client did not succeed, since Ribbon's HTTP / Rest client is actually not used by Spring Cloud at all.
The solution is to implement a Spring Cloud LoadBalancerRequestTransformer which (contrary to its name) is a request interceptor.
My solution uses the following implementation:
public class CFLoadBalancerRequestTransformer implements LoadBalancerRequestTransformer {
public static final String CF_APP_GUID = "cfAppGuid";
public static final String CF_INSTANCE_INDEX = "cfInstanceIndex";
public static final String ROUTING_HEADER = "X-CF-APP-INSTANCE";
#Override
public HttpRequest transformRequest(HttpRequest request, ServiceInstance instance) {
System.out.println("Transforming Request from LoadBalancer Ribbon).");
// First: Get the service instance information from the lower Ribbon layer.
// This will include the actual service instance information as returned by Eureka.
RibbonLoadBalancerClient.RibbonServer serviceInstanceFromRibbonLoadBalancer = (RibbonLoadBalancerClient.RibbonServer) instance;
// Second: Get the the service instance from Eureka, which is encapsulated inside the Ribbon service instance wrapper.
DiscoveryEnabledServer serviceInstanceFromEurekaClient = (DiscoveryEnabledServer) serviceInstanceFromRibbonLoadBalancer.getServer();
// Finally: Get access to all the cool information that Eureka provides about the service instance (including metadata and much more).
// All of this is available for transforming the request now, if necessary.
InstanceInfo instanceInfo = serviceInstanceFromEurekaClient.getInstanceInfo();
// If it's only the instance metadata you are interested in, you can also get it without explicitly down-casting as shown above.
Map<String, String> metadata = instance.getMetadata();
System.out.println("Instance: " + instance);
dumpServiceInstanceInformation(metadata, instanceInfo);
if (metadata.containsKey(CF_APP_GUID) && metadata.containsKey(CF_INSTANCE_INDEX)) {
final String headerValue = String.format("%s:%s", metadata.get(CF_APP_GUID), metadata.get(CF_INSTANCE_INDEX));
System.out.println("Returning Request with Special Routing Header");
System.out.println("Header Value: " + headerValue);
// request.getHeaders might be immutable, so we return a wrapper that pretends to be the original request.
// and that injects an extra header.
return new CFLoadBalancerHttpRequestWrapper(request, headerValue);
}
return request;
}
/**
* Dumps metadata and InstanceInfo as JSON objects on the console.
* #param metadata the metadata (directly) retrieved from 'ServiceInstance'
* #param instanceInfo the instance info received from the (downcast) 'DiscoveryEnabledServer'
*/
private void dumpServiceInstanceInformation(Map<String, String> metadata, InstanceInfo instanceInfo) {
ObjectMapper mapper = new ObjectMapper();
String json;
try {
json = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(metadata);
System.err.println("-- Metadata: " );
System.err.println(json);
json = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(instanceInfo);
System.err.println("-- InstanceInfo: " );
System.err.println(json);
} catch (JsonProcessingException e) {
System.err.println(e);
}
}
/**
* Wrapper class for an HttpRequest which may only return an
* immutable list of headers. The wrapper immitates the original
* request and will return the original headers including a custom one
* added when getHeaders() is called.
*/
private class CFLoadBalancerHttpRequestWrapper implements HttpRequest {
private HttpRequest request;
private String headerValue;
CFLoadBalancerHttpRequestWrapper(HttpRequest request, String headerValue) {
this.request = request;
this.headerValue = headerValue;
}
#Override
public HttpHeaders getHeaders() {
HttpHeaders headers = new HttpHeaders();
headers.putAll(request.getHeaders());
headers.add(ROUTING_HEADER, headerValue);
return headers;
}
#Override
public String getMethodValue() {
return request.getMethodValue();
}
#Override
public URI getURI() {
return request.getURI();
}
}
}
The class is looking for the information required for setting the CF App Instance Routing header in the service instance metadata returned by Eureka.
That information is
The GUID of the CF application that implements the service and of which several instances exist for load balancing.
The index of the service / application instance that the request should be routed to.
You need to provide that in the application.yml of your service like so:
eureka:
instance:
hostname: ${vcap.application.uris[0]:localhost}
metadata-map:
# Adding information about the application GUID and app instance index to
# each instance metadata. This will be used for setting the X-CF-APP-INSTANCE header
# to instruct Go-Router where to route.
cfAppGuid: ${vcap.application.application_id}
cfInstanceIndex: ${INSTANCE_INDEX}
client:
serviceUrl:
defaultZone: https://eureka-server.<your cf domain>/eureka
Finally, you need to register the LoadBalancerRequestTransformer implementation in the Spring configuration of your service consumers (which use Ribbon under the hood):
#Bean
public LoadBalancerRequestTransformer customRequestTransformer() {
return new CFLoadBalancerRequestTransformer();
}
As a result, if you use a #LoadBalanced RestTemplate in your service consumer, the template will call Ribbon to make a choice on the service instance to send the request to, will send the request and the interceptor will inject the routing header. Go-Router will route the request to the exact instance that was specified in the routing header and not perform any additional load balancing that would interfere with Ribbon's choice.
In case a retry were necessary (against the same or one or more next instances), the interceptor would again inject the according routing header - this time for a potentially different service instance selected by Ribbon.
This allows you to use Ribbon effectively as the load balancer and de-facto disable load balancing of Go-Router, demoting it to a mere proxy. The benefit being that Ribbon is something you can influence (programmatically) whereas you have little to no influence over Go-Router.
Note: this was tested for #LoadBalanced RestTemplate's and works.
However, for #FeignClients it does not work this way.
The closest I have come to solving this for Feign is described in this post, however, the solution described there uses an interceptor that does not get access to the (Ribbon-)selected service instance, thus not allowing access to the required metadata.
Haven't found a solution so far for FeignClient.

How to redirect request to an URI passed in headers with Jetty AsyncProxyServlet

I'm creating a proxy micro-service with SpringBoot, Jetty and kotlin.
The purpose of this micro-service is to forward requests made by my front-end application to external services (avoiding CORS) and send back the response after checking some custom authentication. The query I'll receive will contain the URL of the target in the headers (i.e: Target-Url: http://domain.api/getmodel).
Based on this answer, I made a class that extends AsyncProxyServlet and overwrote the method sendProxyRequest :
class ProxyServlet : AsyncProxyServlet() {
private companion object {
const val TARGET_URL = "Target-Url"
}
override fun sendProxyRequest(clientRequest: HttpServletRequest, proxyResponse: HttpServletResponse, proxyRequest: Request) {
// authentication logic
val targetUrl = clientRequest.getHeader(TARGET_URL)
if (authSuccess) {
super.sendProxyRequest(clientRequest, proxyResponse, proxyRequest)
} else {
proxyResponse.status = HttpStatus.UNAUTHORIZED.value()
}
}
}
When I query my proxy, I get in this method and successfuly authenticate, but I fail to understand how to use my targetUrl to redirect the request.
The method keeps calling itself as it's redirecting the original request to itself (the request from http://myproxy:port/ to http://myproxy:port/).
It is very difficult to find documentation on this specific implementation of jetty, StackOverflow is my last resort!
First, setup logging for Jetty, and configure DEBUG level logging for the package namespace org.eclipse.jetty.proxy, this will help you understand the behavior much better.
The Request proxyRequest parameter represents a HttpClient/Request object, which is created with an immutable URI/URL destination (this is due to various other features that requires information from the URI/URL such as Connection pooling, Cookies, Authentication, etc), you cannot change the URI/URL on this object after the fact, you must create the HttpClient/Request object with the correct URI/URL.
Since all you want to do is change the target URL, you should instead be overriding the method ...
protected String rewriteTarget(HttpServletRequest clientRequest)
... and returning the new absolute URI String to the destination that you want to use (The "Target-Url" header in your scenario looks like a good candidate)
You can see this logic in the ProxyServlet.service(HttpServletRequest, HttpServletResponse) code block (which AsyncProxyServlet extends from)

Cannot get Spring Boot to lazily resolve a multipart file

I have created a Spring Boot 2 demo application with the Spring Initializr and added the controller below:
#Controller
#RequestMapping("/demo")
public class UploadController {
private final static Logger LOG = LoggerFactory.getLogger(UploadController.class);
#PostMapping("/upload")
public ResponseEntity<String> uploadFile(
#RequestParam("metadata") MultipartFile metadata,
#RequestParam("payload") MultipartFile payload) throws IOException {
ObjectMapper mapper = new ObjectMapper();
Map metadataMap = mapper.readValue(metadata.getInputStream(), Map.class);
LOG.info("Received call to upload file {}", metadataMap.get("filename"));
LOG.info("File size: {}", payload.getBytes().length);
LOG.info("File {} successfully uploaded", metadataMap.get("filename"));
return ResponseEntity.ok().build();
}
}
I then added an application.yaml file containing this configuration:
spring:
servlet:
multipart:
max-file-size: 2000000MB
max-request-size: 2000000MB
resolve-lazily: true
My goal is to have the controller parse and log the metadata file before it starts reading the payload file, but the resolve-lazily setting seems to be ignored by Boot: the code inside the controller won't be executed until the whole body is read.
I use the command below to test the controller:
curl -F metadata=#metadata.json -F payload=#payload.bin http://localhost:8080/demo/upload
Is there anything wrong with my code/configuration? Am I getting the meaning of the setting right?
At present, if you want to avoid reading (and buffering) the whole body all at once, I think you will have to provide your own parser, as described in the answers here. What would be really interesting (but generally unnecessary) would be to do so in the form of a new MultipartResolver implementation.
There are two existing implementations documented for interface MultipartResolver, and both supply a function setResolveLazily(boolean) (standard), (commons). I have tried with both, and neither seem to allow for parsing or streaming multipart files or parameters independently.
Default is "false", resolving the multipart elements immediately, throwing corresponding exceptions at the time of the resolveMultipart(javax.servlet.http.HttpServletRequest) call. Switch this to "true" for lazy multipart parsing, throwing parse exceptions once the application attempts to obtain multipart files or parameters.
Despite what it says in the documentation, I have found that once you call resolveMultipart, the entire body is parsed and buffered before the call returns. I know this because I can watch the temp-files being created.
One note about "Is there anything wrong with my code"...
Answer: Yes, because by using #RequestParam you have indirectly asked Spring to resolve your parameters ahead of time, before your controller is ever called. What you should be able to do instead (if the documentation were correct) is request the parameters independently from inside your controller:
Configuration (application.properties):
spring.servlet.multipart.enabled = true
spring.servlet.multipart.resolve-lazily = true
Controller:
#PostMapping(path = "/upload", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public ResponseEntity<Void> postUpload(HttpServletRequest rawRequest) {
multipartResolver.setResolveLazily(true); // unclear why this is exists
MultipartHttpServletRequest request = multipartResolver.resolveMultipart(rawRequest);
String p1 = request.getParameter("first-parameter");
String p2 = request.getParameter("second-parameter");
System.out.println("first-parameter="+p1+", second-parameter"+p2);
multipartResolver.cleanupMultipart(request);
return new ResponseEntity<Void>(HttpStatus.ACCEPTED);
}
One useful aspect of resolve-lazily that I have discovered is that it allows you to write your own parser for some rest controllers while using the built-in parser for others (see my answer here). In other words, you don't have to use spring.servlet.multipart.enabled = false to get your parser to work. This is a minor breakthrough relative to other advice that I had seen previously.

Resources