Spring-Cloud Zuul breaks UTF-8 symbols in forwarded multipart request filename - spring-boot

this is first time for me on SO, so please be patient for my first question.
I think i have some kind of configuration problem, but after a day of experiments i'm stuck. Our application is based on Spring-Cloud [Brixton release]. We have configuration like this: Portal (web application serving angular-based web-ui), which has zuul proxy with single route configured to our gateway service, like so:
zuul:
ignoredServices: '*'
prefix: /api
routes:
api-proxy:
path: /**
serviceId: api-gateway
which has another Zuul configured and relays requests to inner bussiness logic services:
zuul:
ignoredServices: '*'
routes:
service1:
path: /service1/**
serviceId: service1
service2:
path: /service2/**
serviceId: service2
All this configuration is working with no problem.
The problem now that i am facing is with file upload multipart requests. To be more precise - those multipart requests, when file to be uploaded has non latin symbols (e.g. ąčęėįš) from UTF-8. When request reaches service which has to deal with #RequestPart MultipartFile file, then file.getOriginalFilename() returns questionmarks in the places of aforementioned symbols. Now, i have tried to directly upload such file to such controller, and filename comes without questionmarks, that is, not broken, which suggests, that some bad interpretation/parsing of multipart request occurs somewhere in Zuul filters, when proxy relays incomming request.
Maybe someone had similar experience with Zuul and can direct me some way to resolve this problem?

I just ran into the same issue myself, and created the following issue:
https://jira.spring.io/browse/SPR-15396
Hopefully this is getting configurable in Spring 4.3.8.
In the meantime, you have to create a bean of type FormBodyWrapperFilter (that overrides the one in ZuulConfiguration). In the constructor, you pass a copy of FormHttpMessageConverter, which extends from FormHttpMessageConverter, and you change the encoding used in FormHttpMessageConverter.MultipartHttpOutputMessage#getAsciiBytes(String) to UTF-8 (you might also want to delete any references to javax-mail, unless you have that on classpath). You need a pretty recent version of Spring Cloud Netflix to do this.
Example:
#Bean
FormBodyWrapperFilter formBodyWrapperFilter() {
return new FormBodyWrapperFilter(new MyFormHttpMessageConverter());
}
Then you create a copy of FormHttpMessageConverter, and change the following method:
private byte[] getAsciiBytes(String name) {
try {
// THIS IS THE ONLY MODIFICATION:
return name.getBytes("UTF-8");
} catch (UnsupportedEncodingException ex) {
// Should not happen - US-ASCII is always supported.
throw new IllegalStateException(ex);
}
}

Still having issues after modifying response even with newer versions
Using spring boot 2.3.8.RELEASE
Managed to fix it by forcing the following spring properties
server.servlet.encoding.force= true
server.servlet.encoding.charset= UTF-8

Related

How to configure Spring Boot to use OIDC while app is behind an SSL termination proxy

I am trying to configure a Spring Boot app to use OIDC. The server is behind an SSL termination proxy.
Here are the properties I use:
spring:
security:
oauth2:
client:
provider:
oidc:
authorization-uri: https://example.org/oidc/oauth2/authorize
token-uri: https://example.org/oidc/oauth2/access_token
user-info-uri: https://example.org/oidc/oauth2/userinfo
jwk-set-uri: https://example.org/oidc/oauth2/connect/jwk_uri
custom-params: param_name1,param_value1,param_name2,param_value2,nonce,123456789
registration:
oidc:
client-id: myclientid
client-secret: myclientsecret
authorization-grant-type: authorization_code
scope:
- openid
- email
- profile
redirect-uri: https://mydomain/myapp/login/oauth2/code/oidc
Here's where it goes wrong:
1. The OIDC server requires a nonce param to be added to the request URL
I have solved this by using a custom OAuth2AuthorizationRequest that reads the custom-params property and appends those values to the request URL
2. The OidcAuthorizationCodeAuthenticationProvider throws an exception caused by invalid_redirect_uri_parameter
I have tried many approaches to fix this.
I have tried creating a filter that adds the X-Forwarded-Proto to the request (because the proxy doesn't handle that).
The headers are added, I have also added the following properties:
server:
forward-headers-strategy: native
tomcat.protocol-header: x-forwarded-proto
But it doesn't seem to work.
OidcAuthorizationCodeAuthenticationProvider still throws an exception because this condition is false:
!authorizationResponse.getRedirectUri().equals(authorizationRequest.getRedirectUri())
I have debugged the code and the only difference is one being http and the other https.
I have found a VERY hacky solution that I don't like at all, which is another filter that modifies the URL just for that particular URL.
I would prefer a more elegant solution.
3. When using the custom nonce parameter, the OidcAuthorizationCodeAuthenticationProvider throws an exception cause by invalid_nonce
Now I am stuck. I considered writing my own Authentication Provider, but I have no guarantee that mine will be picked up before the OIDC one provided by Spring.
And with the nonce, it's a catch 22:
if I don't use the custom param, I couldn't find a way to make Spring add the nonce to the request
if I use that one, Spring doesn't recognize it when it's part of the JWT and freaks out
Any help would be GREATLY appreciated, as this has been driving me nuts for days if not weeks.
Thank you.
EDIT
The 2 urls that are compared in case 2 come from:
OAuth2AuthorizationRequest
OAuth2AuthorizationResponse
OAuth2AuthorizationRequest is built in the
OAuth2AuthorizationRequestRedirectFilter at the following line:
OAuth2AuthorizationRequest authorizationRequest = this.authorizationRequestResolver.resolve(request);
The redirect uri is built in DefaultOAuth2AuthorizationRequestResolver.expandRedirectUri() which calls
UriComponentsBuilder.fromHttpUrl(UrlUtils.buildFullRequestUrl(request))
OAuth2AuthorizationResponse is built in the OAuth2LoginAuthenticationFilter.attemptAuthentication() which also calls
UriComponentsBuilder.fromHttpUrl(UrlUtils.buildFullRequestUrl(request))
and then
OAuth2AuthorizationResponseUtils.convert(params, redirectUri)
I will double check, but I don't remember UriComponentsBuilder.adaptFromForwardedHeaders(HttpHeaders headers) being called when building these URLs.
And even if that works, that still leaves the nonce issue :(
We stumbled upon the same problem , the problem was mainly because our server was behind a reverse proxy, and it seems the proxy changed the url somehow causing this check to fail
!authorizationResponse.getRedirectUri().equals(authorizationRequest.getRedirectUri())
this line was removed in later versions of spring security , at commit
24500fa3ca23aa23ede86dfcfe02113671d5b8bc
commit at github
which was introduced on Dec 6, 2019 and was in spring security release 5.1.13
so the solution was to upgrade spring boot to at least 2.1.17 for spring boot 2.1.X line of versions.
while the OP said he can't upgrade his libraries, I hope this can help those who can.
We also did the solution mentioned above by Kavithakaran Kanapathippilla and configured our reverse proxy to add X-forwarded-proto http headers , also i believed we configured spring boot application.properties to check for them
spring boot documentation for working behind proxies

hosting zipkin inside proxy/zuul-gateway

zipkin is a tool for tracing request as well as tracking the span of time a service took to process the request useful in multi-service projects it doesnt require much effort for setting up u just have to add zipkin dependency in your services and define a sampler bean.
add the following dependency in project
compile group: 'org.springframework.cloud', name: 'spring-cloud-starter-zipkin', version: '1.3.2.RELEASE'
add sampler bean inside ur project
` #Value("${spring.sleuth.sampler.percentage}")
String percentage;
#Bean
public PercentageBasedSampler defaultSampler() {
SamplerProperties configuration= new SamplerProperties();
configuration.setPercentage(Float.parseFloat(percentage));
return new PercentageBasedSampler(configuration);
}
`
add above bean when u want only fraction of ur requests traces to send to zipkin else define a bean
` #Bean
public AlwaysSampler defaultSampler() { return new AlwaysSampler();
}
`
add
spring.zipkin.base-url=localhost:9411 in ur properties file and host the zipkin server on the same port defined above.
but if u r using api-gateway for accessing zipkin (in case of deplyment in cloud ) or inside proxy u may face the issue of broken ui elements when accessing thru gateway in this case im using zuul with propertis as:
zuul.routes.zipkin.path=/zipkin/*
zuul.routes.zipkin.url=http://localhost:9411
the simplest solution i have found for solving broken ui for zipkin thru gateway
is by changing following property of zipkin-server-shared.yml file inside zipkin server
zipkin:
ui:
base-path: /zipkin
change above property to
zipkin:
ui:
base-path: /api/tracing/zipkin
and change ur zuul path to following
zuul.routes.zipkin.path=/api/tracing/*
and than access zipkin using follwing url
https://gatewayhost:port/api/tracing/zipkin/
give attention to small details in config and dont forget to put trailing "/" after zipkin in url

Spring Cloud: default redirecting from Gateway to UI

I'm new to microservices and Spring Boot. I have a few Spring Cloud microservices with a Zuul gateway running on port 8080.
browser
|
|
gateway (:8080)
/ \
/ \
/ \
resource UI (:8090)
There is a UI microservice on port 8090, which has a controller with a method inside, returning index.html.
I configured Zuul routing like this for UI (I'm using Eureka too):
zuul:
routes:
ui:
path: /ui/**
serviceId: myproject.service.ui
stripPrefix: false
sensitiveHeaders:
If I call http://localhost:8080/ui/ everything works fine and I see rendering of my index.html.
Is it possible to configure Spring Cloud in some way to make the following flow work: calling http://localhost:8080/ redirects us to controller of UI microservice, which returns index.html?
So the idea is to open UI from the root of my site.
Thanks in advance!
Finally, I've made my code work! Thanks to #pan for mentioning Zuul Routing on Root Path question and #RzvRazvan for revealing about how Zuul routing works.
I've just added controller to Zuul routed Gateway microservice with one endpoint to redirect from root http://localhost:8080/ to http://localhost:8080/ui/:
#Controller
public class GateController {
#GetMapping(value = "/")
public String redirect() {
return "forward:/ui/";
}
}
Zuul properties for redirecting from Gateway microservice on port 8080 as http://localhost:8080/ui/ to UI microservice, which implemented as a separate Spring Boot application on port 8090 as http://localhost:8090/ui/:
zuul:
routes:
ui:
path: /ui/**
serviceId: myproject.service.ui
stripPrefix: false
sensitiveHeaders:
UI microservice's properties:
server:
port: 8090
servlet:
contextPath: /ui
Eventually, this call http://localhost:8080/ redirects us to controller of UI microservice, which returns view index.html:
#Controller
public class UiController {
#GetMapping(value = "/")
public String index() {
return "index.html";
}
}
Actually, I had another problem with rendering static content in such architecture, but it was connected with configuration of my front-end, which I develop using Vue.js framework. I will describe it here in a few sentences, in case it might be helpful for someone.
I have the following folders structure of UI microservice:
myproject.service.ui
└───src/main/resources
└───public
|───static
| ├───css
| └───js
└───index.html
All content of public folder is generated by npm run build task from webpack and vue.js. First time, I called my http://localhost:8080/ I got 200 OK for index.html and 404 for all other static resources, because they was called like this:
http:\\localhost:8080\static\js\some.js
So it was configured wrong public path for static assets in webpack. I changed it in config\index.js:
module.exports = {
...
build: {
...
assetsPublicPath: '/ui/',
...
}
...
}
And static assets became to be called properly. e.g.:
http:\\localhost:8080\ui\static\js\some.js
If you would like to have on Zuul the UI(front-end) you can add the static content in resources/static folder(html, css and js files). On this way your proxy is able to render the index.html (of course you must have an index.html in static folder). O this way on http://localhost:8080 the proxy will render index.html; also you can have another paths but all these path are managed by index.html)
About routing, the Zuul only redirect the http request. http://localhost:8080/ui/. On 8080 is running the proxy (Zuul) BUT /ui is the context path of resource server. Se when you make a call on this path http://localhost:8080/ui/ the proxy will redirect to resource server and will actually make a request to http://localhost:8090/ui/
It is a difference between browser path and http request path. The browser path is managed by index.html and the http request is managed by Zuul. I don't know if I was explicit enough.
One more thing... You can have the same path (/ui) on http request and index.html and when your browser will access the http://localhost:8080/ui/ a .js file with http request method will make an http request to http://localhost:8080/ui/ and then will be redirected to http://localhost:8090/ui/ and the response from the resource server will be rendered on the page from http://localhost:8080/ui/.

Zuul request headers not forwarded

I have an application that uses Zuul and is hosted in a Kubernetes Cluster on AWS. It would appear that Zuul is not forwarding headers from the request to the destination, so I m having to do this:
private void copyRequestHeaders() {
RequestContext context = RequestContext.getCurrentContext();
Enumeration<String> headerNames = context.getRequest().getHeaderNames();
while(headerNames.hasMoreElements()) {
String headerName = headerNames.nextElement();
context.addZuulRequestHeader(headerName, context.getRequest().getHeader(headerName));
}
}
I'm guessing that I shouldn't have to do this and that I'm missing a configuration option somewhere or something like that. Can anyone shed any light on this. Interestingly the application works fine in my local development environment.
I'd also like to know the purpose of the ZuulRequestHeader and ZuulResponseHeadera and how they differ from standard request and response headers. Are they wrappers perhaps?
Thanks
Try adding the sensitiveHeaders in your configuration.
zuul:
routes:
sample:
sensitiveHeaders:
path: /sample
Removing any values on sensitiveHeaders means that all headers passed on your Zuul Gateway will be passed deep within the microservices.
For more information, refer to this documentation.
By default, Zuul API Gateway will not let sensitive information to be forwarded to downstream Microservices.
The default value for this property is
zuul.sensitiveHeaders: Cookie,Set-Cookie,Authorization
we can override this value to allow certain headers by
zuul.sensitiveHeaders: Cookie,Set-Cookie
And to allow all the headers we can keep it blank
zuul.sensitiveHeaders:

Spring Cloud Netflix: Whats happening in ZuulConfiguration with the ZuulServlet?

Looking in ZuulConfiguration, I see the following:
#Bean
public ZuulController zuulController() {
return new ZuulController();
}
#Bean
public ServletRegistrationBean zuulServlet() {
return new ServletRegistrationBean(new ZuulServlet(),
this.zuulProperties.getServletPattern());
}
ZuulController wraps the ZuulServlet and manages its lifecycle as if it were a Spring Controller. What throws me is that the ZuulConfiguration class registers the servlet anyways using a ServletRegistrationBean. Perhaps I am thinking the wrong thing, but I would think that you would do one or the other. Could someone explain why both are necessary?
Using this configuration, is the ZuulServlet running as a true servlet (known by the embedded servlet container), a controller (which delegates to the servlet) or both?
Thanks,
Joshua
From this commit:
Allow streaming of multipart requests in Zuul proxy
It turns out that the suckiness of Zuul with multipart requests
comes almost entirely from the Multipart handling in Spring's
DispatcherServlet. This change makes the proxy routes available
on an alternative path /zuul/ (where
/zuul is the default value of zuul.servletPath). I have
tested those with 800MB file uploads using the main method in
the FormZuulServletProxyApplicationTests and the main
observation is that there is no OutOfMemory error (no-one tries
to download the complete request body). It works with Ribbon
and with the simple (HttpClient) filter. With Ribbon you
will need to set some timeouts if you want to upload files
as large as that, e.g. see application.yml in the tests:
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds: 60000
ribbon:
ConnectTimeout: 3000
ReadTimeout: 60000
You need to set "Transfer-Encoding: chunked" in the
incoming request. Chrome does not do this by default
apparently, but I was able to test with curl, e.g.
$ curl -v -H "Transfer-Encoding: chunked" \
-F "file=#mylarg.iso" \
localhost:9999/zuul/direct/file
The old proxy paths through the DispatcherServlet are still
available (for backwards compatibility and for convenience of
having the paths available at the root of the context path).

Resources