Orbeon form in chinese returned with question marks - utf-8

Question

I'm using ORBEON 2018.2.3.201905172253 PE within SAP Commerce (Hybris) on a Tomcat 7.0. When Hybris hits the Orbeon app to create a new form and get the inline HTML in Chinese, I'm getting question marks instead of Chinese characters.
URL: http://localhost:9001/orbeon/fr/yforms/myform/new?orbeon-embeddable=true&fr-language=zh-Hans
Method: GET
Headers:
[
{
"key":"hybris-Username",
"value":"-"
},
{
"key":"hybris-Group",
"value":"-"
},
{
"key":"hybris-Roles",
"value":"-"
},
{
"key":"Orbeon-Client",
"value":"portlet"
},
{
"key":"hybris-Proxy-09e4ff02-4715-4547-8f81-30082598eec9",
"value":"37bb0017-2675-4617-945d-6693bdae8eb9"
},
{
"key":"Content-Type",
"value":"text/html;charset=UTF-8"
}
]
I've found there's a known issue in Tomcat 7 for Chinese characters:
https://books.google.co.nz/books?id=kNVe8lzTec0C&pg=PA166&lpg=PA166&dq=webapp+http+request+chinese+question+marks&source=bl&ots=F11m7FJGYD&sig=ACfU3U1fufLJggVpnu4iUFT9SUJ6SdhqmA&hl=en&sa=X&ved=2ahUKEwjJq5L-_8njAhVi7nMBHXnMD0sQ6AEwAnoECAkQAQ#v=onepage&q=webapp%20http%20request%20chinese%20question%20marks&f=false
And apparently, Orbeon includes a way to set the character encoding to UTF-8 using the oxf.xforms.renderer.default-encoding param in the web.xml file:
https://doc.orbeon.com/xforms/filter
I already tried that in my web.xml and got the same results from the Orbeon app. I even executed the same request using Postman to check if it's an integration problem with my Hybris storefront, but the results were the same.
This is an excerpt of my web.xml:
<filter>
<filter-name>orbeon-xforms-filter</filter-name>
<filter-class>org.orbeon.oxf.servlet.OrbeonXFormsFilter</filter-class>
<!-- Uncomment this for the separate WAR deployment -->
<init-param>
<param-name>oxf.xforms.renderer.context</param-name>
<param-value>/orbeon</param-value>
</init-param>
<init-param>
<param-name>oxf.xforms.renderer.default-encoding</param-name>
<param-value>UTF-8</param-value>
</init-param>
<!-- End separate WAR deployment -->
</filter>
<filter-mapping>
<filter-name>orbeon-xforms-filter</filter-name>
<url-pattern>/xforms-jsp/*</url-pattern>
<!--Servlet 2.4 configuration allowing the filter to run upon forward in addition to request-->
<dispatcher>REQUEST</dispatcher>
<dispatcher>FORWARD</dispatcher>
</filter-mapping>
Changing the url-pattern of the filter to "/orbeon/fr/yforms/*" broke the whole thing and I couldn't even render the form in English.
When debugging the request in Java I tried setting the Character Encoding of the HttpRequest object to UTF-8. Same results.
I've also tried these without good results:
• Set the URIEncoding attribute on the element in server.xml to URIEncoding="UTF-8".
• Set the useBodyEncodingForURI attribute on the element in server.xml to true.
Related to this:
Orbeon form localization using current site's session language
The answer to that thread explains how I'm localizing my forms.
THE WEIRDEST THING IS that the response of that URL DOES include some characters in Chinese: the label of the dropdown language selector. All other labels are question marks. Super weird!
I'd like to know how to fix this behaviour either changing my Tomat config or my Orbeon app config.
Thanks,
David

Answer 1

Well, as expected the problem was not related to Orbeon.
In my Spring Controller, the method that was getting my form from the database was annotated with a #ResponseBody. The String being returned by the Controller was in a perfect state containing the proper Chinese characters, but the response object was then intercepted by a Spring converter.
#RequestMapping(method = RequestMethod.GET, value = ...")
#ResponseBody
public String getFormDefinition(#PathVariable final String aaa, #PathVariable final String aaa,
#RequestParam(value = "document", required = false) final String aaa, final HttpServletResponse response)
throws ServletException, IOException, YFormServiceException
{
return "html content";
}
When there's a #ResponseBody annotation, the response is processed by Spring's message-converters. The default converters are listed here: https://www.baeldung.com/spring-httpmessageconverter-rest
One of those converters is the magical StringHttpMessageConverter. If you check its source code, you can see it uses ISO-8859-1 as a default charset. Every response processed by these converter is going to be encoded using ISO-8859-1, which does not support Chinese characters.
The Fix:
In my web spring config file I added:
<mvc:annotation-driven>
<mvc:message-converters register-defaults="false">
<bean class="org.springframework.http.converter.ByteArrayHttpMessageConverter"/>
<bean class="org.springframework.http.converter.ResourceHttpMessageConverter"/>
<bean class="org.springframework.http.converter.xml.SourceHttpMessageConverter"/>
<bean class="org.springframework.http.converter.StringHttpMessageConverter">
<constructor-arg index="0" name="defaultCharset" value="UTF-8"/>
</bean>
<bean class="org.springframework.http.converter.FormHttpMessageConverter"/>
<bean class="org.springframework.http.converter.json.MappingJackson2HttpMessageConverter"/>
<bean class="org.springframework.http.converter.xml.Jaxb2RootElementHttpMessageConverter"/>
<bean class="org.springframework.http.converter.json.MappingJackson2HttpMessageConverter"/>
</mvc:message-converters>
</mvc:annotation-driven>
With register-defaults="false" you say to Spring not to use any of the default converters, but use the provided list below. When defining StringHttpMessageConverter, I specify UTF-8 (which supports Chinese characters) as the default encoding.
Et voila