Spring form and UTF-8 bad encoding - spring

Question

In our Web app we've faced a bad encoding problem. In order to reproduce this problem user selects in browser non unicode encoding(as example in chrome -> More tools->encoding->Koi8) and tries to set Cyrillic text.
Chars were spoiled when it goes to controller (just checked on debug) and even it's stored incorrectly and incorrectly rendered.
We've followed all recommendations: http://balusc.blogspot.com/2009/05/unicode-how-to-get-characters-right.html and seems this is a problem with submitting of application/x-www-form-urlencoded encoding content type. Because it's impossible to set charset during such forms submits.
As example if submit the same data using json and set necessry content type everything is stored correctly.
We've also tried example with this article:
http://www.codejava.net/frameworks/spring/spring-mvc-form-handling-tutorial-and-example and added additionally UTF8 filer with the following method:
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException,
ServletException {
request.setCharacterEncoding("UTF-8");
response.setCharacterEncoding("UTF-8");
chain.doFilter(request, response);
}
But the same problem was reproducible.
Could somebody suggest how to resolve mentioned problem?
Is it possible to correctly handle mentioned use case in Spring MVC because we tried on simple example and it seems it doesn't work. Does this use case of changing browser encoding valid at all ?

Answer 1

Try : In web.xml
<filter>
<filter-name>encoding-filter</filter-name>
<filter-class>
org.springframework.web.filter.CharacterEncodingFilter
</filter-class>
<init-param>
<param-name>encoding</param-name>
<param-value>UTF-8</param-value>
</init-param>
<init-param>
<param-name>forceEncoding</param-name>
<param-value>true</param-value>
</init-param>
</filter>
<filter-mapping>
<filter-name>encoding-filter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
Refer : http://wiki.apache.org/tomcat/FAQ/CharacterEncoding
Also, in view if you use jstl try to set the default encoding

Answer 2

This behavior can be achieved by using accept-charset="UTF-8" attribute in form.
It can be added in spring form tag. Also there's a bug in older versions of struts (1.1 are affected)
https://issues.apache.org/jira/browse/STR-1636
that makes impossible to add this attribute directly to the form. As a workaround jQuery can be used
jQuery( document ).ready(function() {
jQuery("#formSelector").attr("accept-charset", "UTF-8");
});
So in a nutshell this attribute will force browser send data from this form using specified encoding. In case user will set some control characters into the input they will be sent to backend as well so validation is required to prevent such cases. Also cases where the browser encoding and keyboard languages are not working well together will be handled (for example KOI8-U and Chinese keyboard language).
accept-charset official documentation

Answer 3

Somewhere in your request pipeline your overriding the encoding (ie String.getBytes() or new String(bytes) is being called with out the right encoding.
There are so many places that this can potentially happen and its one of the reasons why Spring Boot and various other frameworks force UTF-8 for both input and output particularly since UTF-8 is the recommended encoding.
Your users should not be changing the encoding. In fact when the page loads both the servlet response and the HTML itself should specify UTF-8 and this is for good reason. The server is saying "I speak UTF-8". If you want a different encoding you will have to specify a different encoding in the HTML (ie jsp), and the servlet request/responses so that the browser will then auto select encoding. Even then your mileage will vary for application/x-www-form-urlencoded particularly if you use Javascript (probably because the spec on the encoding in other characters is somewhat ambiguous). To give you some more example of why the HTML has to have the exact same encoding as what your sending over is that the name value pairs will have different encoding. That is your form has UTF-8 request parameter names (because thats whats in the HTML) but when you override it your supplying a different encoding for the parameter values (ie ?UTF-8Name=KoiValue&UTF-8Name=KoiValue). Hopefully you can see why that is bad and I'm not sure chrome is smart enough (or if it even should do it) to change the request parameter names back to Koi8.
Thus if you absolutely must support other character encoding you probably should use multipart/form (you specify this in the enctype attribute on the form element) AND NOT USE the encoding filters that set UTF-8 as those will probably cause corruption.