httpclient garbling extended characters - utf-8

Question

I'm using httpclient to retrieve remote urls and need to grab things such as titles.
In some cases, I get garbled extended characters as in the case of this url
http://olhardigital.uol.com.br/noticia/bilionaria-mais-jovem-da-historia-quer-revolucionar-exames-de-sangue/43586
I've tried playing around with all sorts of settings, to no avail. Any suggestions? My config is as follows:
private CloseableHttpClient httpclient = RemotePageUtils.getThreadSafeClient();
public String processMethod(String url, OutputStream out) throws IOException, IllegalArgumentException{
[...]
BufferedReader in = null;
HttpEntity entity = null;
HttpGet httpget = null;
CloseableHttpResponse resp = null;
try {
httpget = new HttpGet(url);
resp = httpclient.execute(httpget);
entity = resp.getEntity();
String inLine;
in = new BufferedReader(new InputStreamReader(entity.getContent(),"UTF-8"));
while ((inLine = in.readLine()) != null) {
out.write(inLine.getBytes("UTF-8"));
}
} finally {
[...]
}
return null;
}
private static CloseableHttpClient getThreadSafeClient() {
SocketConfig socketConfig = SocketConfig.custom()
.setTcpNoDelay(true)
.build();
RequestConfig config = RequestConfig.custom()
.setConnectTimeout(3000)
.setSocketTimeout(7000)
.setStaleConnectionCheckEnabled(false)
.build();
List<Header> headers = new ArrayList<Header>();
headers.add(new BasicHeader("Accept-Charset","ISO-8859-1,US-ASCII,UTF-8,UTF-16;q=0.7,*;q=0.7"));
//accept gzipped
headers.add(new BasicHeader("Accept-Encoding","gzip,x-gzip,deflate,sdch"));
CloseableHttpClient client = HttpClientBuilder.create()
.setDefaultHeaders(headers)
.setDefaultRequestConfig(config)
.setDefaultSocketConfig(socketConfig)
.build();
return client;
}

Answer 1

You are blindly interpreting all downloaded pages as UTF-8, but the example link you gave is not in UTF-8, but ISO-8859-1.
An accented letter in ISO-8859-1 is one byte >=128, where in UTF-8 such bytes have to follow specific patterns, in other case they are treated as damaged.
But why are you decoding bytes you have downloaded, just to write bytes to a file?
Instead of:
in = new BufferedReader(new InputStreamReader(entity.getContent(),"UTF-8"));
while ((inLine = in.readLine()) != null) {
out.write(inLine.getBytes("UTF-8"));
}
and converting bytes to strings and back, you should just copy the bytes.
You can do it with Apache Commons IO:
import org.apache.commons.io.IOUtils;
IOUtils.copy(entity.getContent(), out);
or manually, with pure Java:
byte[] buf = new byte[16 * 1024];
int len = 0;
InputStream in = entity.getContent();
while ((len = in.read(buf)) >= 0) {
out.write(buf, 0, len);
}