HttpsURLConnection for rest service call - httpsurlconnection

i am using HttpsURLConnection for calling POST method and i am unable to add request payload to the call. same request works when i execute manually from soapui tool. does any one know how to do this?
Here is the java code is am using:
// Keystore
String jksFile = "/Users/......jks";
String password = "password";
System.setProperty("javax.net.ssl.keyStore", jksFile);
System.setProperty("javax.net.ssl.keyStorePassword", password);
String u = "https://example.com:443/private/loan-application";
URL url = new URL(u);
String version = "3";
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setUseCaches(false);
conn.setConnectTimeout(10000);
conn.setReadTimeout(10000);
conn.setRequestProperty("Accept", "application/json;v=" + version);
conn.setRequestProperty("Api-Key", "XYZ");
String jsonParamString = "{\"didNumber\":\"DI457\", \" .... ";
//Create JSONObject here
JSONObject jsonParam = new JSONObject();
byte[] postData = jsonParamString.toString().getBytes();
Map<String, List<String>> requestProperties = conn.getRequestProperties();
displayHeaders(requestProperties);
OutputStream out = conn.getOutputStream();
out.write(postData);
out.close();
int respCode = conn.getResponseCode();
System.out.println("Response headers:");
displayHeaders(
conn.getHeaderFields());
InputStream is = null;
if (respCode == 200 || respCode == 203) {
is = conn.getInputStream();
}
if (is == null) {
System.out.println("Using error stream");
is = conn.getErrorStream();
}
is.close();

One possible explanation for why your HttpsURLConnection call is failing is that you have not configured your SSL settings properly. Specifically, the domain example.com is presenting a certificate which is not in your trust store, and therefore your program is failing during the SSL handshake. Try adding the following code, which will configure your program to trust all certificates:
SSLContext sslContext = SSLContext.getInstance("TLS");
TrustManager tm = new X509TrustManager() {
public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException { }
public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException { }
public X509Certificate[] getAcceptedIssuers() { return null; }
};
sslContext.init(null, new TrustManager[]{tm}, null);
conn.setSSLSocketFactory(sslContext.getSocketFactory());
Note that you should probably not use this code in production, because it accepts any site as secure. In production, you should configure your program to only accept trusted certificates.

Related

Adding InclusiveNamespaces with WSS4J

In a WS-Security header, I'd like to add InclusiveNamespaces like this but I do not know how:
<ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
<InclusiveNamespaces PrefixList="wsse oas soapenv urn urn1 urn2 urn3 urn4" xmlns="http://www.w3.org/2001/10/xml-exc-c14n#"/>
</ds:Transform>
Here is my code that creates the security header, how do I add the InclusiveNamespaces with WSS4J as seen in above XML snippet?
public static SOAPMessage signSoapMessage(SOAPMessage message,
String keystorePassword, String irsPrivateKeyPassword,
char[] passphrase) throws WSSecurityException {
PrivateKeyEntry privateKeyEntry = getPrivateKeyEntry(keystorePassword,
irsPrivateKeyPassword);
PrivateKey signingKey = privateKeyEntry.getPrivateKey();
X509Certificate signingCert = (X509Certificate) privateKeyEntry
.getCertificate();
final String alias = "signingKey";
final int signatureValidityTime = 3600; // 1hour in seconds
WSSConfig config = new WSSConfig();
config.setWsiBSPCompliant(false);
WSSecSignature builder = new WSSecSignature(config);
builder.setX509Certificate(signingCert);
builder.setUserInfo(alias, new String(passphrase));
builder.setUseSingleCertificate(true);
builder.setKeyIdentifierType(WSConstants.X509_KEY_IDENTIFIER);
builder.setDigestAlgo(WSConstants.SHA1);
builder.setSignatureAlgorithm(WSConstants.RSA_SHA1);
builder.setSigCanonicalization(WSConstants.C14N_EXCL_WITH_COMMENTS);
try {
Document document = toDocument(message);
WSSecHeader secHeader = new WSSecHeader();
//secHeader.setMustUnderstand(true);
secHeader.insertSecurityHeader(document);
WSSecTimestamp timestamp = new WSSecTimestamp();
timestamp.setTimeToLive(signatureValidityTime);
document = timestamp.build(document, secHeader);
List<WSEncryptionPart> parts = new ArrayList<WSEncryptionPart>();
WSEncryptionPart timestampPart = new WSEncryptionPart("Timestamp",
WSConstants.WSU_NS, "");
WSEncryptionPart aCATransmitterManifestReqDtlPart = new WSEncryptionPart(
"ACATransmitterManifestReqDtl",
"urn:us:gov:treasury:irs:ext:aca:air:7.0", "");
WSEncryptionPart aCABusinessHeaderPart = new WSEncryptionPart(
"ACABusinessHeader",
"urn:us:gov:treasury:irs:msg:acabusinessheader", "");
parts.add(timestampPart);
parts.add(aCATransmitterManifestReqDtlPart);
parts.add(aCABusinessHeaderPart);
builder.setParts(parts);
Properties properties = new Properties();
properties.setProperty("org.apache.ws.security.crypto.provider",
"org.apache.ws.security.components.crypto.Merlin");
Crypto crypto = CryptoFactory.getInstance(properties);
KeyStore keystore = KeyStore.getInstance("JKS");
keystore.load(null, passphrase);
keystore.setKeyEntry(alias, signingKey, passphrase,
new Certificate[] { signingCert });
((Merlin) crypto).setKeyStore(keystore);
crypto.loadCertificate(new ByteArrayInputStream(signingCert
.getEncoded()));
document = builder.build(document, crypto, secHeader);
updateSOAPMessage(document, message);
} catch (Exception e) {
// throw new
// WSSecurityException(WSSecurityException.Reason.SIGNING_ISSUE, e);
e.printStackTrace();
}
return message;
}
private static Document toDocument(SOAPMessage soapMsg)
throws TransformerConfigurationException, TransformerException,
SOAPException, IOException {
Source src = soapMsg.getSOAPPart().getContent();
TransformerFactory tf = TransformerFactory.newInstance();
Transformer transformer = tf.newTransformer();
DOMResult result = new DOMResult();
transformer.transform(src, result);
return (Document) result.getNode();
}
//https://svn.apache.org/repos/asf/webservices/wss4j/branches/WSS4J_1_1_0_FINAL/test/wssec/SOAPUtil.java
private static SOAPMessage updateSOAPMessage(Document doc,
SOAPMessage message)
throws Exception {
DOMSource domSource = new DOMSource(doc);
message.getSOAPPart().setContent(domSource);
return message;
}
EDIT WITH ANSWER
The above code is for v1.6.x of WSS4J and so the solution to above code is
WSSConfig config = new WSSConfig();
config.setWsiBSPCompliant(true);

httpclient garbling extended characters

I'm using httpclient to retrieve remote urls and need to grab things such as titles.
In some cases, I get garbled extended characters as in the case of this url
http://olhardigital.uol.com.br/noticia/bilionaria-mais-jovem-da-historia-quer-revolucionar-exames-de-sangue/43586
I've tried playing around with all sorts of settings, to no avail. Any suggestions? My config is as follows:
private CloseableHttpClient httpclient = RemotePageUtils.getThreadSafeClient();
public String processMethod(String url, OutputStream out) throws IOException, IllegalArgumentException{
[...]
BufferedReader in = null;
HttpEntity entity = null;
HttpGet httpget = null;
CloseableHttpResponse resp = null;
try {
httpget = new HttpGet(url);
resp = httpclient.execute(httpget);
entity = resp.getEntity();
String inLine;
in = new BufferedReader(new InputStreamReader(entity.getContent(),"UTF-8"));
while ((inLine = in.readLine()) != null) {
out.write(inLine.getBytes("UTF-8"));
}
} finally {
[...]
}
return null;
}
private static CloseableHttpClient getThreadSafeClient() {
SocketConfig socketConfig = SocketConfig.custom()
.setTcpNoDelay(true)
.build();
RequestConfig config = RequestConfig.custom()
.setConnectTimeout(3000)
.setSocketTimeout(7000)
.setStaleConnectionCheckEnabled(false)
.build();
List<Header> headers = new ArrayList<Header>();
headers.add(new BasicHeader("Accept-Charset","ISO-8859-1,US-ASCII,UTF-8,UTF-16;q=0.7,*;q=0.7"));
//accept gzipped
headers.add(new BasicHeader("Accept-Encoding","gzip,x-gzip,deflate,sdch"));
CloseableHttpClient client = HttpClientBuilder.create()
.setDefaultHeaders(headers)
.setDefaultRequestConfig(config)
.setDefaultSocketConfig(socketConfig)
.build();
return client;
}
You are blindly interpreting all downloaded pages as UTF-8, but the example link you gave is not in UTF-8, but ISO-8859-1.
An accented letter in ISO-8859-1 is one byte >=128, where in UTF-8 such bytes have to follow specific patterns, in other case they are treated as damaged.
But why are you decoding bytes you have downloaded, just to write bytes to a file?
Instead of:
in = new BufferedReader(new InputStreamReader(entity.getContent(),"UTF-8"));
while ((inLine = in.readLine()) != null) {
out.write(inLine.getBytes("UTF-8"));
}
and converting bytes to strings and back, you should just copy the bytes.
You can do it with Apache Commons IO:
import org.apache.commons.io.IOUtils;
IOUtils.copy(entity.getContent(), out);
or manually, with pure Java:
byte[] buf = new byte[16 * 1024];
int len = 0;
InputStream in = entity.getContent();
while ((len = in.read(buf)) >= 0) {
out.write(buf, 0, len);
}

Web service request working in old version of android (2.3.3) but not in later versions (4.0.3, 4.3)

While working on an application for android that uses web services I encounterd a bad request (response code 400) message when trying to retrieve some data in android versions 4.0.3 and 4.3. The perculiar thing however is that when sending the same request using the same code but on a device using android version 2.3.3 it works without any problems. I have also tried using httpGet instead of HttpsURLConnection, while this work for all versions it does not provide a solution as I need the added security.
My code is as follows:
private String executeRequest(String urlAddress)
{
String responce = null;
String msg = null;
int error = 0;
try {
URL url = new URL(urlAddress);
HttpsURLConnection connection = (HttpsURLConnection)url.openConnection();
SSLSocketFactory factory = SecureSocketFactory.getSSLSocketFactory();
connection.setSSLSocketFactory(factory);
connection.setHostnameVerifier(new Verifier());
connection.setDoOutput(true);
connection.setDoInput(true);
if (method == RequestMethod.POST)
{
connection.setRequestMethod("POST");
}
msg = connection.getResponseMessage();
error = connection.getResponseCode();
if ("OK".equals(msg))
{
InputStream content = (InputStream) connection.getContent();
responce = convertStreamToString(content);
}
else
{
responce = "Error " + error;
}
connection.disconnect();
} catch (Exception e) {
responce = e.toString();
}
return responce;
}
And the code of SecureSocketFactory.getSSLSocketFactory():
public static SSLSocketFactory getSSLSocketFactory()
throws IOException
{
if(ssf_ == null)
{
javax.net.ssl.KeyManager kms[] = null;
javax.net.ssl.TrustManager tms[] = null;
SSLContext context = null;
try
{
tms = CustomTrustManager.getTrustManagers();
context = SSLContext.getInstance("TLS");
context.init(kms, tms, null);
}
catch(GeneralSecurityException e)
{
IOException io = new IOException(e.getLocalizedMessage());
io.setStackTrace(e.getStackTrace());
throw io;
}
ssf_ = context.getSocketFactory();
}
return ssf_;
}
and the code of CustomTrustManager.getTrustManagers()
static TrustManager[] getTrustManagers(String trustStoreFile, String trustStorePW)
throws NoSuchAlgorithmException, KeyStoreException
{
String alg = TrustManagerFactory.getDefaultAlgorithm();
TrustManagerFactory tmFact = TrustManagerFactory.getInstance(alg);
tmFact.init((KeyStore)null);
TrustManager tms[] = tmFact.getTrustManagers();
for(int i = 0; i < tms.length; i++)
if(tms[i] instanceof X509TrustManager)
tms[i] = new CustomTrustManager((X509TrustManager)tms[i]);
return tms;
}
static TrustManager[] getTrustManagers()
throws NoSuchAlgorithmException, KeyStoreException
{
return getTrustManagers(null, null);
}
I have looked everywhere, but can't seem to find a solution please help.
I found my error, because do connection.setDoInput(true) it silencly sets my Requestmethod to post in version 4 which gives an error on the server causing it to return bad request.
apparently it does not set this in version 2, which explains why it does work there.
The following execute request method change fixed my code:
private String executeRequest(String urlAddress)
{
String responce = null;
String msg = null;
int error = 0;
try {
URL url = new URL(urlAddress);
HttpsURLConnection connection = (HttpsURLConnection)url.openConnection();
SSLSocketFactory factory = SecureSocketFactory.getSSLSocketFactory();
connection.setSSLSocketFactory(factory);
connection.setHostnameVerifier(new Verifier());
if (method == RequestMethod.POST)
{
connection.setDoOutput(true);
connection.setRequestMethod("POST");
}
else
{
connection.setDoInput(true);
connection.setRequestMethod("GET");
}
msg = connection.getResponseMessage();
error = connection.getResponseCode();
if ("OK".equals(msg))
{
InputStream content = (InputStream) connection.getContent();
responce = convertStreamToString(content);
}
else
{
responce = "Error " + error;
}
connection.disconnect();
} catch (Exception e) {
responce = e.toString();
}
return responce;
}

Session cookies - CookieContainer on stack rather than heap causing issue

I've had a look here
C# WebRequest using Cookies
Multiple WebRequest in same session
Reuse Connection with HttpWebRequest in C#
C# keep session id over httpwebrequest
And that's what I'm doing except I wish to store my CookieContainer as a member (named session_cookie) in my class called connector. My problem is that if I use a temporary object in my code then the cookies work fine:
CookieContainer t = new CookieContainer();
HTTPReq = (HttpWebRequest)WebRequest.Create(scriptURL);
HTTPReq.CookieContainer = t;
But if I use
HTTPReq = (HttpWebRequest)WebRequest.Create(scriptURL);
HTTPReq.CookieContainer = session_cookie;
Then it doesn't work! I cannot figure out why
Here is the connector class code:
public class Connector
{
public CookieContainer session_cookie;
private string session_id;
private HttpWebRequest HTTPReq;
private HttpWebResponse Response;
//Session oriented connection
public string serverRequest(string scriptURL, string payLoad)
{
try
{
HTTPReq = (HttpWebRequest)WebRequest.Create(scriptURL);
HTTPReq.CookieContainer = session_cookie;
HTTPReq.Method = "POST";
//Data arguments
byte[] byteArray = Encoding.UTF8.GetBytes(payLoad);
HTTPReq.ContentType = "application/x-www-form-urlencoded";
HTTPReq.ContentLength = byteArray.Length;
//Get the stream to write into
Stream dataStream = HTTPReq.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
Response = (HttpWebResponse)HTTPReq.GetResponse();
Encoding enc = Encoding.GetEncoding(1252); // Western latin alphabet (windows default)
//Get the repsonse from the server
StreamReader ResponseStream = new StreamReader(Response.GetResponseStream(), enc);
string response = ResponseStream.ReadToEnd().Trim();
Response.Close();
ResponseStream.Close();
return response;
}
catch (WebException ex)
{
Console.WriteLine(ex.ToString());
return null;
}
}
}
Any ideas?
You are missing a constructor for your Connector class which needs to initialize your CookieContainer property. You should also use a CookieCollection for capturing the cookies from the response and sending to the next request. Something like this should work:
public class Connector
{
public CookieContainer session_cookie;
private CookieCollection cookies;
private string session_id;
private HttpWebRequest HTTPReq;
private HttpWebResponse Response;
public Connector()
{
session_cookie = new CookieContainer();
cookies = new CookieCollection();
}
//Session oriented connection
public string serverRequest(string scriptURL, string payLoad)
{
try
{
HTTPReq = (HttpWebRequest)WebRequest.Create(scriptURL);
HTTPReq.CookieContainer = session_cookie;
HTTPReq.CookieContainer.Add(cookies);
HTTPReq.Method = "POST";
//Data arguments
byte[] byteArray = Encoding.UTF8.GetBytes(payLoad);
HTTPReq.ContentType = "application/x-www-form-urlencoded";
HTTPReq.ContentLength = byteArray.Length;
//Get the stream to write into
Stream dataStream = HTTPReq.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
Response = (HttpWebResponse)HTTPReq.GetResponse();
cookies = Response.Cookies; // capture cookies from response for next request
Encoding enc = Encoding.GetEncoding(1252); // Western latin alphabet (windows default)
//Get the repsonse from the server
StreamReader ResponseStream = new StreamReader(Response.GetResponseStream(), enc);
string response = ResponseStream.ReadToEnd().Trim();
Response.Close();
ResponseStream.Close();
return response;
}
catch (WebException ex)
{
Console.WriteLine(ex.ToString());
return null;
}
}
}

Proxy Username/Password in Apache HttpClient

I'm looking to perform a GET on the yahoo currency rate service via Apache HttpClient 4.1.2, but I'm getting an UknownHostException when I'm accessing via company firewall. The code works fine when I try it from home(without any proxy config, of course), though.
Also, the URL opens on my browser, but can't be pinged from command prompt.
A sample URL is http://quote.yahoo.com/d/quotes.csv?f=l1&s=USDINR=X
EDIT 2: Here's the complete code I used to connect to the Yahoo finance service:
GetRate.java
public class GetRate {
public static void main(String[] args) {
final String FROM = "USD";
final String TO = "INR";
ArrayList<String> paramsList = new ArrayList<String>();
paramsList.add(FROM + TO);
System.out.println("Tracking "+ TO + " vs. " + FROM + " Exchange Rate...");
try {
double _new = new Double(RestClient.doGet(paramsList));
double _old = _new;
while(true) {
_new = new Double(RestClient.doGet(paramsList));
if(_old != _new)
_old = _new;
Thread.sleep(1000);
}
} catch (HttpException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (URISyntaxException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
RestClient.java
public class RestClient {
public static final int HTTP_OK = 200;
public static final String SERVER_URL = "http://quote.yahoo.com/d/quotes.csv";
public static final String DEFAULT_ENCODING = "UTF-8";
public static String doGet(final ArrayList<String> params) throws HttpException,
IOException, URISyntaxException {
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpConnectionParams
.setConnectionTimeout(httpClient.getParams(), 10000);
httpClient = proxyConfig(httpClient);
HttpHost targetHost = new HttpHost(SERVER_URL);
String urlParams = "?f=l1";
if(!params.isEmpty()) {
for(String param : params) {
String paramString = "s=" + URLEncoder.encode(param, DEFAULT_ENCODING) + "=X";
urlParams += (urlParams.length() > 1) ? ("&" + paramString) : paramString;
}
}
HttpGet httpget = new HttpGet(urlParams);
System.out.println("Final URL: " + httpget.getURI().toString());
HttpResponse response = httpClient.execute(targetHost, httpget);
HttpEntity entity = response.getEntity();
InputStream instream = entity.getContent();
return read(instream);
}
private static String read(InputStream in) throws IOException {
StringBuilder sb = new StringBuilder();
BufferedReader r = new BufferedReader(new InputStreamReader(in), 1000);
for (String line = r.readLine(); line != null; line = r.readLine()) {
sb.append(line + ",");
}
in.close();
return sb.toString().substring(0, sb.length() - 1);
}
/** Proxy config Approach 1 */
private static DefaultHttpClient proxyConfig(DefaultHttpClient httpClient) {
AuthScope auth = new AuthScope("proxy.tcs.com", 8080);
Credentials creds = new UsernamePasswordCredentials("USER_NAME", "PASSWORD");
httpClient.getCredentialsProvider().setCredentials(auth, creds);
return httpClient;
}
} for(String param : params) {
String paramString = "s=" + URLEncoder.encode(param, DEFAULT_ENCODING) + "=X";
urlParams += (urlParams.length() > 1) ? ("&" + paramString) : paramString;
}
}
HttpGet httpget = new HttpGet(urlParams);
HttpResponse response = httpClient.execute(targetHost, httpget);
HttpEntity entity = response.getEntity();
InputStream instream = entity.getContent();
Approach 2: I also tried the following proxy config, but couldn't find out how to add the username/password.
/** Proxy config Approach 2 */
HttpHost proxy = new HttpHost("PROXY_HOST", PROXY_PORT);
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
Thanks,
Debojit
EDIT 1:
Stacktrace for Approach 1:
java.net.UnknownHostException: http://quote.yahoo.com/d/quotes.csv
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(Unknown Source)
at java.net.InetAddress.getAddressFromNameService(Unknown Source)
at java.net.InetAddress.getAllByName0(Unknown Source)
at java.net.InetAddress.getAllByName(Unknown Source)
at java.net.InetAddress.getAllByName(Unknown Source)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:242)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:130)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:573)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:776)
at ws.client.RestClient.doGet(RestClient.java:48)
at ws.client.GetRate.main(GetRate.java:22)
Stacktrace for Approach 2:
Exception in thread "main" java.lang.NumberFormatException: For input string: "<HEAD><TITLE>Proxy Authorization Required</TITLE></HEAD>,<BODY BGCOLOR="white" FGCOLOR="black"><H1>Proxy Authorization Required</H1><HR>,<FONT FACE="Helvetica,Arial"><B>,Description: Authorization is required for access to this proxy</B></FONT>,<HR>,<!-- default "Proxy Authorization Required" response (407) -->,</BODY>,"
at sun.misc.FloatingDecimal.readJavaFormatString(Unknown Source)
at java.lang.Double.valueOf(Unknown Source)
at java.lang.Double.<init>(Unknown Source)
at ws.client.GetRate.main(GetRate.java:22)
The thing is, I'm not sure where the code is getting the HTML as input, and why.
Are you using "PROXY_HOST" literally for proxy constructor? If so, you have to use the proxy host in your browser configuration. Same for PROXY_PORT.
You will not be able to ping Yahoo from Your company because of the firewall, but you can access through your browser because it is configured to use a proxy server.

Resources