I am trying to gather and store URLs and associated metadata from the Cochrane Library review collection (https://www.cochranelibrary.com/cdsr/reviews/topics) for a project. But when I do a regular HTTP Get requests I get a timed out 419 error.
HttpResponseProxy{HTTP/1.1 419 status code 419 [Date: Thu, 08 Dec 2022 11:36:36 GMT, Content-Length: 0, Connection: keep-alive, Cf-Railgun: direct (starting new WAN connection), CF-Cache-Status: DYNAMIC, Server: cloudflare, CF-RAY: 7765344b8cb53338-EWR] [Content-Length: 0,Chunked: false]}
My only requirements for this project are java 1.7 and Apache HttpClient 4.5
This is my code to reproduce the error:
SSLContext sslContext = SSLContexts.createSystemDefault();
SSLConnectionSocketFactory sslConnectionFactory = new SSLConnectionSocketFactory(sslContext);
CloseableHttpClient httpClient = HttpClients.custom()
.setSSLSocketFactory(sslConnectionFactory)
.build();
HttpGet request = new HttpGet("https://www.cochranelibrary.com/");
CloseableHttpResponse response = httpClient.execute(request);
HttpEntity entity = response.getEntity();
System.out.println(entity);
httpClient.close();
response.close();
Related
I have a REST API with the following signature:
#POST
#Path("/bulkControlsMapping")
#Consumes({ MediaType.MULTIPART_FORM_DATA })
Response saveControlsMapping(#NonNull MultipartFormDataInput input);
and in my API I'm extracting my data from the input as:
Map<String, List<InputPart>> uploadForm = input.getFormDataMap();
// For text
InputPart inputPart = uploadForm.get(TEXT_KEY).get(0);
// For file
InputPart inputPart = uploadForm.get(CONTROLS_FILE_KEY).get(0);
I'm calling this REST API from Spring MVC web service, where I'm constructing my request as:
Client client = ClientBuilder.newClient().register(MultipartWriter.class);
String endPoint = "https://localhost:8080/";
WebTarget webTarget = client.target(endPoint).path("bulkControlsMapping");
MultipartFormDataOutput multipartFormDataOutput = new MultipartFormDataOutput();
multipartFormDataOutput.addFormData(CONTROLS_FILE_KEY,
file, MediaType.APPLICATION_OCTET_STREAM_TYPE);
multipartFormDataOutput.addFormData(TEXT_KEY, countryCode, MediaType.TEXT_PLAIN_TYPE);
Response response = webTarget.request()
.post(Entity.entity(multipartFormDataOutput, MediaType.MULTIPART_FORM_DATA));
But I'm getting 400 bad Request Exception in this due to no Content-Disposition header found within the part. What is the correct way to build this request?
[java] 03 Sep 2017 21:16:56,300 [WARN] (http-bio-0.0.0.0-8080-exec-12) org.jboss.resteasy.core.ExceptionHandler: Failed executing POST /bulkControlsMapping
[java] org.jboss.resteasy.spi.ReaderException: java.lang.RuntimeException: Could find no Content-Disposition header within part
[java] at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:183) ~[resteasy-jaxrs-3.0.10.Final.jar:?]
[java] at org.jboss.resteasy.core.MethodInjectorImpl.injectArguments(MethodInjectorImpl.java:89) ~[resteasy-jaxrs-3.0.10.Final.jar:?]
[java] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:112) ~[resteasy-jaxrs-3.0.10.Final.jar:?]
[java] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296) ~[resteasy-jaxrs-3.0.10.Final.jar:?]
[java] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250) ~[resteasy-jaxrs-3.0.10.Final.jar:?]
When I test my API with Postman Client, I could see that correct header structure should be like this, but I'm not able to build it manually:
POST /bulkControlsMapping HTTP/1.1
Host: localhost:8080
Cache-Control: no-cache
Postman-Token: 0af37ee0-9623-43b6-f5f3-bd2c01bdd84c
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="bulkMappingFile"; filename="Beta-Test-2.csv"
Content-Type: text/csv
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="countryCode"
MX
------WebKitFormBoundary7MA4YWxkTrZu0gW--
UPDATE:
I got it to work by using MultiPartFeature,
Client client = ClientBuilder.newClient().register(MultiPartFeature.class, JacksonFeature.class);
and different methods for generating a request using FormDataMultiPart and FormDataContentDisposition.
FormDataMultiPart formDataMultiPart = new FormDataMultiPart();
FormDataContentDisposition.FormDataContentDispositionBuilder builder = FormDataContentDisposition
.name("controlsBulkMappingFile");
log.error("FILENAME: " + file.getName());
builder.fileName(file.getName());
FormDataContentDisposition formDataContentDisposition = builder.build();
FileInputStream fileInputStream = null;
try {
fileInputStream = new FileInputStream(file);
formDataMultiPart.bodyPart(
new FormDataBodyPart((file.getName()), fileInputStream,
MediaType.APPLICATION_OCTET_STREAM_TYPE)
.contentDisposition(formDataContentDisposition));
} catch (FileNotFoundException ex) {
System.out.println("&&&&&&: " + ex.getLocalizedMessage());
}
// COUNTRY CODE
FormDataContentDisposition.FormDataContentDispositionBuilder builder2 = FormDataContentDisposition
.name("countryCode");
FormDataContentDisposition formDataContentDisposition2 = builder2.build();
formDataMultiPart.bodyPart(new FormDataBodyPart("countryCode", countryCode,
MediaType.TEXT_PLAIN_TYPE)).contentDisposition(formDataContentDisposition2);
log.error("MULTIPART: " + formDataMultiPart.getBodyParts().toString());
Entity<FormDataMultiPart> entity = Entity
.entity(formDataMultiPart, MediaType.MULTIPART_FORM_DATA_TYPE);
Response response = webTarget.request().accept(MediaType.APPLICATION_JSON)
.post(entity);
log.error("BULK MAPPING: Response 1: {}", response);
BulkControlsMappingResponse mappingResponse = response
.readEntity(BulkControlsMappingResponse.class);
My API consumes MediaType.MULTIPART_FORM_DATA and produces MediaType.APPLICATION_JSON. Now when I use the MultiPartFeature alone, I'm able to send the request but not able to parse the response. I'm getting
[tomcat:launchProperties] org.glassfish.jersey.message.internal.MessageBodyProviderNotFoundException: MessageBodyWriter not found for media type=multipart/form-data, type=class org.glassfish.jersey.media.multipart.FormDataMultiPart, genericType=class org.glassfish.jersey.media.multipart.FormDataMultiPart.
I got this to work by using different features while sending the request, and different for response as follows:
Client client = ClientBuilder.newClient().register(MultiPartFeature.class);
WebTarget webTarget = client.target(endPoint).path("bulkControlsMapping")
.register(JacksonFeature.class);
I am trying to POST data to a server who is just using TLS1.2
When i am running the code i am getting the following response from server.
HttpResponseProxy{HTTP/1.1 415 Unsupported Media Type [Date: Wed, 07 Sep 2016 17:42:57 CST, Server: , Strict-Transport-Security: max-age=63072000; includeSubdomains; preload, Pragma: no-cache, Content-Length: 0, charset: ISO-8859-1, X-ORACLE-DMS-RID: 0, X-ORACLE-DMS-ECID: 343fa6c0-ed24-4003-ad58-342caf000404-00000383, Set-Cookie: JSESSIONID=1ZMIvt1NqrtWpHgHs4mMmYyTPUGTOQgrA9biCE3Dok5v0gDCPXu6!681252631; path=/; secure; HttpOnly;HttpOnly;Secure, Cache-Control: no-store, P3P: policyref="/w3c/p3p.xml", CP="WBP DSP NOR AMT ADM DOT URT POT NOT", Keep-Alive: timeout=5, max=250, Connection: Keep-Alive, Content-Type: text/xml, Content-Language: en] [Content-Type: text/xml,Content-Length: 0,Chunked: false]}
I am using the below code to post the data to server . I am using apache httpcomponents-client-4.5.2.
private static Registry<ConnectionSocketFactory> getRegistry() throws KeyManagementException, NoSuchAlgorithmException {
SSLContext sslContext = SSLContexts.custom().build();
SSLConnectionSocketFactory sslConnectionSocketFactory = new SSLConnectionSocketFactory(sslContext,
new String[]{"TLSv1.2"}, null, SSLConnectionSocketFactory.getDefaultHostnameVerifier());
return RegistryBuilder.<ConnectionSocketFactory>create()
.register("https", sslConnectionSocketFactory)
.register("https", sslConnectionSocketFactory)
.build();
}
public static void main(String[] args) throws Exception
{
PoolingHttpClientConnectionManager clientConnectionManager = new PoolingHttpClientConnectionManager(getRegistry());
clientConnectionManager.setMaxTotal(100);
clientConnectionManager.setDefaultMaxPerRoute(20);
HttpClient client = HttpClients.custom().setConnectionManager(clientConnectionManager).build();
HttpPost request = new HttpPost("https://someserver.com/dataupload");
File file = new File("C://Nible//code//_client//Request.xml");
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
builder.setContentType(ContentType.TEXT_XML);
FileBody fileBody = new FileBody(file);
builder.addPart("my_file", fileBody);
HttpEntity reqEntity = builder.build();
request.setEntity(reqEntity);
HttpResponse response = client.execute(request);
System.out.println(response);
}
Can you please tell me what wrong am i doing?
I tried using below code instead of MultipartEntityBuilder. I am still getting the same error.
EntityBuilder builder = EntityBuilder.create();
builder.setFile(file);
builder.setContentType(ContentType.TEXT_XML);
If i am sending the BLANK REQUEST to server then also i am getting the same error. Blank error means i am not putting any thing in request just
HttpPost request = new HttpPost("https://someserver.com/dataupload");
HttpResponse response = client.execute(request);
I suspect of these lines in your code:
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
builder.setContentType(ContentType.TEXT_XML);
Multipart entities cannot be of content-type xml. They must be one of these types:
multipart/mixed
multipart/alternative
multipart/digest
multipart/parallel
(See RFC 1341 7.2)
I guess you should use one of these content-types for the multipart entity, and set text/xml as the content type of the single part:
FileBody fileBody = new FileBody(file, ContentType.TEXT_XML);
(Another issue is that I don't see necessary to send a multipart for just one file: You could leave out the MultipartEntityBuilder object and build directly a FileEntity.)
The scenario I am talking is local debugging scenario as I have not deployed to azure yet.
I have a MVC Web App authenticating against AAD that calls WebAPI
I have the right permission set in the AAD tenant, the reason I say it set correctly because I am able to get the AccessToken for the WebAPI, but then when I try to call the WebAPI it gives me the following error
{StatusCode: 401, ReasonPhrase: 'Unauthorized', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Pragma: no-cache
X-SourceFiles: =?UTF-8?B?xxxxxxxxxx
Cache-Control: no-cache
Date: Sun, 27 Mar 2016 02:06:31 GMT
Server: Microsoft-IIS/10.0
WWW-Authenticate: Bearer
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Content-Length: 61
Content-Type: application/json; charset=utf-8
Expires: -1
}}
Code
string userObjectID = ClaimsPrincipal.Current.FindFirst("http://schemas.microsoft.com/identity/claims/objectidentifier").Value;
NaiveSessionCache nsc = new NaiveSessionCache(userObjectID);
AuthenticationContext authContext = new AuthenticationContext(Startup.Authority, nsc);
ClientCredential credential = new ClientCredential(clientId, appKey);
result = authContext.AcquireTokenSilent(todoListClientId, credential, new UserIdentifier(userObjectID, UserIdentifierType.UniqueId));
Uri webApiUri = new Uri(string.Format("{0}{1}", todoListBaseAddress, "/api/TodoList"));
string accessTknString = Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(result.AccessToken));
HttpClient client = new HttpClient();
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, webApiUri);
request.Headers.Add(System.Net.HttpRequestHeader.Authorization.ToString(), string.Format("Bearer {0}", accessTknString));
HttpResponseMessage response = await client.SendAsync(request);
final WebClient webClient = new WebClient();
final HtmlPage page = webClient.getPage(webPageURL);
final String pageAsXml = page.asXml();
final String pageAsText = page.asText();
List <NameValuePair> response = page.getWebResponse().getResponseHeaders();
for (NameValuePair header : response) {
log.info(header.toString() + " = " + header.getValue());
The web page returns more than one header. But the log shows only the first header. How do I get rest of headers? The header I am looking for is of Content Type: application/javascript;charset=ISO-8859-1
The web page is an internal web page.
The code you provided is working for me. Actually, I replaced the header.toString() for header.getName():
final WebClient webClient = new WebClient();
final HtmlPage page = webClient.getPage("http://www.debian.org");
List<NameValuePair> response = page.getWebResponse().getResponseHeaders();
for (NameValuePair header : response) {
System.out.println(header.getName() + " = " + header.getValue());
}
The output is:
Date = Thu, 22 Aug 2013 19:48:54 GMT
Server = Apache
Content-Location = index.en.html
Vary = negotiate,accept-language,Accept-Encoding
TCN = choice
Last-Modified = Thu, 22 Aug 2013 15:31:17 GMT
ETag = "3887-4e48afb257b40"
Accept-Ranges = bytes
Cache-Control = max-age=86400
Expires = Fri, 23 Aug 2013 19:48:54 GMT
Content-Encoding = gzip
Content-Length = 4605
Keep-Alive = timeout=15, max=100
Connection = Keep-Alive
Content-Type = text/html
Content-Language = en
As you can see, the Content-Type header is there. Can you confirm the server is actually sending that piece of data (it is a common header so it should).
I have problems when trying to connect to the internet via a proxy using the new httocomponent-client module
If I use directly the Proxy object and HttpURLConnection everything goes fine:
URL u = new URL("http://www.google.com");
Proxy proxy = new Proxy(Type.HTTP, new InetSocketAddress("somehost", 8080));
HttpURLConnection con = (HttpURLConnection) u.openConnection(proxy);
con.setRequestMethod("GET");
System.out.println(con.getResponseCode());
Now I try to do the same with the new api:
HttpHost proxy = new HttpHost("somehost", 8080, "http");
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
HttpHost targetHost = new HttpHost("http://www.google.com");
HttpGet httpGet = new HttpGet("/");
try {
HttpResponse httpResponse = httpClient.execute(targetHost, httpGet);
System.out.println(httpResponse.toString());
} catch (Exception e) {
e.printStackTrace();
}
but I get this:
HTTP/1.1 407 Proxy Authentication Required ( Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied. ) [Via: 1.1 xxx, Proxy-Authenticate: Negotiate, Proxy-Authenticate: Kerberos, Proxy-Authenticate: NTLM, Connection: Keep-Alive, Proxy-Connection: Keep-Alive, Pragma: no-cache, Cache-Control: no-cache, Content-Type: text/html, Content-Length: 7079 ]
I also tried
ProxySelectorRoutePlanner routePlanner = new ProxySelectorRoutePlanner(
httpClient.getConnectionManager().getSchemeRegistry(),new MyProxySelector());
httpClient.setRoutePlanner(routePlanner);
Where MyProxySelector return the Proxy that I nned but no result.
Why using the new API makes the proxy authentication required inside the code?
I don't know why the solution with ProxySelectorRoutePlanner doesn't work, are you sure you start your JVM with the proxy settings ?
It looks like you need to add this line :
httpClient.getCredentialsProvider().setCredentials(new AuthScope("yourProxyHost", Port),
new UsernamePasswordCredentials("yourUsername", "yourPass"));