I am trying to gather and store URLs and associated metadata from the Cochrane Library review collection (https://www.cochranelibrary.com/cdsr/reviews/topics) for a project. But when I do a regular HTTP Get requests I get a timed out 419 error.
HttpResponseProxy{HTTP/1.1 419 status code 419 [Date: Thu, 08 Dec 2022 11:36:36 GMT, Content-Length: 0, Connection: keep-alive, Cf-Railgun: direct (starting new WAN connection), CF-Cache-Status: DYNAMIC, Server: cloudflare, CF-RAY: 7765344b8cb53338-EWR] [Content-Length: 0,Chunked: false]}
My only requirements for this project are java 1.7 and Apache HttpClient 4.5
This is my code to reproduce the error:
SSLContext sslContext = SSLContexts.createSystemDefault();
SSLConnectionSocketFactory sslConnectionFactory = new SSLConnectionSocketFactory(sslContext);
CloseableHttpClient httpClient = HttpClients.custom()
.setSSLSocketFactory(sslConnectionFactory)
.build();
HttpGet request = new HttpGet("https://www.cochranelibrary.com/");
CloseableHttpResponse response = httpClient.execute(request);
HttpEntity entity = response.getEntity();
System.out.println(entity);
httpClient.close();
response.close();
I am calling a web service using Spring WebClient (Spring 5.1.3). The service responds with content-type: application/json and content-encoding: gzip
ClientResponse.bodyToMono then fails with the error "JSON decoding error: Illegal character ((CTRL-CHAR, code 31))" which I assume is because the content has not been decoded before trying to parse the JSON.
Here is code snippet (simplified) of how I create the WebClient
HttpClient httpClient = HttpClient.create().secure(sslContextSpec -> sslContextSpec.sslContext(sslContext));
return WebClient.builder().clientConnector(new ReactorClientHttpConnector(httpClient)).build();
I then use the WebClient to make the call:
webClient.get().uri(uri)
.accept(MediaType.APPLICATION_JSON)
.header(HttpHeaders.ACCEPT_ENCODING, "gzip")
.exchange()
The HTTP request has 2 headers:
Accept: application/json
Accept-Encoding: gzip
The response has the following headers:
set-cookie: xxx
content-type: application/json; charset=utf-8
content-length: 1175
content-encoding: gzip
cache-control: no-store, no-cache
By doing the following I am able to manually decode the GZIP content and get valid JSON from the result
webClient.get().uri(uri)
.accept(MediaType.APPLICATION_JSON)
.header("accept-encoding", "gzip")
.exchange()
.flatMap(encodedResponse -> encodedResponse.body((inputMessage, context) ->
inputMessage.getBody().flatMap(dataBuffer -> {
ClientResponse.Builder decodedResponse = ClientResponse.from(encodedResponse);
try {
GZIPInputStream gz = new GZIPInputStream(dataBuffer.asInputStream());
decodedResponse.body(new String(gz.readAllBytes()));
} catch (IOException e) {
e.printStackTrace();
}
decodedResponse.headers(headers -> {
headers.remove("content-encoding");
});
return Mono.just(decodedResponse.build());
}).flatMap(clientResponse -> clientResponse.bodyToMono(Map.class))
This feature is supported natively by the reactor netty client.
You should create HttpClient like this:
HttpClient httpClient = HttpClient.create()
.secure(sslContextSpec -> sslContextSpec.sslContext(sslContext))
.compress(true);
And then there's no need to add the accept encoding request header since it's done for you.
Note that this bit is done by the connector itself when you don't provide a custom HttpClient instance.
I have a client Angular 2 application, and a Spring boot (1.4.1.RELEASE) server application. I wrote the client code to send a connection request to the server using the angular library ng2-stomp-service, and the server code to manage the connection request.
All works fine when using http, but when using https I see a connection error (HTTP/1.1 400) in the javascript console:
GET https://tomcatunisvid.lunagest.com/unisvid/workflow/046/mmesis4u/websocket [HTTP/1.1 400 358ms]
Firefox non può stabilire una connessione con il server wss://tomcatunisvid.lunagest.com/unisvid/workflow/046/mmesis4u/websocket.
In Firefox I found the request and the response headers:
-- REQUEST HEADERS
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encodinggzip, deflate, br
Accept-Language it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3
Cache-Control no-cache
Connection keep-alive, Upgrade
Host tomcatunisvid.lunagest.com
Origin https://tomcatunisvid.lunagest.com
Pragma no-cache
Sec-WebSocket-Extensions permessage-deflate
Sec-WebSocket-Key J2Aqm3n2r6vcE7F8SrPr3w==
Sec-WebSocket-Version 13
Upgrade websocket
User-AgentMozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:56.0) Gecko/20100101 Firefox/56.0
-- RESPONSE HEADERS
Access-Control-Allow-Credentials true
Access-Control-Allow-Originh ttps://tomcatunisvid.lunagest.com
Cache-Control no-cache, no-store, max-age=0, must-revalidate
Connection close
Date Mon, 06 Nov 2017 11:50:42 GMT
Expires 0
Pragma no-cache
Server Apache/2.4.18 (Ubuntu)Transfer-Encoding chunked
Vary Origin
X-Content-Type-Options nosniff
X-Frame-Options DENY
X-XSS-Protection1; mode=block
and the response is empty.
Here is my relevant client code:
// In the constructor, where stomp is a StompService:
stomp.configure({
host: this.url,
debug: true,
queue: { 'init': false, '/user/queue/user/reply': false },
// headers: { 'upgrade': 'WebSocket' }
});
// in another method:
this.stomp.startConnect().
then(() => this.openSocket_OK_Callback(this)).
catch((error) => this.openSocket_KO_Callback(this, error));
Here is my relevant server code:
#Configuration
#EnableWebSocketMessageBroker
public class WebSocketConfig extends
AbstractWebSocketMessageBrokerConfigurer {
#Override
public void configureMessageBroker(MessageBrokerRegistry config) {
config.setApplicationDestinationPrefixes("/wfm");
config.enableSimpleBroker("/topic", "/queue");
}
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
SessionDisconnectionListener sdl = new SessionDisconnectionListener();
String endpoint = "/workflow";
registry.addEndpoint(endpoint).
setAllowedOrigins("*").withSockJS();
registry.addEndpoint(endpoint).setHandshakeHandler(new DefaultHandshakeHandler(new TomcatRequestUpgradeStrategy())).
setAllowedOrigins("*").withSockJS();
}
}
Any hints how to solve this issue?
I put here the answer I found after some debugging, in case it is useful to others. In the release environment Tomcat was proxied by Apache Http Server. The solution in this case was to remove Apache Http Server and configure Https directly on Tomcat.
I am trying to POST data to a server who is just using TLS1.2
When i am running the code i am getting the following response from server.
HttpResponseProxy{HTTP/1.1 415 Unsupported Media Type [Date: Wed, 07 Sep 2016 17:42:57 CST, Server: , Strict-Transport-Security: max-age=63072000; includeSubdomains; preload, Pragma: no-cache, Content-Length: 0, charset: ISO-8859-1, X-ORACLE-DMS-RID: 0, X-ORACLE-DMS-ECID: 343fa6c0-ed24-4003-ad58-342caf000404-00000383, Set-Cookie: JSESSIONID=1ZMIvt1NqrtWpHgHs4mMmYyTPUGTOQgrA9biCE3Dok5v0gDCPXu6!681252631; path=/; secure; HttpOnly;HttpOnly;Secure, Cache-Control: no-store, P3P: policyref="/w3c/p3p.xml", CP="WBP DSP NOR AMT ADM DOT URT POT NOT", Keep-Alive: timeout=5, max=250, Connection: Keep-Alive, Content-Type: text/xml, Content-Language: en] [Content-Type: text/xml,Content-Length: 0,Chunked: false]}
I am using the below code to post the data to server . I am using apache httpcomponents-client-4.5.2.
private static Registry<ConnectionSocketFactory> getRegistry() throws KeyManagementException, NoSuchAlgorithmException {
SSLContext sslContext = SSLContexts.custom().build();
SSLConnectionSocketFactory sslConnectionSocketFactory = new SSLConnectionSocketFactory(sslContext,
new String[]{"TLSv1.2"}, null, SSLConnectionSocketFactory.getDefaultHostnameVerifier());
return RegistryBuilder.<ConnectionSocketFactory>create()
.register("https", sslConnectionSocketFactory)
.register("https", sslConnectionSocketFactory)
.build();
}
public static void main(String[] args) throws Exception
{
PoolingHttpClientConnectionManager clientConnectionManager = new PoolingHttpClientConnectionManager(getRegistry());
clientConnectionManager.setMaxTotal(100);
clientConnectionManager.setDefaultMaxPerRoute(20);
HttpClient client = HttpClients.custom().setConnectionManager(clientConnectionManager).build();
HttpPost request = new HttpPost("https://someserver.com/dataupload");
File file = new File("C://Nible//code//_client//Request.xml");
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
builder.setContentType(ContentType.TEXT_XML);
FileBody fileBody = new FileBody(file);
builder.addPart("my_file", fileBody);
HttpEntity reqEntity = builder.build();
request.setEntity(reqEntity);
HttpResponse response = client.execute(request);
System.out.println(response);
}
Can you please tell me what wrong am i doing?
I tried using below code instead of MultipartEntityBuilder. I am still getting the same error.
EntityBuilder builder = EntityBuilder.create();
builder.setFile(file);
builder.setContentType(ContentType.TEXT_XML);
If i am sending the BLANK REQUEST to server then also i am getting the same error. Blank error means i am not putting any thing in request just
HttpPost request = new HttpPost("https://someserver.com/dataupload");
HttpResponse response = client.execute(request);
I suspect of these lines in your code:
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
builder.setContentType(ContentType.TEXT_XML);
Multipart entities cannot be of content-type xml. They must be one of these types:
multipart/mixed
multipart/alternative
multipart/digest
multipart/parallel
(See RFC 1341 7.2)
I guess you should use one of these content-types for the multipart entity, and set text/xml as the content type of the single part:
FileBody fileBody = new FileBody(file, ContentType.TEXT_XML);
(Another issue is that I don't see necessary to send a multipart for just one file: You could leave out the MultipartEntityBuilder object and build directly a FileEntity.)
I'm currently working on site that uses various Ajax-requests to save, load and autocomplete data. It is build using C#, MVC and JQuery. All actions on the MVC controllers require the users to be authorized, and we use IdentityServer3 for authentication. It was installed using NuGet, and the current version is 2.3.0.
When I open the page and push buttons, everything is working just fine. The problem seem to occur when a certain session expires. If I stay idle for a while, and try to use an Ajax-function, it generates the following error:
XMLHttpRequest cannot load https://identityserver.domain.com/connect/authorize?client_id=Bar&redirect_uri=http%3a%2f%2flocalhost%3a12345&response_mode=form_post&response_type=id_token+token&scope=openid+profile+email+phone+roles [...]. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:12345' is therefore not allowed access.
From what I know about Ajax, the problem itself is pretty simple. The MVC site has lost track of the current session, and it is asking the client to authenticate again. The response I get from the Ajax-request is a "302 Found", with a Location-header that points to our IdentityServer. The IdentityServer happens to be on another domain, and while this works fine when you are performing regular HTTP-requests, it does not work particularly well for Ajax-requests. The "Same Origin Policy" is straight up blocking the Ajax-function from authenticating. If I refresh the page, I will be redirected to the IdentityServer and authenticate normally. Things will then go back to normal for a few minutes.
The solution is probably to add an extra header in the response message from the IdentityServer, that explicitly states that cross-origin requests are allowed for this service.
I am currently not getting this header from the IdentityServer (checked in Fiddler).
According to the docs, it should be enabled by default. I have checked that we have indeed enabled CORS this way:
factory.CorsPolicyService = new Registration<ICorsPolicyService>(new DefaultCorsPolicyService { AllowAll = true });
This is one of my clients:
new Client
{
Enabled = true,
ClientName = "Foo",
ClientId = "Bar",
ClientSecrets = new List<Secret>
{
new Secret("Cosmic")
},
Flow = Flows.Implicit,
RequireConsent = false,
AllowRememberConsent = true,
AccessTokenType = AccessTokenType.Jwt,
PostLogoutRedirectUris = new List<string>
{
"http://localhost:12345/",
"https://my.domain.com"
},
RedirectUris = new List<string>
{
"http://localhost:12345/",
"https://my.domain.com"
},
AllowAccessToAllScopes = true
}
These settings do not work. I am noticing that I have an extra forward slash in the URIs here, but if I remove them, I get the default IdentityServer-error that states that the client is not authorized (wrong URI). If I deploy the site (instead of running a localhost debug), I use the domain name without a trailing slash, and I get the exact same behaviour as I do in debug. I do notice that there is no trailing slash in the error message above, and I figured this could be the problem until I saw the same thing in the deployed version of the site.
I also made my own policy provider, like this:
public class MyCorsPolicyService : ICorsPolicyService
{
public Task<bool> IsOriginAllowedAsync(string origin)
{
return Task.FromResult(true);
}
}
... and I plugged it into the IdentityServerServiceFactory like this:
factory.CorsPolicyService = new Registration<ICorsPolicyService>(new MyCorsPolicyService());
The idea is for it to return true regardless of origin. This did not work either; exactly the same results as before.
I've read about a dozen other threads on this particular subject, but I'm getting nowhere. To my knowledge, we are not doing anything unusual when it comes to the setup of the different sites. It's all pretty much out-of-the-box. Any advice?
----- UPDATE -----
The problem persists. I have now tried some fresh tactics. I read somewhere that cookie authentication was bad for Ajax-requests, and that I should be using bearer tokens instead. I set this up in Ajax like this:
$(function () {
$(document).ajaxSend(function (event, request, settings) {
console.log("Setting bearer token.");
request.setRequestHeader("Authorization", "Bearer " + $bearerToken);
});
});
Both the console in Chrome and Fiddler confirms that the token is indeed present and sent by JQuery. The token I use comes from the access_token-property on claims principal object from HttpContext.GetOwinContext().Authentication.User.
This didn't do much. I still get a 302-response from the server, and Fiddler reveals that the token is not sent on the following Ajax-request (which is a GET-request) to the IdentityServer.
From there, I read this thread:
Handling CORS Preflight requests to ASP.NET MVC actions
I tried to put this code in to the startup.cs of the IdentityServer, but there does not appear to be a "preflight" request going in. All I see in Fiddler is this (from the beginning):
1 - The initial Ajax-request from the client to the MVC controller:
POST http://localhost:12345/my/url HTTP/1.1
Host: localhost:12345
Connection: keep-alive
Content-Length: pretty long
Authorization: Bearer <insert long token here>
Origin: http://localhost:12345
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Accept: application/json, text/javascript, */*; q=0.01
X-Requested-With: XMLHttpRequest
Referer: http://localhost:12345/my/url
Accept-Encoding: gzip, deflate
Accept-Language: nb-NO,nb;q=0.8,no;q=0.6,nn;q=0.4,en-US;q=0.2,en;q=0.2
Cookie: OpenIdConnect.nonce.<insert 30 000 lbs of hashed text here>
param=fish&morestuff=salmon&crossDomain=true
2 - The redirect response from the MVC controller:
HTTP/1.1 302 Found
Cache-Control: private
Location: https://identityserver.domain.com/connect/authorize?client_id=Bar&redirect_uri=http%3a%2f%2flocalhost%3a12345%2f&response_mode=form_post&response_type=id_token+token&scope=openid+profile+email [...]
Server: Microsoft-IIS/10.0
X-AspNetMvc-Version: 5.2
X-AspNet-Version: 4.0.30319
Set-Cookie: OpenIdConnect.nonce.<lots of hashed text>
X-SourceFiles: <more hashed text>
X-Powered-By: ASP.NET
Date: Fri, 15 Jan 2016 12:23:08 GMT
Content-Length: 0
3 - The Ajax-request to the IdentityServer:
GET https://identityserver.domain.com/connect/authorize?client_id=Bar&redirect_uri=http%3a%2f%2flocalhost%3a12345%2f&response_mode=form_post&response_type=id_token+token&scope=openid+profile+email [...]
Host: identityserver.domain.com
Connection: keep-alive
Accept: application/json, text/javascript, */*; q=0.01
Origin: http://localhost:12345
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://localhost:12345/my/url
Accept-Encoding: gzip, deflate, sdch
Accept-Language: nb-NO,nb;q=0.8,no;q=0.6,nn;q=0.4,en-US;q=0.2,en;q=0.2
4 - The response from IdentityServer3
HTTP/1.1 302 Found
Content-Length: 0
Location: https://identityserver.domain.com/login?signin=<some hexadecimal id>
Server: Microsoft-IIS/8.5
Set-Cookie: SignInMessage.<many, many, many hashed bytes>; path=/; secure; HttpOnly
X-Powered-By: ASP.NET
Date: Fri, 15 Jan 2016 12:23:11 GMT
5 - The meltdown of Chrome
XMLHttpRequest cannot load https://identityserver.domain.com/connect/authorize?client_id=Bar&blahblahblah. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:12345' is therefore not allowed access.
I was having a similar issue using OWIN Middleware for OpenIDConnect with a different identity provider. However, the behavior occurred after 1 hour instead of 5 minutes. The solution was to check if the request was an AJAX request, and if so, force it to return 401 instead of 302. Here is the code that performed this:
app.UseOpenIdConnectAuthentication(new OpenIdConnectAuthenticationOptions
{
ClientId = oktaOAuthClientId,
Authority = oidcAuthority,
RedirectUri = oidcRedirectUri,
ResponseType = oidcResponseType,
Scope = oauthScopes,
SignInAsAuthenticationType = "Cookies",
UseTokenLifetime = true,
Notifications = new OpenIdConnectAuthenticationNotifications
{
AuthorizationCodeReceived = async n =>
{
//...
},
RedirectToIdentityProvider = n => //token expired!
{
if (IsAjaxRequest(n.Request))
{
n.Response.StatusCode = 401;//for web api only!
n.Response.Headers.Remove("Set-Cookie");
n.State = NotificationResultState.HandledResponse;
}
return Task.CompletedTask;
},
}
});
Then, I used an Angular interceptor to detect a statusCode of 401, and redirected to the authentication page.
I came across this problem as well and UseTokenLifetime = false was not solving the problem since you loose the token validity on STS.
When I tried to reach the authorized api method, I still got 401 even if I was valid on Owin.
The solution I found is keeping UseTokenLifetime = true as default but to write a global ajax error handler (or angular http interceptor) something like this:
$.ajaxSetup({
global: true,
error: function(xhr, status, err) {
if (xhr.status == -1) {
alert("You were idle too long, redirecting to STS") //or something like that
window.location.reload();
}
}});
to trigger the authentication workflow.
I had this issue recently, it was caused by the header X-Requested-With being sent with the AJAX request. Removing this header or intercepting it and handling it with a 401 will put you on the right track.
If you don't have this header, the issue is most likely being caused by a different header triggering the Access-Control-Allow-Origin response.
As you found, nothing you do in Identity Server regarding CORS will solve this.
As it turns out, the problem was in the client configuration in MVC. I was missing the UseTokenLifetime property, which should have been set to false.
app.UseOpenIdConnectAuthentication(
new OpenIdConnectAuthenticationOptions
{
ClientId = "Bar",
Scope = "openid profile email phone roles",
UseTokenLifetime = false,
SignInAsAuthenticationType = "Cookies"
[...]
For some reason, IdentityServer sets all these cookies to expire within 5 minutes of them being distributed. This particular setting will override IdentityServer's tiny expiration time, and instead use aprox. 10 hours, or whatever the default is in your client application.
One could say that this is good enough for solving the problem. It will however inevitably return if the user decides to spend 10 hours idling on the site, clicking nothing but Ajax-buttons.
https://github.com/IdentityServer/IdentityServer3/issues/2424
Assumptions:
.NET Framework 4.8 WebForms
OWIN-based auth lib i.e. Microsoft.Owin.Security.OpenIdConnect v4.2.2.0
UseOpenIdConnectAuthentication() with Azure AD endpoint
UseTokenLifetime=true
In Layout.Master:
$.ajaxSetup({
global: true,
error: function (xhr, status, err) {
if (xhr.status == 401) {
window.location.reload();
}
}
});
In startup.cs:
app.UseOpenIdConnectAuthentication(new OpenIdConnectAuthenticationOptions
{
...
Notifications = new OpenIdConnectAuthenticationNotifications()
{
...
RedirectToIdentityProvider = RedirectToIdentityProvider
}
});
...
public Task RedirectToIdentityProvider(RedirectToIdentityProviderNotification<OpenIdConnectMessage, OpenIdConnectAuthenticationOptions> context)
{
if (IsAjaxRequest(context.Request))
{
context.Response.StatusCode = 401;
context.Response.Headers.Remove("Set-Cookie");
context.State = NotificationResultState.HandledResponse;
}
}
public bool IsAjaxRequest(this IOwinRequest request)
{
if (request == null)
{
throw new ArgumentNullException("Woopsie!");
}
var context = HttpContext.Current;
var isCallbackRequest = false;
if (context != null && context.CurrentHandler != null && context.CurrentHandler is System.Web.UI.Page page)
{
isCallbackRequest = page.IsCallback;
}
return isCallbackRequest || (request.Cookies["X-Requested-With"] == "XMLHttpRequest") || (request.Headers["X-Requested-With"] == "XMLHttpRequest");
}