I am getting this error only on my local workstation and prod server.
In Dev and Cert it is working fine.
local workstation - 20 GB memory, Win 7 64 bit, IIS Express, VS
2013 dev, cert & prod - 8 GB memory , 2008 R2 64 Bit, IIS 7.5
I have a web api (.net 4.0) which takes the incoming request body and uploads it to a storage server. configured web api as per this website.
I have these in my web.config
<system.webServer>
<security>
<requestFiltering>
<requestLimits maxAllowedContentLength="2147483648" />
</requestFiltering>
</security>
</system.webServer>
<system.web>
<httpRuntime maxRequestLength="2097152" />
</system.web>
I also have an implementation of IHostBufferPolicySelector which returns false for PUT & POST requests. so the request to this web api for PUt & POST are not buffered.
for any files < ~350 MB it is working fine. but web api is throwing out of memory exceptions when file size >= ~ 400 MB and this is happening only on Local workstation and Prod server.
Web Api controller calls below code to stream the request to the destination server
public async Task<HttpResponseMessage> StoreObjectAsync(Uri namespaceUrl, string userName, string password, string objectName, Stream objectContent, string contentType = "application/octet-stream", IDictionary<string, string> systemMetadata = null)
{
Uri namespaceRootUrl = Utilities.GetNamespaceRootUrl(namespaceUrl);
using (var request = new HttpRequestMessage() { Method = HttpMethod.Put })
{
request.RequestUri = Utilities.CreateRequestUri(namespaceRootUrl, objectName);
request.Content = new StreamContent(objectContent);
request.Content.Headers.ContentType = new MediaTypeHeaderValue(contentType);
HttpResponseMessage response;
response = await this.httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
return response;
}
}
After doing some research online, i understand from this link & this link that HttpClient on .Net 4.0 buffers the request body and because of that behavior it seemed to me that it is throwing outofmemory exception
so I changed my code to below this time using HttpWebRequest using which I have the control to specify that request should be streamed but not buffered.
public async Task<HttpResponseMessage> StoreObjectAsync(Uri namespaceUrl, string userName, string password, string objectName, Stream content, long contentLength, string contentType = "application/octet-stream", IDictionary<string, string> systemMetadata = null)
{
Uri namespaceRootUrl = Utilities.GetHCPNamespaceRootUrl(namespaceUrl);
HttpWebRequest httpWebRequest = ((HttpWebRequest)WebRequest.Create(requestUri));
httpWebRequest.Method = "PUT";
httpWebRequest.KeepAlive = true;
httpWebRequest.AllowWriteStreamBuffering = false;
httpWebRequest.ContentType = contentType;
httpWebRequest.ContentLength = contentLength;
using (Stream requestStream = await httpWebRequest.GetRequestStreamAsync())
{
await content.CopyToAsync(requestStream);
}
var webResponse = await httpWebRequest.GetResponseAsync();
HttpWebResponse httpWebResponse = (HttpWebResponse)webResponse;
Stream httpWebResponseContent = httpWebResponse.GetResponseStream();
HttpResponseMessage response = new HttpResponseMessage()
{
StatusCode = httpWebResponse.StatusCode,
ReasonPhrase = httpWebResponse.StatusDescription,
Content = new StreamContent(httpWebResponseContent)
};
return response;
}
Now it is working fine on my local machine. I am able to upload files around 1GB without any errors or memory exceptions. Havent pushed this to Prod yet.
But I still dont understand why the same code using HttpClient on .net 4.0 worked on Dev and Cert servers but not on Prod and my local.
please help me in understanding
How to find out why it worked on Dev and Cert? What system/server
configurations will affect the memory allocations to this api?
Related
I'm trying to have a controller in MVC core (3.1) download an audio file to present to the user but everything I try ends up with the error:
HttpRequestException: Received an invalid header name: 'Content=Transfer-Encoding'.
I have tried webclient, httpclient and just lately webrequest/webresponse. My code looks like this right now (specifics removed for security):
WebRequest downloadRequest = WebRequest.Create({url here});
downloadRequest.Headers.Add("Cookie", "{cookie data here};");
downloadRequest.Headers.Add("Accept", "application/json, text/javascript, */*; q=0.01");
downloadRequest.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0");
downloadRequest.Headers.Add("Sec-Fetch-Dest", "empty");
downloadRequest.Headers.Add("X-Requested-With", "XMLHttpRequest");
using (WebResponse webResponse = await downloadRequest.GetResponseAsync())
{
return File(webResponse.GetResponseStream(), "audio/wav", wavFile + ".txt");
}
The error is on the line
using (WebResponse webResponse = await downloadRequest.GetResponseAsync())
Using fiddler the request looks good and the header it is complaining about is: Content=Transfer-Encoding: binary, it has the file in the content-disposition as an attachement. I have no control at all over the sending server unfortunately but this works fine in a browser. Can anyone tell me what i'm doing wrong please?
Thanks
The incoming request looks malformed. Transfer-Encoding is its own header name. You cannot assign it to the value of another header. And AFAIK, Content isn't even a standard HTTP header name.
Refer to the list at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#Message_body_information for valid Content-* headers. The Transfer-Encoding header is documented at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding.
Edit for framework 4.8 and below
I had to add the following lines to app.config to let the client process invalid responses from the server.
<system.net>
<settings>
<httpWebRequest useUnsafeHeaderParsing="true" />
</settings>
</system.net>
The program itself is shown below.
try
{
var httpclient = new HttpClient();
var response = await httpclient.GetAsync("http://localhost:9091");
var attachment = new byte[1024];
foreach (var h in response.Content.Headers)
{
var e = h.Value.GetEnumerator();
e.MoveNext();
attachment = Encoding.UTF8.GetBytes(e.Current); // Convert UTF-8 encoded string into raw bytes
Console.WriteLine($"{h.Key}, {Encoding.UTF8.GetString(attachment)}"); // Save raw bytes to disk instead of converting back to string and writing to Console
}
}
catch (Exception exception)
{
Console.WriteLine(exception.Message);
}
Edit for dotnet core
try
{
var s = new Socket(SocketType.Stream, ProtocolType.Tcp);
var a = IPAddress.Parse("127.0.0.1");
await s.ConnectAsync(a, 9091);
var request = "GET http://localhost:9091/";
var outBytes = Encoding.UTF8.GetBytes(request);
var outBuffer = new ArraySegment<byte>(outBytes);
await s.SendAsync(outBuffer, SocketFlags.None);
var inBytes = new byte[1024];
var inBuffer = new ArraySegment<byte>(inBytes);
await s.ReceiveAsync(inBuffer, SocketFlags.None);
var r = Encoding.UTF8.GetString(inBuffer.Array);
}
catch (SocketException exception)
{
Console.WriteLine($"Error: {exception.Message}");
}
I have 3 projects, an MVC .net core website, an API service and an IdentityServer (IdentityServer4). Logging in to the website works like a charm. It's when I want to get data from the API the problems are starting. Getting the access_token from the httpcontext fails and returns null. The strange thing is that when I debug this, I seem to be getting an access_token back from the httpcontext. Only when I run the website from the live webserver it causes problems.
I configured my OpenIdConnect as follows:
services.AddAuthentication(options =>
{
options.DefaultScheme = "Cookies";
options.DefaultAuthenticateScheme = "Cookies";
options.DefaultChallengeScheme = "oidc";
})
.AddCookie("Cookies")
.AddOpenIdConnect("oidc", options =>
{
options.SignInScheme = "Cookies";
options.Authority = "https://idserverurl";
options.RequireHttpsMetadata = true;
options.ClientId = "clientid";
options.ClientSecret = "xxxxxxxx";
options.ResponseType = "code id_token";
options.SaveTokens = true;
options.GetClaimsFromUserInfoEndpoint = true;
options.Scope.Add("openid");
options.Scope.Add("profile");
options.Scope.Add("offline_access");
};
});
To set the bearertoken for the API calls I use de following code:
var client = new HttpClient();
var accessToken = await HttpContext.GetTokenAsync("access_token");
client.SetBearerToken(accessToken);
When I run this code in debug I get the access_token from the HttpContext. When I run this from the live servers, I get null.
Does anybody have any idea what I'm doing wrong? Could it be a configuration mistake at server level?
I think I solved it myself.
It had to do with the line
JwtSecurityTokenHandler.DefaultInboundClaimTypeMap.Clear();
By removing this line and altering the lines of code where I get my claims (they have a different key now), I am getting the access_token.
FYI I published the project to my own IIS. That way I could attach Visual Studio to the dotnet process and debug it.
Why it worked locally and not online I still don't know.
The above code looks OK - ultimately the tokens should come back from the Authorization Server after user login, then be persisted between requests in the authentication cookie.
Are you able to run Fiddler on your live web server and capture a trace of traffic, including that between your Web App and the Authorization Server? That's the ideal way to troubleshoot - and see if the expected tokens / cookies are present in the HTTPS requests.
In case you're not aware, in .Net Core 2.0 you can construct your web app's HttpClient with an HttpClientHandler that sets proxy details like this:
public class ProxyHttpHandler : HttpClientHandler
{
public ProxyHttpHandler()
{
this.Proxy = new WebProxy("http://127.0.0.1:8888");
}
}
In older versions of MS OpenIdConnect libraries I've seen problems related to load balancing, where not all servers could decrypt the authentication cookie. So if your live servers are load balanced the details in this older link may be relevant.
I have created a custom POST api for getting login information in MCS. when i check in SOAPUI it works perfectly fine. the parameters passed are
1. header
Oracle-Mobile-Backend-Id: ********************
2. Authentocation
Username:****************
password: **************
and basic login info username and password as "User1" and "user1" respectively.
Step2:
when i call the API from MAF i am getting an error 400
the post method used is
public static Response callPost(String restURI, String jsonRequest) {
String responseJson = "";
Response response = new Response();
RestServiceAdapter restServiceAdapter = Model.createRestServiceAdapter();
restServiceAdapter.clearRequestProperties();
//restServiceAdapter.setConnectionName("MiddlewareAPI");
// restServiceAdapter.setConnectionName("");
restServiceAdapter.setRequestType(RestServiceAdapter.REQUEST_TYPE_POST);
restServiceAdapter.addRequestProperty("Content-Type", "application/json");
restServiceAdapter.addRequestProperty("Accept", "application/json; charset=UTF-8");
restServiceAdapter.addRequestProperty("Oracle-Mobile-Backend-Id", "**********");
restServiceAdapter.addRequestProperty("Domain", "mcsdem0001");
restServiceAdapter.addRequestProperty("Username", "******");
restServiceAdapter.addRequestProperty("Password", "*****");
//restServiceAdapter.addRequestProperty("Authorization", "Basic "+new String(encodedBytes));
System.out.println("**** Authorization String ****=>"+new String(encodedBytes));
System.out.println("**** RestURI ******=>"+restURI);
System.out.println("**** jsonRequest ******=>"+jsonRequest);
restServiceAdapter.setRequestURI(restURI);
restServiceAdapter.setRetryLimit(0);
try {
responseJson = restServiceAdapter.send(jsonRequest);
int responseCode = restServiceAdapter.getResponseStatus();
response.setResponseCode(responseCode);
response.setResponseMessage(responseJson);
response.setHeader(restServiceAdapter.getResponseHeaders());
} catch (Exception e) {
int responseCode = restServiceAdapter.getResponseStatus();
response.setResponseCode(responseCode);
response.setResponseMessage(responseJson);
}
System.out.println("Response:" + responseJson);
return response;
}
Could anyone please tell me is there any error in the post method??
This can be due to the version conflict. Try to use HttpUrlConnection instead of RestServiceAdapter and let me know if it works.
actually this bit
restServiceAdapter.addRequestProperty("Username", "******");
restServiceAdapter.addRequestProperty("Password", "*****");
doesn't work because you attempt to pass username and password as a HTTP header. Instead it should be passed as you were trying here
restServiceAdapter.addRequestProperty("Authorization", "Basic "+new String(encodedBytes));
However, these should not be encoded bytes but a base64 encoded string in the form
Basis (without the < abd >)
Note that user identity domains only need to be provided in multi-tenant environments. In MCS, the user domain is defined through the mobile backend you connect to.
Frank
Use the MAF MCS Utility library to make it allot easier.
The developer guide can be found here: http://download.oracle.com/otn_hosted_doc/maf/mafmcsutility-api-doc-082015.pdf
Example code:
MBEConfiguration mbeConfiguration =
new MBEConfiguration(
<mbe rest connection>,<mobileBackendId>,
<anonymous key string>,<application key string>,
MBEConfiguration.AuthenticationType.BASIC_AUTH);
mbeConfiguration.setEnableAnalytics(true);
mbeConfiguration.setLoggingEnabled(false)
mbeConfiguration.setMobileDeviceId(
DeviceManagerFactory.getDeviceManager().getName());
MBE mobileBackend = MBEManager.getManager().
createOrRenewMobileBackend(<mobile backend Id>, mbeConfiguration);
CustomAPI customApiProxy = mbe.getServiceProxyCustomApi();
MCSRequest request = new MCSRequest(mobileBackend.getMbeConfiguration());
request.setConnectionName(<Rest connection name>);
request.setRequestURI("/moile/custom/mockup/employees");
request.setHttpMethod(MCSRequest.HttpMethod.POST);
request.setPayload("{\"id\":\"1\"\"name\":\"nimphius\",\"firstName\":\"frank\"}");
request.setRetryLimit(0);
HashMap<String, String> headers = new HashMap<String, String>();
headers.put("Content-Type","application/json");
request.setHttpHeaders(headers);
MCSResponse response = customApiProxy .sendForStringResponse(request);
String jsonResponse = (String) response.getMessage();
I wrote the following code in Xamarin to connect the Web Server:
var request = WebRequest.Create( "http://srv21.n-software.de/authentication.json") as HttpWebRequest;
// request.Method = "GET";
request.Method = "POST";
request.Headers.Add("name", "demo");
request.Headers.Add("password", "demo");
request.ContentType = "application/x-www-form-urlencoded";
HttpWebResponse Httpresponse = (HttpWebResponse)request.GetResponse();
It connects to the web server and the web server gets the request for "authentication.json", but doesn't get the parameters of the header ("name" and "password").
What is wrong with my code?
Most likely your parameters need to be in the body of the POST request instead of in the headers. Alternatively you might try to use a GET request instead and provide the parameters through the URL, if your server supports it (i.e. http://srv21.n-software.de/authentication.json?name=demo&password=demo).
This worked for me
using System.Net.Http;
string URL = "http://www.here.com/api/postForm.php";
string DIRECT_POST_CONTENT_TYPE = "application/x-www-form-urlencoded";
HttpClient client = new HttpClient();
string postData = "username=usernameValueHere&password=passwordValueHere");
StringContent content = new StringContent(postData, Encoding.UTF8, DIRECT_POST_CONTENT_TYPE);
HttpResponseMessage response = await client.PostAsync(DIRECT_GATEWAY_URL, content);
string result = await response.Content.ReadAsStringAsync();
I'm trying to save cookies in a post request. Here is my code :
CookieContainer myCookieContainer = new CookieContainer();
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(url);
myHttpWebRequest.ContentType = "application/x-www-form-urlencoded";
myHttpWebRequest.UserAgent = userAgent;
myHttpWebRequest.CookieContainer = myCookieContainer;
myHttpWebRequest.Method = "POST";
byte[] postdata = encoding.GetBytes(submitString);
myHttpWebRequest.BeginGetRequestStream(async1 =>
{
using (Stream stream = myHttpWebRequest.EndGetRequestStream(async1))
stream.Write(postdata, 0, postdata.Length);
myHttpWebRequest.BeginGetResponse(async2 =>
{
HttpWebResponse rep = (HttpWebResponse)myHttpWebRequest.EndGetResponse(async2);
CookieCollection cookies = rep.Cookies;
using (Stream stream = rep.GetResponseStream())
using (StreamReader sr = new StreamReader(stream))
{
String content = sr.ReadToEnd();
if (pageDownloadedEventHandler != null)
pageDownloadedEventHandler(content);
}
}, null);
}, null);
Alaways the CookieContainer is empty.
How to get the cookies?
Your code seems to be perfect, if the server sends you back any cookies you should see them in rep.Cookies, as well as in myCookieContainer.
If you want to be sure use Fiddler or Wireshark to analyze the HTTP network traffic and look for the cookies, but if I'm right you won't find them. In this case my idea is to analyze the network traffic doing the same request with your browser, maybe the php/asp.net/other app decided not to set cookies due to some missing request headers.