I use in my code calls to HttpWebRequest.BeginGetResponse() method to get data from my server. The server produces content that may range from few KB to few GB.
My problem is that HttpWebRequest.BeginGetResponse completes too late. It should complete immediately after the connection to the server is established and the HTTP header is received.
Here is sample code using GET method:
public bool StartDownload()
{
try
{
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(m_getUrl);
myHttpWebRequest.Method = "GET";
// Start the asynchronous request.
m_requestState = new RequestState();
m_requestState.request = myHttpWebRequest;
myHttpWebRequest.BeginGetResponse(new AsyncCallback(ResponseCompleted), m_requestState);
}
catch (Exception)
{
m_requestState = null;
}
return m_requestState != null;
}
private void ResponseCompleted(IAsyncResult result)
{
RequestState myRequestState = (RequestState)result.AsyncState;
HttpWebRequest myHttpWebRequest = myRequestState.request;
m_logger.LogMessage("ResponseCompleted notification received!");
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)myHttpWebRequest.EndGetResponse(result);
}
catch (Exception)
{
}
.......
}
I run the code using "http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.1.tar.bz2" for example and the result looks like:
hh:mm:ss.ms
12:51:30.9440000 - Download started!
12:53:04.8520000 - ResponseCompleted notification received!
12:53:04.8560000 - Header received!
12:53:04.8570000 - DataReceived: 524288 bytes
.........................................
12:53:04.8940000 - DataReceived: 78818 bytes
12:53:04.8940000 - Request data received!
12:53:04.8940000 - Received bytes: 76100578
The problem can be easily detected in the log. It is not possible to spend more that one minute to connect and 38 ms to download about 72.5 MB.
It seems that the data is downloaded somewhere on the phone and the RequestComplete notification is sent to the application only when the full content is available locally. This is not OK for me because I need to show progress for the operation.
I get the same result on the device and emulator for WP7 (also on WP7.1).
I run same code on Windows desktop and it run correctly: the request completes within one second and the rest of the download takes about 1-2 minutes.
Is there any solution on WP7 or WP 7.1?
The newly introduced WP 7.1 API "Background File Transfers" does not help because I need full control over the HTTP headers and content. Not all HTTP requests that I make to the server produce files as output.
Thank you!
Mihai
You need to disable response buffering if you want to stream the data down. You can do this by setting AllowReadStreamBuffering to false.
HttpWebRequest myHttpWebRequest = WebRequest.CreateHttp(m_getUrl);
myHttpWebRequest.Method = "GET";
myHttpWebRequest.AllowReadStreamBuffering = false;
Related
I have an OWIN Selfhosted WebAPI with a Controller that accepts a large file as input. The code works in PC without any problems or errors. However in MONO on OSX, the message stays in hung mode in Chrome. When I supply a small size file using Base64 string, the call gets through. This helped me conclude that my file which is of size 1.5 MB is not an acceptable Base64 message size for POST on MONO with OSX.
However I tried using maxContentSize to a gig on HttpRunTime and Also tried OWIN Middleware implementation with setting MaxRequestSizeInBytes to a Gig. Both of these did not let me POST the file still to Web API Controller and request stays in pending status.
Please let me know, if you have any other ideas for setting up Max Message Size or know if there is something on MONO preventing file of size 1.5 MB.
I found a workaround for MAC using OwinMiddleware. Even if I increased the message size, it didn't work, however below solution let me bypass the message size limitation (65K max).
public override async Task Invoke(IOwinContext context)
{
try
{
IOwinRequest request = context.Request;
if (request.Path.HasValue && request.Path.Value.Contains("METHODNAMEFORLARGEFILE"))
{
//TODO: Remove Workaround for MAC to explicitly download whole stream to avoid issues with message size. PC works without this workaround.
LogHelper.Logger.Info("OWIN METHODNAMEFORLARGEFILE Request");
var stream = request.Body;
string requestBody = await (new StreamReader(stream)).ReadToEndAsync();
var requestData = Encoding.UTF8.GetBytes(requestBody);
context.Request.Body = new MemoryStream(requestData);
}
await Next.Invoke(context);
}
catch(Exception ex)
{
LogHelper.Logger.Error(ex.Message, ex);
throw ex;
}
}
I am trying to send and receive process gzip-ed data from server on my client device application (not web).
I am sending gzip-ed content and on client side, I have following method that returns WebResponse:
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse res = base.GetWebResponse(request);
if (((System.Net.HttpWebResponse)(res)).ContentEncoding.Contains("gzip"))
{
Stream responseStream = res.GetResponseStream();
responseStream = new GZipStream(responseStream, CompressionMode.Decompress);
}
//This returns g-ziped content as WebResponse, but I need to return
//above decompressed responseStream as WebResponse, how do I do that?
return res;
}
I am new to this but I am thinking that intercepting every response comming to my app in GetWebResponse is excellent centralized spot to decompress all responses. But the problem is how to pass the decompressed stream as response back?
Much appreciated
In our app (Silverlight 5 out-of-browser client hitting a WebApi server) we routinely use an HttpClient for posting/getting/deleting and so on all our entities between client and server. This all works fine most of the time, but recently we have run into an issue when uploading (posting) larger entities (> 30/35mb): we start the streaming process and BEFORE it is finished our Post method on the Web API is hit, receiving a null entity.
We can't understand what is going on, and suspect there must be some timing issue related since it all depends on the size of the upload.
To further explain, our client in summary is doing this:
HttpResponseMessage response = await _client.SendAsync(request);
string jsonResult = await response.Content.ReadAsStringAsync();
... where _client is our HttpClient and request our HttpRequestMessage. In case it is also relevant (I am trying not to flood the question with code :), the content in the request is created like this:
request.Content = new StringContent(JsonConvert.SerializeObject(content), Encoding.UTF8, "application/json");
Well, when we debug this the Post method on our server is hit before the await _client.SendAsync(request) finishes, which sort of "explains" why it is receiving a null entity in such cases (larger entities), where when it works that await call is finished and THEN the Post is hit.
In case if sheds more light into it, due to certain limitations on the HttpClient (regarding access to AllowWriteStreamBuffering), we have also tested an equivalent scenario but using directly an HttpWebRequest... unfortunately, the behavior is exactly the same. This is the relevant extract:
httpRequest.BeginGetRequestStream(RequestStreamCallback, httpRequest);
(where httpRequest is our HttpWebRequest with AllowWriteStreamBuffering = false), and the callback to handle the request stream is as follows:
private void RequestStreamCallback(IAsyncResult ar)
{
var request = ar.AsyncState as System.Net.HttpWebRequest;
if (request != null)
{
var requestStream = request.EndGetRequestStream(ar);
var streamWriter = new StreamWriter(requestStream) {AutoFlush = true};
streamWriter.Write(_jsonContent);
streamWriter.Close();
requestStream.Close(); // Belt and suspenders... shouldn't be needed
// Make async call for response
request.BeginGetResponse(ResponseCallback, request);
}
}
Again, for larger entities when we debug the Post method on the Web API is hit (with a null parameter) BEFORE the streamWriter.Write finalizes and the streamWriter.Close is hit.
We've been reading all over the place and fighting with this for days on now. Any help will be greatly appreciated!
In case somebody runs into this, I finally figured out what was going on.
In essence, the model binding mechanism in the Web API Post method was throwing an exception when de-serializing the JSON, but the exception was somewhat "hidden"... at least if you did not know that much about the inner workings of the Web API, as was my case.
My Post method originally lacked this validation check:
var errors = "";
if (!ModelState.IsValid)
{
foreach (var prop in ModelState.Values)
{
foreach (var modelError in prop.Errors.Where(modelError => modelError != null))
{
if (modelError.Exception != null)
{
errors += "Exception message: " + modelError.Exception.Message + Environment.NewLine;
errors += "Exception strack trace: " + modelError.Exception.StackTrace + Environment.NewLine;
}
else
errors += modelError.ErrorMessage + Environment.NewLine;
errors += " --------------------- " + Environment.NewLine + Environment.NewLine;
}
}
return Request.CreateErrorResponse(HttpStatusCode.NoContent, errors);
}
This is a "sample" check, the main idea being verifying the validity of the ModelState... in our breaking scenarios is wasn't valid because the Web API hadn't been able to bind the entity, and the reason could be found within the Errors properties of the ModelState.Values. The Post was being hit ok, but with a null entity, as mentioned.
By the way, the problem was mainly caused by the fact that we weren't really streaming the content, but using a StringContent which was attempted to be de-serialized in full... but that is another story, we were mainly concerned here with not understanding what was breaking and where.
Hope this helps.
I'm looking to increase the performance of a high-throughput producer that I'm writing against ActiveMQ, and according to this useAsyncSend will:
Forces the use of Async Sends which adds a massive performance boost;
but means that the send() method will return immediately whether the
message has been sent or not which could lead to message loss.
However I can't see it making any difference to my simple test case.
Using this very basic application:
const string QueueName = "....";
const string Uri = "....";
static readonly Stopwatch TotalRuntime = new Stopwatch();
static void Main(string[] args)
{
TotalRuntime.Start();
SendMessage();
Console.ReadLine();
}
static void SendMessage()
{
var session = CreateSession();
var destination = session.GetQueue(QueueName);
var producer = session.CreateProducer(destination);
Console.WriteLine("Ready to send 700 messages");
Console.ReadLine();
var body = new byte[600*1024];
Parallel.For(0, 700, i => SendMessage(producer, i, body, session));
}
static void SendMessage(IMessageProducer producer, int i, byte[] body, ISession session)
{
var message = session.CreateBytesMessage(body);
var sw = new Stopwatch();
sw.Start();
producer.Send(message);
sw.Stop();
Console.WriteLine("Running for {0}ms: Sent message {1} blocked for {2}ms",
TotalRuntime.ElapsedMilliseconds,
i,
sw.ElapsedMilliseconds);
}
static ISession CreateSession()
{
var connectionFactory = new ConnectionFactory(Uri)
{
AsyncSend = true,
CopyMessageOnSend = false
};
var connection = connectionFactory.CreateConnection();
connection.Start();
var session = connection.CreateSession(AcknowledgementMode.AutoAcknowledge);
return session;
}
I get the following output:
Ready to send 700 messages
Running for 2430ms: Sent message 696 blocked for 12ms
Running for 4275ms: Sent message 348 blocked for 1858ms
Running for 5106ms: Sent message 609 blocked for 2689ms
Running for 5924ms: Sent message 1 blocked for 2535ms
Running for 6749ms: Sent message 88 blocked for 1860ms
Running for 7537ms: Sent message 610 blocked for 2429ms
Running for 8340ms: Sent message 175 blocked for 2451ms
Running for 9163ms: Sent message 89 blocked for 2413ms
.....
Which shows that each message takes about 800ms to send and the call to session.Send() blocks for about two and a half seconds. Even though the documentation says that
"send() method will return immediately"
Also these number are basically the same if I either change the parallel for to a normal for loop or change the AsyncSend = true to AlwaysSyncSend = true so I don't believe that the async switch is working at all...
Can anyone see what I'm missing here to make the send asynchronous?
After further testing:
According to ANTS performance profiler that vast majority of the runtime is being spent waiting for synchronization. It appears that the issue is that the various transport classes block internally through monitors. In particular I seem to get hung up on the MutexTransport's OneWay method which only allows one thread to access it at a time.
It looks as though the call to Send will block until the previous message has completed, this explains why my output shows that the first message blocked for 12ms, while the next took 1858ms. I can have multiple transports by implementing a connection-per-message pattern which improves matters and makes the message sends work in parallel, but greatly increases the time to send a single message, and uses up so many resources that it doesn't seem like the right solution.
I've retested all of this with 1.5.6 and haven't seen any difference.
As always the best thing to do is update to the latest version (1.5.6 at the time of this writing). A send can block if the broker has producer flow control enabled and you've reached a queue size limit although with async send this shouldn't happen unless you are sending with a producerWindowSize set. One good way to get help is to create a test case and submit it via a Jira issue to the NMS.ActiveMQ site so that we can look into it using your test code. There have been many fixes since 1.5.1 so I'd recommend giving that new version a try as it could already be a non-issue.
I've been racking my brain for a couple of days now on how to approach a new requirement.
I have two websites. The first one lets the user fill out an application. The second website is an internal website use to manage the users applications. I need to develop a "web service" that sends the application data from website 1 to website 2 and return a response to website 2 of success or failure. I have never done a web service before and I'm a bit confused on where to start. I've been reading various examples online but they all seem to be just a starting point for building a webservice... no specific examples.
So for posting the data website 1, what would my controller method look like? Do I use Json to post the data to website 2? What would and example of that look like? Is there some form of redirect in the method that points to website 2?
So for posting the response back to website 2 what would that controller method look like? I assume I would use Json again to send the response back to website 1? Is there some form of redirect in the method that points back to website 1?
I would use JSON and POST the application to the web service.
First I am assuming the application data is contained in some type of object. Use JSON.Net to serialize the object into JSON. It will look something like the following code.
var application = new Application();
string serializedApplication = JsonConvert.Serialize(application);
Second is to POST the code your endpoint(webservice, mvc action). To this you'll need to make a HTTPRequest to the endpoint. The following code is what I use to make to POST the code.
public bool Post(string url, string body)
{
//Make the post
ServicePointManager.ServerCertificateValidationCallback = (sender, certificate, chain, errors) => true;
var bytes = Encoding.Default.GetBytes(body);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
Stream stream = null;
try
{
request.KeepAlive = false;
request.ContentLength = bytes.Length;
request.ContentType = "application/x-www-form-urlencoded";
request.Timeout = -1;
request.Method = "POST";
stream = request.GetRequestStream();
stream.Write(bytes, 0, bytes.Length);
}
finally
{
if (stream != null)
{
stream.Flush();
stream.Close();
}
}
bool success = GetResponse(request);
return success;
}
public bool GetResponse(HttpWebRequest request)
{
bool success;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
if (response.StatusCode != HttpStatusCode.OK && response.StatusCode != HttpStatusCode.Created)
{
throw new HttpException((int)response.StatusCode, response.StatusDescription);
}
var end = string.Empty;
using (StreamReader reader = new StreamReader(responseStream))
{
end = reader.ReadToEnd();
reader.Close();
success = JsonConvert.DeserializeObject<bool>(end);
}
response.Close();
}
}
return success;
}
So now you have can POST JSON to an endpoint and receive a response the next step is to create the endpoint. The following code will get you started on an endpoint in mvc that will receive an application and process it.
[HttpPost]
public ActionResult SubmitApplication()
{
//Retrieve the POSTed payload
string body;
using (StreamReader reader = new StreamReader(Request.InputStream))
{
body = reader.ReadToEnd();
reader.Close();
}
var application = JsonConvert.Deserialize<Application>(body);
//Save the application
bool success = SaveApplication(application);
//Send the server a response of success or failure.
return Json(success);
}
The above code is a good start. Please note, I have not tested this code.
You have obviously more than one client for the data & operations. so a service is what you are looking for.
ASP.NET MVC is a good candidate for developing RESTful services. If you (and your Manager) are ready to use beta version, Then Checkout ASP.NET-Web API.
If you want to stay with a stable product, Go for MVC3. you may need to write some custom code to return the data in XML as well as JSON to server different kind of clients. There are some tutorials out there.
So create a Service (ASP.NET MVC / WCF Service) .You may then create 2 client apps, one for the external clients and another for the Internal users. Both of this apps can call methods in the Service to Create/ Read the user accounts / or whatever operation you want to do.
To make the apps more interactive and lively , you may conside including a wonderful thing called SiganalR, which helps you to get some real time data without continuosly polling the data base/ middle tier very in every n seconds !