Intermittent Server cannot set status after HTTP headers have been sent - asp.net-web-api

Question

I am getting an intermittent error in the event logs "Server cannot set status after HTTP headers have been sent". The webapi site is working fine and delivering the correct output even when this happens.
Exception information:
Exception type: HttpException
Exception message: Server cannot set status after HTTP headers have been sent. at System.Web.HttpResponse.set_StatusCode(Int32
value) at System.Web.HttpResponseWrapper.set_StatusCode(Int32
value) at
System.Web.Http.WebHost.HttpControllerHandler.d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Web.Http.WebHost.HttpControllerHandler.d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Web.Http.WebHost.HttpControllerHandler.d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar) at
System.Web.HttpTaskAsyncHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult
result) at
System.Web.HttpApplication.CallHandlerExecutionStep.InvokeEndHandler(IAsyncResult
ar) at
System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult
ar)
The response object is manipulated directly (the format of the json is not known until runtime) as follows.
var response= HttpContext.Current.Response;
response.StatusCode = (int) HttpStatusCode.OK;
response.Headers.Add("X-Robots-Tag", "noindex, nofollow");
response.Write("Some json");
response.Flush();
response.End();
response.End is required, I do not understand why, a similar error occurs if this line is missed out.
Why does this happen, and is there a better way of doing it?

Answer 1

Since you tagged your question with asp.net-web-api I assume that you are using that code in an api controller method. If so, I suggest returning the unknown Json like this:
var response = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent("Some json", System.Text.Encoding.UTF8, "application/json")
};
response.Headers.Add("X-Robots-Tag", "noindex, nofollow");
return response;
Above code was taken from the following links and adapted:
Put content in HttpResponseMessage object
and
Add a custom response header in ApiController (questions and answers on SO)