Caching issue with WebClient/HttpClient - caching

I'm developing a code to poll a Bitstamp exchange ticker every 30 seconds. This is a code I have:
public IObservable<string> Stream(Uri way, WebClient wc)
{
Func<IObserver<string>, Task> Fun = async Observer =>
{
var res = await wc.DownloadStringTaskAsync(way);
Observer.OnNext(value: res);
};
return Observable.Create<string>(Fun);
}
public IObservable<string> GetDelay(int secs)
{
var exe = TimeSpan.FromSeconds(secs);
return Observable.Empty<string>("x").Delay(exe);
}
Stream(new Uri("https://bitstamp.net/api/ticker"), new WebClient { }).Concat(GetDelay(30))
.Repeat(5).Subscribe(res => Debug.WriteLine("got result: {0}", res));
The problem is that WebClient (and HttpClient, too) both return cached results after the first call, it can be seen by the same timestamp:
got result: {"high": "690.00", "last": "645.10", "timestamp": "1387715532" ... }
got result: {"high": "690.00", "last": "645.10", "timestamp": "1387715532" ... }
...
Even after turning the networks off they return the result normally so obviously they cache it somewhere. Adding something like "?cache=random" does not work because request parameters are not allowed for ticker on Bitstamp. Setting Headers[HttpRequestHeader.CacheControl] = "no-cache" for WebRequest does not work either.
How can I fix this weird caching behavior?

Solved by setting wc.Headers[HttpRequestHeader.IfModifiedSince] = DateTime.UtcNow.ToString(); before each subsequent call.

For Windows Phone Setting with Language=Chinese and "date+time" is not "24-hour clock" ,
the code wc.Headers[HttpRequestHeader.IfModifiedSince] = DateTime.UtcNow.ToString() causes exception throwing since DateTime.ToString() generates Chinese characters "下午" in the header.
The safer solution is to output the date time format as RFC1123 pattern:
req.Headers[HttpRequestHeader.IfModifiedSince] = DateTime.UtcNow.ToString("R");
This will ensure the date time is in format of "Sat, 05 Jul 2014 13:38:28 GMT", and there will no any Chinese characters within the HTTP headers.

Related

SSE connection keeps failing every 5 minutes

I'm exposing a simple SSE endpoint using the SseEmitter Spring API, persisting all the emitters in a ConcurrentHashMap. The timeout for each emitter is set to 24 hours. Every 10 seconds I'm sending a message to all the clients. Clients are subscribed with native EventSource implementation, listening for events of particular name.
Unfortunately, I've noticed that every 5 minutes the connection is lost and reestablished again - even though timeout of emitter was explicitly set to 24 hours. Client's part does log it as an error, however on server side there's nothing. The issue occurs on both Tomcat and Jetty. I'd like to keep the session open without any interruptions, so resetting the connection every 5 minutes is unacceptable. Any ideas why this could be happening?
#RestController
#RequestMapping("api/v1/sse")
class SseController {
private val emitters = ConcurrentHashMap<String, SseEmitter>()
#GetMapping
fun initConnection(#RequestParam token: String): SseEmitter {
logger.info { "Init connection from $token" }
val emitter = SseEmitter(24 * 60 * 60 * 1000)
emitter.onCompletion {
logger.info { "Completion" }
emitters.remove(token)
}
emitter.onTimeout { logger.info { "Timeout " } }
emitter.onError { logger.error(it) { "Error" } }
emitters[token] = emitter
return emitter
}
#Scheduled(fixedRate = 10000)
fun send() {
emitters.forEach { (k, v) ->
logger.info { "Sending message to $k" }
v.send(
SseEmitter.event()
.id(UUID.randomUUID().toString())
.name("randomEvent")
.data("some data")
)
}
}
}
const eventSource = new EventSource(url);
eventSource.addEventListener('randomEvent', (e) =>
console.log(e.data)
);
eventSource.onerror = (e) => console.log(e);
Alright, seems it was an issue with Stackblitz's service worker. I've just implemented the same client-side solution in Chrome's plain console and the disconnecting is no longer happening.

App running in Android Emulator fails to perform an HTTP Post to localhost

I'm unable to perform an HTTP Post with an app running in an Android Emulator.
{StatusCode: 400, ReasonPhrase: 'Bad Request', Version: 1.1, Content:
System.Net.Http.HttpConnection+HttpConnectionResponseContent, Headers:
{ Server: Microsoft-HTTPAPI/2.0 Date: Wed, 23 Oct 2019 00:58:01
GMT Connection: close Forwarded: host=XXX.XXX.X.XX:XXXXX;
proto=https Content-Type: text/html; charset=us-ascii
Content-Length: 374 }}
Setup:
I'm using an IP address generated by Conveyor by Keyoti
I installed a security certificate on the emulator required by Conveyor by Keyoti
I swapped out Microsoft.AspNetCore.Mvc.HttpPost attribute with System.Web.Http.HttpPost
Emulator:
Successful: HTTP Get
Failed: HTTP Post
Integration Test:
Successful: HTTP Post (using same endpoint)
Code:
I wrote an automated test that calls the same HTTP Post implementation.
Because I executed the same code successfully on my laptop via an automated test, I don't think the actual code is the issue:
open Microsoft.AspNetCore.Mvc
open Newtonsoft.Json
[<ApiController>]
[<Route("api/[controller]")>]
type RegisterController () =
inherit ControllerBase()
[<System.Web.Http.HttpPost>]
member x.Post([<FromBody>] json:string) =
...
Summary:
In conclusion, I have isolated the environment to the Android Emulator and not my laptop. Hence, the emulator can successfully trigger an HTTP Get. However, it fails to perform a HTTP Post even though my laptop device can do both.
UPDATE:
I applied guidance from this Xamarin Android ASP.Net Core WebAPI document.
Specifically, I installed another security certificate on the Android emulator.
I was then able to observe an HTTP Get on the Android Emulator.
However, I continue to get an error for HTTP Post.
OperationCanceledException
Physical Device:
If I run the app from a physical android device I observe the following:
{StatusCode: 500, ReasonPhrase: 'Internal Server Error', Version: 1.1, Content: System.Net.Http.HttpConnection+HttpConnectionResponseContent, Headers:
{
Date: Wed, 23 Oct 2019 13:33:20 GMT
Server: Kestrel
Transfer-Encoding: chunked
Forwarded: host=xxx.xxx.x.xx:xxxxx; proto=https
Content-Type: text/plain
}}
New Update:
I disabled debugging on just my code on the server implementation and discovered the following exception:
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: 'Bad chunk size data.'
Any suggestions?
this might not be a direct answer to your question, but i would like to suggest
localtunnel. a very easy way to temporarily expose your local api so that you can test it either on emulator or even physical device. Have used this alot my self, as it is very convenient to just type a single line in terminal to start it.
The following reference solved my issue.
Infrastructure:
type GlobalHttpClient private () =
static let mutable (httpClient:System.Net.Http.HttpClient) = null
static member val Instance = httpClient with get,set
Xamarin.Android project:
using Android.Http;
using Android.Net;
using Javax.Net.Ssl;
using System.Net.Http;
using Xamarin.Android.Net;
using Xamarin.Forms;
using WebGatewaySupport;
[assembly: Dependency(typeof(HTTPClientHandlerCreationService_Android))]
namespace Android.Http
{
public class HTTPClientHandlerCreationService_Android : IHTTPClientHandlerCreationService
{
public HttpClientHandler GetInsecureHandler()
{
return new IgnoreSSLClientHandler();
}
}
internal class IgnoreSSLClientHandler : AndroidClientHandler
{
protected override SSLSocketFactory ConfigureCustomSSLSocketFactory(HttpsURLConnection connection)
{
return SSLCertificateSocketFactory.GetInsecure(1000, null);
}
protected override IHostnameVerifier GetSSLHostnameVerifier(HttpsURLConnection connection)
{
return new IgnoreSSLHostnameVerifier();
}
}
internal class IgnoreSSLHostnameVerifier : Java.Lang.Object, IHostnameVerifier
{
public bool Verify(string hostname, ISSLSession session)
{
return true;
}
}
}
Xamarin.Forms App:
switch (Device.RuntimePlatform)
{
case Device.Android:
GlobalHttpClient.Instance = new HttpClient(DependencyService.Get<IHTTPClientHandlerCreationService>().GetInsecureHandler());
break;
default:
ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
GlobalHttpClient.Instance = new HttpClient(new HttpClientHandler());
break;
}
Client Gateway:
let postTo (baseAddress:string) (resource:string) (payload:Object) =
GlobalHttpClient.Instance.BaseAddress <- Uri(baseAddress)
let encoded = Uri.EscapeUriString(resource)
let result = GlobalHttpClient.Instance.PostAsJsonAsync(encoded, payload) |> toResult
result
Looks like you have a .NET Core Api. .NET Core does not have System.Web in Asp.NET. The HttpPost attribute and HttpGet attributes should come from Microsoft.AspNetCore.Mvc namespace which you have open.
Also since you are using the ApiController attribute model binding will just work as long as you bind to a model and not just a json string.
Create a model that you want the json to bind to and use that type for your parameter on Post and remove the FromBody attribute. Also if you do that you probably don't need newtonsoft.json.
open Microsoft.AspNetCore.Mvc
[<ApiController>]
[<Route("api/[controller]")>]
type RegisterController () =
inherit ControllerBase()
[<HttpPost>]
member x.Post(thing:TypeOfThing) =

Geocoding requests to HERE API randomly fails

I am trying to geocode addresses with HERE API. I am not free plan. I try following code (Spring Boot in Kotlin):
override fun geocode(address: Address): Coordinate? {
val uriString = UriComponentsBuilder
.fromHttpUrl(endpoint)
.queryParam("app_id", appId)
.queryParam("app_code", appCode)
.queryParam("searchtext", addressToSearchText(address))
.toUriString()
logger.info("Geocode requested with url {}", uriString)
val response = restTemplate.getForEntity(uriString, String::class.java)
return response.body?.let {
Klaxon().parse<GeocodeResponse>(it)
}?.let {
it.Response.View.firstOrNull()?.Result?.firstOrNull()
}?.let {
Coordinate(
latitude = it.Location.DisplayPosition.Latitude,
longitude = it.Location.DisplayPosition.Longitude
)
}.also {
if (it == null) {
logger.warn("Geocode failed: {}", response.body)
}
}
}
It turned out that when I call this method many times in a row, some requests returns empty responses, like this:
{
"Response":{
"MetaInfo":{
"Timestamp":"2019-04-18T11:33:17.756+0000"
},
"View":[
]
}
}
I could not figure out any rule why some requests fail. It seems to be just random.
However, when I try to call same URLs with curl of in my browser, everything works just fine.
I guess there is some limit for amount requests per seconds, but I could not find anything in HERE documentation.
Does anyone have an idea about the limit? Or may it be something else?
Actually, there was a problem with my code. Requests were failing for addresses having "special" symbols like ü and ö. The problem was with building request URL
val uriString = UriComponentsBuilder
.fromHttpUrl(endpoint)
.queryParam("app_id", appId)
.queryParam("app_code", appCode)
.queryParam("searchtext", addressQueryParam(address))
.build(false) // <= this was missed
.toUriString()

Couldn't make new request verification for Slack API

I'm trying the new request verification process for Slack API on AWS Lambda but I can't produce a valid signature from a request.
The example showed in https://api.slack.com/docs/verifying-requests-from-slack is for a slash command but I'm using for an event subscription, especially, a subscription to a bot event (app_mention). Does the new process support event subscriptions as well?
If so, am I missing something?
Mapping template for Integration request in API Gateway. I can't get a raw request as the slack documentation says but did my best like this:
{
"body" : $input.body,
"headers": {
#foreach($param in $input.params().header.keySet())
"$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end
#end
}
}
My function for verification:
def is_valid_request(headers, body):
logger.info(f"DECODED_SECRET: {DECODED_SECRET}")
logger.info(f"DECRYPTED_SECRET: {DECRYPTED_SECRET}")
timestamp = headers.get(REQ_KEYS['timestamp'])
logger.info(f"timestamp: {timestamp}")
encoded_body = urlencode(body)
logger.info(f"encoded_body: {encoded_body}")
base_str = f"{SLACK_API_VER}:{timestamp}:{encoded_body}"
logger.info(f"base_str: {base_str}")
base_b = bytes(base_str, 'utf-8')
dgst_str = hmac.new(DECRYPTED_SECRET, base_b, digestmod=sha256).hexdigest()
sig_str = f"{SLACK_API_VER}={dgst_str}"
logger.info(f"signature: {sig_str}")
req_sig = headers.get(REQ_KEYS['sig'])
logger.info(f"req_sig: {req_sig}")
logger.info(f"comparing: {hmac.compare_digest(sig_str, req_sig)}")
return hmac.compare_digest(sig_str, req_sig)
Lambda Log in CloudWatch. I can't show the values for security reasons but it seems like each variable/constant has a reasonable value:
DECODED_SECRET: ...
DECRYPTED_SECRET: ...
timestamp: 1532011621
encoded_body: ...
base_str: v0:1532011621:token= ... &team_id= ... &api_app_id= ...
signature: v0=3 ...
req_sig: v0=1 ...
comparing: False
signature should match with req_sig but it doesn't. I guess there is something wrong with base_str = f"{SLACK_API_VER}:{timestamp}:{encoded_body}". I mean, the concatination or urlencoding of the request body, but I'm not sure. Thank you in advance!

Spring Cloud Stream w/Kafka + Confluent Schema Registry Client broken?

Curious if anyone has got this working as I'm currently struggling.
I have created simple Source and Sink applications to send and receive an Avro schema based message. The schema for the message is held in a Confluent Schema Registry. Both apps are configured to use the ConfluentSchemaRegistryClient class but I think there might be a bug in here somewhere. Here's what I see that makes me wonder.
If I interact with the Confluent registry's REST API I can see that there is only one version of the schema in question (lightly edited to obscure what I'm working on):
$ curl -i "http://schemaregistry:8081/subjects/somesubject/versions"
HTTP/1.1 200 OK
Date: Fri, 05 May 2017 16:13:37 GMT
Content-Type: application/vnd.schemaregistry.v1+json
Content-Length: 3
Server: Jetty(9.2.12.v20150709)
[1]
When the Source app sends off its message over Kafka I noticed that the version in the header looked a bit funky:
contentType"application/octet-stream"originalContentType/"application/vnd.somesubject.v845+avro"
I'm not 100% clear about why the application/vnd.somesubject.v845+avro content type is wrapped up in application/octet-stream but ignoring that, note that it is saying version 845 not version 1.
Looking at the ConfluentSchemaRegistryClient implementation I see that it POSTs to /subjects/(string: subject)/versions and returns the id of the schema not the version. This then gets put into SchemaReference's version field: https://github.com/spring-cloud/spring-cloud-stream/blob/master/spring-cloud-stream-schema/src/main/java/org/springframework/cloud/stream/schema/client/ConfluentSchemaRegistryClient.java#L81
When the Sink app tries to fetch the schema for the message based upon the header it fails because it tries to fetch version 845 that its plucked out of the header: https://github.com/spring-cloud/spring-cloud-stream/blob/master/spring-cloud-stream-schema/src/main/java/org/springframework/cloud/stream/schema/client/ConfluentSchemaRegistryClient.java#L87
Anyone have thoughts on this? Thanks in advance.
** UPDATE **
OK pretty convinced this is a bug. Took the ConfluentSchemaRegistryClient and modified the register method slightly to POST to /subjects/(string: subject) (i.e. dropped the trailing /versions) which per Confluent REST API docs returns a payload with the version in it. Works like a charm:
public SchemaRegistrationResponse register(String subject, String format, String schema) {
Assert.isTrue("avro".equals(format), "Only Avro is supported");
String path = String.format("/subjects/%s", subject);
HttpHeaders headers = new HttpHeaders();
headers.put("Accept",
Arrays.asList("application/vnd.schemaregistry.v1+json", "application/vnd.schemaregistry+json",
"application/json"));
headers.add("Content-Type", "application/json");
Integer version = null;
try {
String payload = this.mapper.writeValueAsString(Collections.singletonMap("schema", schema));
HttpEntity<String> request = new HttpEntity<>(payload, headers);
ResponseEntity<Map> response = this.template.exchange(this.endpoint + path, HttpMethod.POST, request,
Map.class);
version = (Integer) response.getBody().get("version");
}
catch (JsonProcessingException e) {
e.printStackTrace();
}
SchemaRegistrationResponse schemaRegistrationResponse = new SchemaRegistrationResponse();
schemaRegistrationResponse.setId(version);
schemaRegistrationResponse.setSchemaReference(new SchemaReference(subject, version, "avro"));
return schemaRegistrationResponse;
}

Resources