Dailymotion API return very old cached version - caching

My application load regulary the number of viewers watching a livestream on dailymotion.
I use the dailymotion API with the field "audience" to do that.
But the server is sending to me a really really old cached version of the precious JSON file.
For example, a streamer is live since 2 hours, but the API is sending this to me :
curl https://api.dailymotion.com/video/x25eyo8?fields=audience
{"audience":0}
If i add another field just a few seconds later, just to see :
curl https://api.dailymotion.com/video/x25eyo8?fields=audience,onair
{"audience":1177, onair: true}
And if I re-send the first request
curl https://api.dailymotion.com/video/x25eyo8?fields=audience
{"audience":0}
More interesting, the headers sent by the server show this :
curl -I https://api.dailymotion.com/video/x25eyo8?fields=audience
HTTP/1.1 200 OK
Server: DMS/1.0.42
X-Dm-Api-Object: video
X-DM-BackNode: web-011.adm.dailymotion.com:80
Cache-Control: public, max-age=10, stale-if-error=900
X-Dm-Api-Method: info
Content-Type: application/json; charset=UTF-8
X-DM-LB: 195.8.215.130
Access-Control-Allow-Origin: *
X-DM-BackNode-Response-Time: 47
Etag: W/"8xO_txIM6arAYYIALcRUgg"
X-Robots-Tag: noindex
Last-Modified: Fri, 06 Nov 2015 20:20:31 GMT
Access-Control-Allow-Headers: Content-Type, Authorization
Access-Control-Allow-Methods: GET, POST, DELETE
X-Dm-Page: fr.rest.rest_api
Via: 1.1 varnish
Fastly-Debug-Digest: 1d9daef237214a02cb79d06c44fdaa26329d5bd16c9afda535f3f9b104438b84
Content-Length: 17
Accept-Ranges: bytes
Date: Mon, 30 Nov 2015 01:16:59 GMT
Via: 1.1 varnish
Age: 116
Connection: keep-alive
X-Served-By: cache-fra1225-FRA, cache-lhr6323-LHR
X-Cache: HIT, MISS
X-Cache-Hits: 1, 0
Vary: X-DM-EC-Geo-Country, Accept-Encoding
Age : 116 with a max-age of 10 ?
And even when i receive an "up to date" version (age < 10), the file still contains 0 viewers whereas the stream is online and 1000+ viewers are watching.
Now there are 2 questions :
Why this happens ?
Can I force a non-cached version ?
Thanks for your help.
EDIT :
It looks like it's the same problem when you watch a stream on http://games.dailymotion.com/. For all channels, numbers of viewers are correct inside the player but for some ones, the number displayed under it is not (most of the time, it indicate 0)
Example of wrong number of viewers

Related

Dynamics CRM + Execute requests in parallel

I have a complex azure function which creates/updates approx 400 records in a table in dataverse while checking several conditions
I am using ExecuteTransactionRequest to create/update the records and it takes around 15 seconds to perform the logic. Is there a way I can make it faster, introduce parallel processing along with executetransation?
I'm not sure if my answer is applicable in your scenario, but I'll mention it for reference.
WebApi accepts a batch requests which saves you the latency time of sequential and concurrent requests. One request with multiple internal requests is sent once to the api and will run on their servers at once and get back the results.
That is if you are using WebApi and not using connectors to communicate with CDS.
If so, please refer to the following example to post 2 currencies and get all:
--batch_ABC123
Content-Type: multipart/mixed; boundary=changeset_ABC111
--changeset_ABC111
Content-Type: application/http
Content-Transfer-Encoding: binary
POST YOUR-RESOURCE_URL/api/data/v9.1/transactioncurrencies HTTP/1.1
Content-ID: 1
Accept: application/json;
Content-Type: application/json;type=entry
{"currencyname": "Dansk Krona1","currencysymbol": "$","isocurrencycode": "PPV","exchangerate": 0.90}
--changeset_ABC111
Content-Type: application/http
Content-Transfer-Encoding: binary
POST YOUR-RESOURCE_URL/api/data/v9.1/transactioncurrencies HTTP/1.1
Content-ID: 2
Accept: application/json;
Content-Type: application/json;type=entry
{"currencyname": "Dansk Krona2","currencysymbol": "$","isocurrencycode": "UPV","exchangerate": 0.90}
--changeset_ABC111--
--batch_ABC123
Content-Type: application/http
Content-Transfer-Encoding:binary
GET YOUR-RESOURCE_URL/api/data/v9.1/transactioncurrencies HTTP/1.1
Accept: application/json
--batch_ABC123--

Etcd - Outdated index. How to read "X-Etcd-Index" from client

I am using github.com/coreos/etcd/client for communicating with etcd cloud, sometimes i am receiving 401 error "The event in requested index is outdated and cleared" , on server page there is a explanation why it happened and how it can be solved. Actually I want to implement the following scenario.
Get "key" and its modified index
do some job
start watching on "key" from modified index
this way I can be shure that all changes during 2 step will be also received. But Etcd saves only top N changes and sometimes I am receiving 401 error. Acording docu I can use "X-Etcd-Index" + 1 header from Get key request as modified index to watch.
curl 'http://127.0.0.1:2379/v2/keys/foo' -vv
< HTTP/1.1 200 OK
< Content-Type: application/json
< X-Etcd-Cluster-Id: f63cd37d2ff4f650
< X-Etcd-Index: 17245
< X-Raft-Index: 2107637
< X-Raft-Term: 360
< Date: Tue, 15 Dec 2015 09:02:20 GMT
< Content-Length: 791
< ...
But I don't use direct http calls , only github.com/coreos/etcd/client. How can I get (and is it actually possible using api) a http header info on from get key request ?
The X-Etcd-Index header corresponds to the Response.Index field.
The index is available in the response field as the "Index" object. See the documentation here: https://godoc.org/github.com/coreos/etcd/client#Response

Google FIT Rest API Get Sessions

I'm trying to get sessions from a startDate to a EndDate, but every time i try return always a empty record
HTTP/1.1 200 OK
Content-length: 3
X-xss-protection: 1; mode=block
Content-location: https://www.googleapis.com/fitness/v1/users/me/sessions?startTime=2015-11-20T00:00:00.00Z&endTime=2015-11-30T23:59:59.99Z
X-content-type-options: nosniff
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Vary: Origin,X-Origin
Server: GSE
Etag: ""
Pragma: no-cache
Cache-control: no-cache, no-store, max-age=0, must-revalidate
Date: Mon, 30 Nov 2015 16:49:52 GMT
X-frame-options: SAMEORIGIN
Content-type: application/json; charset=UTF-8
{}
Heres the URL :https://www.googleapis.com/fitness/v1/users/me/sessions?startTime=2015-11-20T00:00:00.00Z&endTime=2015-11-30T23:59:59.99Z
What is wrong?
I have data in that range and for example if i execute the dataset it return data (https://www.googleapis.com/fitness/v1/users/me/dataSources/derived:com.google.step_count.delta:com.google.android.gms:estimated_steps/datasets/XXXXXXX000000000000-XXXXXX000000000000 )
Sessions represent a time interval during which users perform a fitness activity.
It will only appear to the sessions resource items that have been converted to a session.
Normally, automatic tracks (the ones that you don't do anything to track) only become sessions when you put a name on it (you can try it using Google Fit App, selecting one of the auto track activities, put a name of it, like "lunch walk"). It takes a while to become available in the API, in my case, it took one hour to become available to the sessions list.
Another aproach to see itens on the session list is to create a session : create a new activity on Google Fit, with start and stop actions.
You may find more information about the session concept here: https://developers.google.com/fit/android/using-sessions

PayPal Payflow Gateway UTF-8 Characters

I'm having a problem getting the PayPal Payflow Gateway accept my HTTPS POST with non US-ASCII characters in it. Not matter what I post with these special characters, it seems to want to accept only US-ASCII encoded bytes. If I send UTF-8 encoded bytes, it still works, but fails to parse some of the request NVP values. I know it is possible because I've posted with another developer's test page to my account (http://ccaples.com/ NVP Quick Test) and that post seems to preserve the special characters.
Here's an example. I've tried with MANY different Content-Type headers and Accept/Accept-Charset values including specifying "; charset=UTF-8" or "; charset=utf-8" in the Content-Type after the text/namevalue.
POST https://pilot-payflowpro.paypal.com/ HTTP/1.1
Content-Type: text/namevalue
User-Agent: KLMS Payflow API for Java
X-VPS-Request-ID: 36A4ED051A8B492ABF70E6BE51CB13D5
X-VPS-CLIENT-TIMEOUT: 20
Connection: close
X-VPS-VIT-INTEGRATION-PRODUCT: KLMS Payflow API for Java
X-VPS-VIT-INTEGRATION-VERSION: 2.0.008
X-VPS-VIT-PROXY: Y
X-VPS-VIT-RUNTIME-VERSION: 20.45-b01
X-VPS-VIT-OS-ARCHITECTURE: amd64
X-VPS-VIT-OS-VERSION: 6.1
X-VPS-VIT-OS-NAME: Windows 7
Cache-Control: no-cache
Pragma: no-cache
Host: pilot-payflowpro.paypal.com
Content-Length: 694
USER[8]=XXXXXXXX&VENDOR[13]=XXXXXXXXXXXXX&PARTNER[6]=PayPal&PWD[8]=XXXXXXXX&VERBOSITY[4]=HIGH&BILLTOEMAIL[28]=XXXXXXXX#kineticlearning.com&TRXTYPE[1]=A&TENDER[1]=C&ACCT[16]=4111111111111111&EXPDATE[4]=1117&CVV2[3]=123&BILLTOFIRSTNAME[4]=Josè&BILLTOLASTNAME[7]=Elkjærd&BILLTOSTREET[14]=123 Elm Street&BILLTOCITY[9]=Elm Creek&BILLTOSTATE[2]=VA&BILLTOZIP[5]=22203&BILLTOCOUNTRY[2]=US&BILLTOPHONENUM[12]=763-221-5593&CUSTBROWSER[108]=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36&CUSTHOSTNAME[12]=192.168.1.10&CUSTIP[12]=192.168.1.10&AMT[6]=600.00&CURRENCY[3]=USD&COMMENT1[26]=Law Enforcement Curriculum&COMMENT2[16]=Standard Version
HTTP/1.1 200 OK
Connection: close
Server: VPS-3.033.00
X-VPS-Request-ID: 36A4ED051A8B492ABF70E6BE51CB13D5
Date: Sun, 25 Jan 2015 17:45:05 GMT
Content-type: text/namevalue
Content-length: 199
RESULT=104&PNREF=B70P7B6EBAE7&RESPMSG=Timeout waiting for Processor response&TRANSTIME=2015-01-25 09:44:47&BILLTOFIRSTNAME=Jos &BILLTOLASTNAME=NotProvided&AMT=600.00&ACCT=1111&EXPDATE=1117&CARDTYPE=0
Ignore the 104 timeout for now. PayPal is working on that. What you can see is that the BILLTOFIRSTNAME is Jos(weird character that's not è) and the æ character in BILLTOLASTNAME must have caused the server's NVP parsing algorithm to have completely failed to parse because it says NotProvided.
Here's the code that converts the Java string in NVP formatted form to bytes:
final byte[] requestBytes = requestString.getBytes(Charsets.UTF_8);
Any idea what character encoding the server-side NVP parser is looking for by default, or how to tell it what character encoding I'm sending the body of the POST in?
I have found the solution. It appears that it is not supported by the Payflow Gateway Docs. The docs seem to be out of date and need updating. I worked with PayPal support and it does appear that the Payflow Gateway supports UTF-8. Here is the trick: either you need to NOT specify the length parameter in the name/value pairs you submit (i.e. BILLTOFIRSTNAME=Josè) or you have to specify in the length parameter as not the length in CHARACTERS (i.e. not value.length()), but the actual length in BYTES of the UTF-8 encoded byte stream for the value (i.e. value.getBytes("UTF-8").length). This transaction worked, stored the UTF-8 characters and displayed properly in the PayPal Manager, and the proper value was returned in the name/value pairs of the Payflow Gateway response. As you can see, the difference from above is that the BILLTOFIRSTNAME and BILLTOLASTNAME length tags are now larger and specify the length in bytes instead of the length in characters. Josè [4 characters but 5 bytes: 4A 6F 73 C3 A8] and Elkjærd & Grün [14 characters but 16 bytes: 45 6C 6B 6A C3 A6 72 64 20 26 20 47 72 C3 BC 6E].
POST https://pilot-payflowpro.paypal.com/ HTTP/1.1
Content-Type: text/namevalue
Connection: close
User-Agent: KLMS Payflow for Java/2.0.008
X-VPS-Request-ID: B8DABD9BDFE246EC909B4CF741030133
X-VPS-CLIENT-TIMEOUT: 20
X-VPS-VIT-INTEGRATION-PRODUCT: KLMS Payflow for Java
X-VPS-VIT-INTEGRATION-VERSION: 2.0.008
X-VPS-VIT-PROXY: Y
X-VPS-VIT-RUNTIME-VERSION: 20.45-b01
X-VPS-VIT-OS-ARCHITECTURE: amd64
X-VPS-VIT-OS-VERSION: 6.1
X-VPS-VIT-OS-NAME: Windows 7
Cache-Control: no-cache
Pragma: no-cache
Host: pilot-payflowpro.paypal.com
Content-Length: 771
USER[8]=XXXXXXXX&VENDOR[13]=XXXXXXXXXXXXX&PARTNER[6]=PayPal&PWD[8]=XXXXXXXX&VERBOSITY[4]=HIGH&BILLTOEMAIL[28]=XXXXXX#kineticlearning.com&TRXTYPE[1]=A&TENDER[1]=C&ACCT[16]=4111111111111111&EXPDATE[4]=0216&CVV2[3]=123&BILLTOFIRSTNAME[5]=Josè&BILLTOLASTNAME[16]=Elkjærd & Grün&BILLTOSTREET[14]=123 Elm Street&BILLTOSTREET2[10]=Elm & Vine&BILLTOCITY[9]=Elm Creek&BILLTOSTATE[2]=VA&BILLTOZIP[5]=12345&BILLTOCOUNTRY[2]=US&BILLTOPHONENUM[12]=763-221-5593&CUSTBROWSER[108]=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36&CUSTHOSTNAME[12]=192.168.1.10&CUSTIP[12]=192.168.1.10&AMT[6]=600.00&CURRENCY[3]=USD&COMMENT1[45]=Victim Advocate Curriculum (Standard Version)&COMMENT2[36]=Purchased for J.C. Hamlin (jchamlin)
HTTP/1.1 200 OK
Connection: close
Server: VPS-3.033.00
X-VPS-Request-ID: B8DABD9BDFE246EC909B4CF741030133
Date: Wed, 28 Jan 2015 01:45:29 GMT
Content-type: text/namevalue
Content-length: 209
RESULT=104&PNREF=B10P7D2F1643&RESPMSG=Timeout waiting for Processor response&TRANSTIME=2015-01-27 17:45:12&BILLTOFIRSTNAME=Josè&BILLTOLASTNAME[16]=Elkjærd & Grün&AMT=600.00&ACCT=1111&EXPDATE=0216&CARDTYPE=0
For anyone else stumbling across this I ran into the same issue as Devin where the è character was being replaced with è when I looked up the transaction in Payflow. I had to convert the encoding to ISO-8859-1. I'm using c# not java so this may not be the same but it was as simple as specifying the encoding in my Stream Writer.
using (StreamWriter sw = new StreamWriter(requestStream, Encoding.GetEncoding("ISO-8859-1")))
An explanation of why this may need to be done can be found here: https://www.i18nqa.com/debug/bug-utf-8-latin1.html

Varnish cache miss with json content type

I'm using the following vcl to help me determine if I'm getting a Varnish cache hit or a miss:
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Varnish-Cache = "HIT";
}
else {
set resp.http.X-Varnish-Cache = "MISS";
}
}
This is working fine, but not for application/json content type.
I'm getting X-Varnish-Cache = "MISS" on every request, but I can't see why this would be happening (and I have a suspicion that it's a false negative).
Here's an example of the response headers for my (apparently) non cached json request:
Accept-Ranges:bytes
Age:0
Cache-Control:public, max-age=10800
Connection:keep-alive
Content-Encoding:gzip
Content-Length:2139
Content-Type:application/json
Date:Thu, 09 Jan 2014 16:08:21 GMT
Etag:"1389283199-1"
Expires:Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified:Thu, 09 Jan 2014 15:59:59 +0000
Server:Apache
Vary:Cookie,Accept-Encoding
Via:1.1 varnish
X-Drupal-Cache:HIT
X-Powered-By:PHP/5.3.23
X-Varnish:544296535
X-Varnish-Cache:MISS
If I look at the response times, it's 8 times quicker on subsequent requests to the same url (so 800ms for the first time, and 100ms the second and then on). This makes me think that Varnish is caching, but my vcl is incorrect.
I should also add that no cookies are being sent with the request, and it's a GET request.
Your VCL is correct (BTW: when varnish response comes from cache X-Varnish header comes with 2 numbers).
The response times improves due to Drupal caching.
Probably JSON is not being cached because of the cookies or your VCL.
You have a working VCl for drupal on https://github.com/NITEMAN/varnish-bites/blob/master/varnish3/drupal-base.vcl
Anyway, to trace the issue your full VCL and a varnishlog trace of the request.

Resources