I am trying to make a scraper for a page of a supermarket. I noticed that this supermarket make API HTTP calls via AJAX, and if I enter to Inspect > Network then I can see the request headers of the calls I need.
Inside the requests headers there is an X-Token. If I use this X-Token with the URL via Postman I can get all the info in JSON format, which is better than scraping the web.
The problem is that this X-Token expires (I think, it still works). Is there any possibility to make a call to the page and "intercept" this API call in order to retrieve this X-Token and use it for the next custom requests?
I'am using Ruby on Rails :)
Related
I am using Laravel on my API server.
I am making an API request from my chrome extension.
Since I couldn't find a way to save my API auth token safely in the chrome extension and reuse it every time I want to make a request, hence I am looking for a way to identify the request on API side if the request is from my extension or not.
Add an extra header in the chrome extension where you are calling your Api .Add the header to api header when calling. Now in the controller use.
if ($request->hasHeader('X-Header-Name')) {
Api call from extension ..
}
Reference Link Request Header
Is there anyway in JAX-RS, Jersey to ensure a request is Ajax only?
The goal here is to ensure that a number of endpoints are only accessed as AJAX calls and not as a Web URL in a browser?
Reason is the request may contain query parameters that are PHI and we don't want them going into Browser history.
A HTTP request triggered by an AJAX call is not different from any other HTTP request from the POV of the server. Even more, if your web application makes an AJAX request, you can use the tools your browser provides to inspect, copy and manipulate the request. Modern browsers provide the option to copy the exact request as a curl command that can be executed in the shell of your OS. To the server there is not difference between the original made by the browser and the copied request.
There is no way to do what you want to do.
I am using django and making some ajax request to server. As the url is visible in javascript someone could easily copy that and start making request via url bar. Is there any way in django that we can distinguish that the coming request is sent by ajax not a regular browser reqeust.
You can use a tag in your ajax,and in code check request from
Yes you can use
HttpRequest.is_ajax()
as in documentation
https://docs.djangoproject.com/en/2.2/ref/request-response/#django.http.HttpRequest.is_ajax
I am trying using facebook from Scrapy.
I can log in successfully. However, I can't get the posts on facebook because the posts comes from ajax calls.
I tried using firebug and check the XHR. I got this:
I tried to check the response of all those request but none of them contains the actual data of the posts.
What is the ajax request that get the posts on facebook?
Thanks in advance
Don't use the AJAX calls from your browser. Those are encrypted since Facebook uses HTTPS connections. Focus on using their API to make the calls you need to get the data you want. That's the entire purpose of their Graph API.
In my application there is a client and a WCf REST service. For invoking some wcf service the client is doing an http POST even though the service is a GET.
i do not want to do any changes in the client or the service.
So is there a way where i can convert this POST request to GET and add the data coming in as the POST to the URL and invoke the REST service.
Thanks in advance.
You can use URL Rewrite to issue 3xx Redirect which will use GET method, but you will loose all POST data.
The only safe way known to me is to rewrite POST request to some another custom page, where you:
collect all POST data/variables;
convert them into GET variables (assemble proper GET request);
issue 301 (or 302) Redirect to the proper URL (it will have all POST data sent as GET variables).
Such rewrite to custom page should be easy -- you need to check what method is used (POST or GET) and only invoke it on POST. The rest will be handled in that post-to-get script.
The reason for all of this complexity is the difference in how POST and GET requests work: with GET all data is sent as part of URL while POST uses request body to transfer variable's data.