Is it possible to retrieve a list of all the http requests that occur when I send a single GET http request via code (ruby in my case).
Basically the same as fiddler or what the NET tab in firebug does.
e.g. when I send a GET request to www.amazon.com firebug says that there are 94 requests for this page. I want to be able to get a list of these 94 requests so that I can use them in my code.
From my knowledge I believe that this would not be possible without using some other type of software such as tcpdump. But to be honest this is out of my depth.
To be clear, I do not want to just use fiddler or firebug as I need to do further data manipulation on the requests within my ruby code.
If you do a single GET request, there will only be only a single GET request.
What you are talking about is when you type the URL in your browser and your browser is configured to download a large number of files references in the HTML (css, javascript, images), which can then be shown in something as Firebug.
I'm afraid you'll have to take another look at just what you are looking for...
Related
I'm working on a simple web browser project that uses QtWebEngine.
What I want to is to log out the httpRequest and httpResponse data that transit in my browser. I only interest in http POST transitions.
In chrome's developer's tool, I can do this by go to Network tab, turn on Preserve log. What I need is
Headers>General>Request URL
>From Data (params of POST)
Response (the raw response data)
Since QtWebEngine uses Chromium, I suppose that most things that chrome can do can also be done in QtWebEngine.
How can I get the above three things using QtWebEngine?
If there is no obvious way to do that, can I write an extension to log them out and make QtWebEngine use this extension? I think in extension I can log out http request header but I have no idea of how to log out the response data.
Edit: I don't want to an external debug tool (like port the log to localhost:myport). I need to use those three pieces of data in my browser application.
Edit2:
chrome.devtools.network.onRequestFinished.addListener
Yes somehow I need this but how can I call this or receive a similar event with Qt5.7?
This is something I was wondering, but could not get a definitive answer elsewhere.
Is a http get request asynchronous?
If they're different, are there any major differences?
Not looking for opinions, just definitive answers.
Googling has just repeatedly led me to examples of one or the other.
HTTP is the most common protocol used to transfer data on the web. It's what the browser users on port 80 for all websites. Pages, AJAX, etc.
GET is a particular "verb" used in an HTTP request. A GET request is usually distinct in that it doesn't have a request body and it doesn't expect to modify anything on the server, simply "get" data.
AJAX requests are essentially HTTP requests made from JavaScript code, rather than from navigation in the browser. They may be GET requests, or they may be other kinds of HTTP requests. Structurally they're no different from any other HTTP request made by the browser, they're just made from code instead of the browser's UI.
There is overlap between these three terms, because they're not mutually exclusive versions of the same thing. They're apples and oranges, really. HTTP isn't different from the other two, it would be different from something like FTP. GET isn't different from the other two, it would be different from something like POST.
You can see a lot of this in action by taking a look at your browser's debugging tools. Visiting any reasonably active page (such as Stack Overflow, for example) will show you a number of requests being made and the server's responses to those requests. As you interact with a page which uses AJAX, watch those requests in the debugging tools and see how they're structured. Load a page or two by navigation and see how those requests are structured.
There's not much to it, really. It's all requests and responses, each of which is simply headers and content.
Ajax used so web applications can send data to and retrieve from a server asynchronously (in the background) without interfering with the display and behavior of the existing page.
HTTP GET or HTTP POST are method in the HTTP Protocol, which are a way to send and receive the data.
While Ajax is the Car, HTTP Protocol is the Driving laws.
Few examples of everyday surfing using Ajax:
Facebook Feed - When scrolling to the bottom of Facebook a Loader circle appears that loads a more prior updates on your wall, this is happening without surfing to another page, but rather retrieving it while still on the same page.
Google Omnibox Prediction - When typing part of the text in the Omnibox, google will suggest you the completion of your text while you're still typing.
First try to get through : GET vs POST.
An ajax call can be GET or POST or PUT or any other.
To differentiate between ajax GET & normal HTTP GET.
Ajax GET seems asynchronous by as the request is sent using another thread by the browser.
Ajax GET request has additional X-Requested-With: XMLHttpRequest.
GET Request is captured by browser history, whilw Ajax GET does not get captured.
I know how to use ajax for submitting a form and all. What I am concerned about is, what is actually happening in the background when a form is submitted via ajax.
How are the values transferred? Encrypted or not? And what is the
need of specifying submission type, I mean get or post, if the URL is
not showing the form fields?
Edit: Found this on w3schools:
GET requests can be cached
GET requests remain in the browser history
GET requests can be bookmarked
GET requests should never be used when dealing with sensitive data
GET requests have length restrictions
GET requests should be used only to retrieve data
POST requests are never cached
POST requests do not remain in the browser history
POST requests cannot be bookmarked
POST requests have no restrictions on data length
How do these apply to ajax form submission?
Basically, when you Ajax-submit a form, it is doing exact same thing as what would happen when you as a user GET or POST submit a form - except that it is done in an asynchronous thread by the browser - i.e. called XMLHttpRequest.
If you submit form as a GET request, all of the form values are stitched together as parameter strings and appended to the URL (form's ACTION URL) - prefixed by a ?. This means anyone who can intercept that communication can read the submitted form data even if request is sent to a HTTPS URL. The POST method sends form data as a separate block (from the URL) and if URL is HTTPS then form data gets encrypted.
It looks like you are just starting out in the world of web development - welcome to the world of programming. I would recommend reading up on some good web development/programming books (I don't want to promote any particular book here). Amazon may help suggest few good ones under "Web Development" kind of search terms.
Also, I suggest that you read up a little on GET vs. POST by googling for it (I can only include one or two links - google will show you hundreds).
For the clear understanding & behind the scene things please refer the links given below.
http://www.jabet.com/
How does AJAX work?
Actually ajax request is same as the normal requests at the server end.
GET or POST has their own use cases. for example: GET has a limit of data transfer depending on the browsers from 1KB to 10 KB. where POST has no such limits.
For a server both AJAX & normal request both are same. so it depends on server code which method you wish to support.
ajax requests are NOT encrypted.
http://www.w3schools.com/tags/ref_httpmethods.asp
It looks like you want a very detailed answer so you can find it yourself:
Google it and read thoroughly the pages (wikipedia for example)
Read http://www.w3.org/TR/XMLHttpRequest/
Inspect the packets between your browser and the server
Google Analytics uses Get Request for .gif image to server
http://www.google-analytics.com/__utm.gif?utmwv=4&utmn=769876874&utmhn=example.com&utmcs=ISO-8859-1&utmsr=1280x1024&utmsc=32-bit&utmul=en-us&utmje=...
We can observer that all parameters are sent in this Get Request and the requested image is no where found useful (Its just 1px by 1px Image)
Known Information: If requesting query string is large then Google are going for Post Request.
Now the question is why not Post Request always irrespective of the query string is large or not.
Being data sent via Get Request its leads to security issue. Since, the parameters will be stored in browser history or in web server logs in case of Get Request.
Could someone give any supportive reasons why Google Analytics is depending on both the things?
Because GET requests is what you use for retrieving information that does not alter stuff.
Please note that the use of POST has quite some downsides, the browser usually warns against reloading a resource requested via POST (to prevent double data-entry), POST requests are not cached (which is why some analytics misuse it), proxied etc.
If you want to retrieve a LOT of data using a URL (advice: rethink if there might be a better option), then it's necessary to use post, from Wikipedia:
There are times when HTTP GET is less suitable even for data retrieval. An example of this is when a great deal of data would need to be specified in the URL. Browsers and web servers can have limits on the length of the URL that they will handle without truncation or error. Percent-encoding of reserved characters in URLs and query strings can significantly increase their length, and while Apache HTTP Server can handle up to 4,000 characters in a URL, Microsoft Internet Explorer is limited to 2048 characters in any URL. Equally, HTTP GET should not be used where sensitive information, such as user names and passwords have to be submitted along with other data for the request to complete. In these cases, even if HTTPS is used to encrypt the message body, data in the URL will be passed in clear text and many servers, proxies, and browsers will log the full URL in a way where it might be visible to third parties. In these cases, HTTP POST should be used.
A POST request would require an ajax call and it wouldn't work because of http://en.wikipedia.org/wiki/Same-origin_policy. But images can easily be cross-site, so they just need to add an img tag to the DOM with the required url and the browser will load it, sending the needed information to their servers for tracking.
so, I have page doing a number of ajax and jsonp(i.e. injection) to get data. I would like to know how to find out the request URL I have made without using debugging tools, like firebug. etc. i.e. the history of GET request of the browser. Thanks
For the most part, browsers do NOT log a history of their HTTP GET requests. Also, if you're using ajax, you're probably doing HTTP POST requests as well.
You don't have to use an in-browser tool like Firebug, but you will need some tool to actually get a history of requests and the request bodies.
You can use a tool like Fiddler2, which is a proxy that gets all HTTP traffic outgoing from your computer (including from your browser and all other applications).
If you have control of the server environment, you can also set up logging on the server to capture this data.
You can use WireShark or a similar tool to monitor the network traffic and inspect the packets.