Monitoring AJAX requests between a Flash applet and a server via a Google Chrome extension - ajax

I am playing a Flash-only game that uses AJAX to communicate with the server. The problem is that all the data is "drawn" and most of it not copy/pastable, so I end up retyping URLs and similar stuff from parts of it (i.e., from the chat).
I thought I'd make a simple page action extension for Chrome that would intercept all the AJAX communication between the game and the server, the way Developer tools can do it, and display only the data I'm interested in (parsing URLs and similar stuff is a no-brainer).
However, looking around the internet, I've found no info on how to do this. Many sites (including answers to some questions here) mention using Developer Tools (I'd prefer having a page action extension, simple enough to share with other players, but any other automation is welcome as well), some mention chrome.webRequest (which seems to be able to provide only the headers),...
I also thought of making a content script along the lines of this answer, but since I'm trying to read the data between a Flash applet (not a web page) and a server, I don't think injecting a JavaScript code is possible.
So, my question is: can this be done and, if yes, how?
In case anyone got the wrong idea, the aim of this is only to monitor the communication and extract the parts I'd want to be able to copy/paste, not change any data (i.e., the purpose is simplification of the game play, not cheating).

Related

Difference between react.js and Ajax

When I googled about React.js what I got is: React.js is a Framework that is used to create user interfaces. If a particular part of the website is frequently updated that means we can use react. But I am confused that Ajax has been used for this only. We can update a part of site using Ajax without page refresh. For templating we would be using handlebars and mustache. Could somebody explain me in what ways react is different from Ajax and why we should use it.
In short, React uses AJAX. They are not related in the way you're asking.
Keep reading for a crash course in what React is, what AJAX is, and how they are used to make modern web applications.
This is probably a more simple explanation than you're looking for, but for anyone else who may be confused...
AJAX and Airplanes
Think about an Airplane. The most important part of an airplane is that it flies. But an airplane also has wheels. And the wheels serve a very important purpose, because without them the airplane would never fly or land, and despite all the awesome stuff a plane could do in the air, it wouldn't matter without wheels.
This is the same relationship that React has with AJAX. React is the airplane, and AJAX are the wheels. But, ya know, other things have wheels too. Tractors, cars, even some boats have wheels, and they're all very important, and crippled without wheels. So too is AJAX to other web technologies, but when you're talking about airplanes, its wheels are usually the farthest thing from your mind.
So React is to AJAX, what an Airplane is to Wheels.
But let's talk about AJAX. What is it? Why is it so important? How it is used in websites today. Then I'll show how it's used by React. Then show you what React does that's so impressive, it makes you forget about AJAX - Like an Airplane to its wheels.
Remember Websites in the 90's?
When you clicked anything, a new page would have to load to show the effect of your click - even if it was nothing. Here's an awesome example. Go to that page and click around... See how clicks whisk you away to a completely different page? That is the Internet before AJAX.
Now, take a look at this very page: next to each answer is an Up Arrow... Go ahead and click one of them... Notice the page doesn't reload, but you are given feedback: the arrow turns Orange. This may seem insignificant, but it represents big advancements in web technology: AJAX, or more accurately: the AJAX approach to web development.
The AJAX approach allows that to happen! And this is no big deal now; it's so intrinsic to the web experience, it's difficult to imagine the Internet without it.
AJAX and a Clock Face
A good analogy of the AJAX methodology, and how it changed the web is a simple wrist watch, or a wall clock... Imagine the minute, hour and second hands moving around the clock's face to show time. Now, suppose every movement of the second hand caused the entire clock to be destroyed and rebuilt?
All that effort of destroying and rebuilding just to show a tiny change?! Well, that would be an outrageous waste of resources, and that was the Internet of the 90's. Thankfully, we have AJAX now. Just as a clock seamlessly displays the time, AJAX allows web pages to show changes in data immediately, without the page needing to be refreshed; you click an up arrow, and it turns orange. No page reload needed!
Originally, AJAX was just the name given to using existing technologies together to show simple updates to the user, but it has become so intrinsic to the web experience that unless you know what you're doing, you wouldn't even know you're using it. For instance, fetch is the preferred way to accomplish the AJAX approach since 2015. Before that it was XMLHttpRequest - even though JSON was used to transfer data more often because it's less verbose. JQuery is the only web technology that actually says AJAX ($.ajax()) to my knowledge, but you typically wouldn't (and shouldn't unless you really know what you're doing) use JQuery with a react application.
And AJAX works just like webpages:
The user performs an action (like pressing an up arrow)
A client (A Web Browser like Firefox) requests data from a server (like the Stack Overflow (SO) Server).
The server processes the request (updates the database to record the upvote).
The server sends a response back to the client that says if the action was successful or not.
Finally, some of the code already loaded into the web page, decides how to process this new information (in our example, javascript would add a class to the up-arrow and CSS rules would dictate that elements with that class are orange).
The user only sees that the arrow is orange. All the other steps are hidden so it seems like one seamless, responsive action.
Single-Page Applications
Since we're not rebuilding the entire page with every click, you can keep information about the site stored in the browser. This can be used throughout your entire visit and future visits.
The first time you visit Stack Overflow, all of the CSS, JS, and HTML is loaded. These three languages define the style (CSS), behavior (JS), and structure (HTML) of the data sent back and forth from the server. And guess how that data is sent! AJAX.
This is how most of the web works now. Google, Facebook, Amazon, Youtube, Reddit, every site built with WordPress and WIX, even Stack Overflow - they all use this basic paradigm for delivering their sites to users efficiently. The difference comes in how the Single-page application is built and managed...
React.js
React is a javascript library for building and maintaining Single-Page Applications.
But that's not even that big of a deal. The big deal about React is how it allows you to build applications...
Basically, you build things separately, then put them together: Components come together to form an Application. So take a look at this totally plausible but fake code for all the answers on this page:
answerArray.map(a => <Answer answerData={a}></Answer>)
This is one line that shows most of the information on this page. That is a big deal. The developers at Stack Overflow created their own component, called "Answer" and its only job is to show an Answer. You run that in a loop, and bam, all the intricacies of all the answers are abstracted, hidden in the Answer Component, which is completely separate from other components.
Now take a look at this:
<App>
<Header />
<LeftSidebar />
<Question>
{ answerArray.map(a => <Answer answerData={a} /> )}
</Question>
<RightSidebar />
<Footer/>
</App>
This is the whole Stack Overflow site.
Each tag (Header, Question, Answer, etc.) is a component. These components are completely separate and have self-contained code, but here they are used together to build the more complex application.
Composition
An important concept of React is composition, and we just defined it above. "Composition allows you to build more complex functionality by combining small and focused functions" (flaviocopes). Our Application is composed of smaller components.
It's also important to note that each component contains its own functionality. That means if the user clicks a button and a warning appears, the button and the code that makes the warning appear are in the same component.
Functional Programming
Surprise, we already defined this too. Functional programming, for our purposes, means 1. objects; and 2. how they behave; are in the same place. Like the button example above. Click a button, get a warning. And that's all in the same file.
This is different than pre-React development where all the buttons would be in one file, and all effects of the buttons would be in another. And this isn't necessarily a wrong way to do things, but for web development, it is easier to think in terms of self-contained building blocks, rather than widely dispersed tools that don't work by themselves.
Why you shouldn't care about Moustache and Handlebars
These two technologies have been cannibalized by React. Similar to how React uses AJAX but makes it easier, Moustache and Handlebars are already inside React, and you're using them all the time without even knowing it. And to me, that's ok. There are arguments to the contrary, and knowledge is never a bad thing, so investigate further if you want, but this is already long enough, so that's all I'll say about that.
Instead, I will tell you about 3 technologies you should care about.
What you should care about instead...
Node
The main point of Node.js is that it executes JavaScript outside a browser. Big whoop, right? Well, it turns out this is one of the most influential advancements for web developers ever. In fact, downloading Node is often done before downloading React.
Node is important for 2 huge reasons:
It lets you download other stuff
It lets you process JavaScript before sending it to a browser
I could write pages and pages about Node, but your takeaway from this should be "Node is important, I should be on the lookout for more knowledge about Node and how it relates to React and web development."
NPM
NPM does not stand for "Node Package Manager", but it should, because that's exactly what it does. React, SASS, Angular, Vue, pretty much everything mentioned here you will probably use npm to install and keep updated.
Webpack
Webpack is a "module bundler". It takes all your js and css files and writes them to one file so you only have to worry about writing one <script> tag.
Each React component will have at least one js file associated with it. Each component should have its own file too. Keeping track of all those files is very demanding. Webpack does it for you, it just makes life easier, so learn about it early and don't shy away from it.
This is something so inherent to React Apps that most of the time it will just be working and you won't even know it. For instance, create-react-app installs it automatically, and does not require you to do anything - same with Babel...
Babel
Translates all your code to ECMA5 so it can be read by most browsers and most versions of those browsers.
Again, this can be installed with npm, or if you just want to play around with React and not get too bogged down with the minutia like this, you can run create-react-app, and this will just work with automatic settings and will be out of your hair while you learn.
They make stuff easier
NPM, Webpack, Babel, and many other Node packages are only there to make your life easier. Building web apps require a lot of maintenance - or small, non-programming annoyances that typically you don't even need to think about.
Try not to be intimidated by new packages because wielding their power can mean countless hours devoted to more interesting things.
Conclusion
Hopefully, this post has helped you learn the difference between React, AJAX, and the ongoing nature of web application development. React and AJAX are not comparable, but React uses AJAX, or rather you - the developer - use AJAX in React to get data without the page needing to reload.
AJAX and other technologies were monumental to the advancement of web applications, but because of how absolutely essential they were to applications, they were assimilated into new technologies so much so that you don't even have to know about them to reap their benefits.
My goal was to correct some misconceptions on your path of learning; explain the "why" of the current state of web dev; and introduce technologies you didn't mention but should know about: Node, npm, Babel.
If you want to continue learning, I highly recommend doing a tutorial in React. I have done some at platform.ui.dev/, and enjoy their approach to learning and their payment structure (I haven't been paid to say this). Good luck out there, and I hope this was helpful.
Ajax is used to refresh a web page without having to reload it : it sends a request to the server, but typically the response is processed by the javascript that displays dynamically a new element on the browser without having to reload the entire page.
React is a javascript library that dynamically update the page with inferface components. The components are calculated either by javascript interactions or by an ajax request that go through the server. So ReactJS can also use Ajax requests to update the page.
Mustache and Handlebars are a bit different from ReactJS as the main goal is to transform a template in a component that will be displayed in a page. It can also use Ajax to get data (for getting templates or json datas).
Ajax
We are using Ajax to send http requests. And we can't re-render a particular area of the page(DOM) by using Ajax alone. We need jQuery to re-render the page after an ajax call came up with the response. Actually comparing jQuery + HTML and React.js is far better than comparing ajax and React.js.
React.js
The role of the react.js is dividing page(DOM) into small pieces (Components). ex:- Profile image area, Main Navigation, Sidebar, Textfield, Button. etc. from Big pieces to small pieces. Most importantly we can bind functionalities into these components. Example:- Let's assume users need a popup to upload a profile image by clicking on above "Profile image area". We can write a function to open a popup. And also we can write another function to upload profile image to the database. In this way we can use ajax inside the React.js
Please follow this tutorial.
To simply put, React is a JavaScript library built by Facebook. It is commonly looked as a framework because of its many extensions but the official docs label it as a library for building user interfaces. Ajax on the other hand is not a library or a framework or a language at all. Ajax is a technique used by programmers to call web APIs without having the flow of your code be interrupted at all. At the end of that day, your JavaScript code is run synchronously line by line and Ajax is run asynchronously within your synchronous code but in a way in which it will never pause your code from and have it wait for the API call to be sent and received. With Ajax, sending and receiving data is all done in the background so you won't have to worry about the delay that it takes to get that data. You can actually use Ajax in your React code. Ajax uses something called Fetch to actually call an API and you can use a variety of methods to handle the data that you receive from the API such as .then and .catch or Async/Await. You also aren't required to use Fetch at all, there are other third party ways of calling an API with Ajax such as by using Axios. I'd advise you to watch a video on how to use these different tools because when you figure out how they all work, you'll find that React and Ajax can be used together to build a great application. Hope this helped, please vote however way you felt about this answer. I'm pretty new to this website.
If you've scrolled down to this point you probably have this feeling of missing something in these answers which are great though. For me, it was hard to grasp what AJAX is. I had to look it up on Wikipedia. You can find a very good explanation there. I also read Jesse James Garrett archived article from 2005 where he coined this term (AJAX) and described it as a new approach to web applications. To dig deeper you can visit MDN.
Asynchronous requests are so obvious today in web development that it's hard to imagine there were websites without them. That's the key to understanding AJAX. At that time XMLHttpRequest API was something new. Now we have Fetch API in JavaScript or we could use Axios.
Google Maps approach was revolutionary in 2005. You could zoom in, grab a map, and scroll around. This instant response you had without page reloading was a result of the approach called AJAX. It consisted of a set of technologies like XMLHttpRequest, DOM, html & css, javascript.
As you can see AJAX is an old term to describe an approach in web development that makes applications more responsive (more than 20 years ago). Thus no matter what framework you use (Vue, Angular) or a library like React you use AJAX approach whenever your calls to API are asynchronous and they don't stop the user from interacting with your app which is a standard approach today.
BTW React is a library because it doesn't have a built-in state management tool, or routing tool in contrast to Ember.js, Angular, or Vue. We often talk about React stack, a set of separate tools for building react apps (Redux, Zustand, context api, react-router).

AJAX Crawling with question mark instead of hashbang

Where I'm at: I've read Google's documentation regarding it's AJAX crawling, and I've searched around a bit in this website and others, but I'm quite confused, as it seems that all problems address the same issue: AJAX crawing with hashbangs?
I've developed an app which, among other purposes, let's the user search for locations worldwide, using an AJAX searcher quite similar to Google's, but my app uses exclusively the question mark in AJAX, instead of hashbang. Due to compatibility issues, changing it to the hashbang is not an option.
Not only am I largely confused by the fact that I could not find anyone else using the question mark instead of the hashbang, I'm also wondering if there is any documentation regarding my issue: how to let google bot crawl all my AJAX content when I'm using the question mark instead of a hashbang in my AJAX app.
The AJAX crawling schema was created explicitly for applications and websites using hashbang (#!) in the URL structure, because the fragment part of the URLs only exist on the client side; the URL rewriting in the specs, i.e. from #! to ?_escaped_fragment_= is meant to solve that.
Since most of the web is already making use of Javascript in a way or other, we (Google) needed a better solution, so we started executing Javascript in the pages we crawled and effectively render every page, just like a normal browser would. To quote our blogpost, Understanding web pages better:
In order to solve this problem, we decided to try to understand pages by executing JavaScript. It’s hard to do that at the scale of the current web, but we decided that it’s worth it. We have been gradually improving how we do this for some time. In the past few months, our indexing system has been rendering a substantial number of web pages more like an average user’s browser with JavaScript turned on.
You can also see what we "see" using Fetch as Google in Search Console (former Webmaster Tools); read more about the feature in our post titled Rendering pages with Fetch as Google
Before you do anything else, please try to fetch a few pages from your site with Fetch as Google. You might not have to do anything at all, it might actually work out of the box. And the good news is that it's not only Google that's rendering pages!

What is AJAX, really?

I have to start using AJAX in a project and I don't know where to start. Can someone please help?
Asynchronous JavaScript And Xml. A technique for achieving bi-directional, script-driven communications between Web browsers and servers via HTTP.
See also:
definition on Wikipedia
AJAX Introduction on w3schools
Ajax Workshop 1 on Ajax Lessons
Edit: As pointed out by Nosredna, JSON is often used in place of XML.
The rough idea in English:
You have a web page. Some event (can be a button press or other form event, or just something triggered by a timer) occurs and triggers JavaScript code that asks the server for fresh information (like the latest value of GOOG stock).
There's a piece of code on the server that collects the info you passed and sends some info back. That's different from the page-serving job the server usually has.
When the server answers, a callback function (that you specified in the JavaScript call to the server) is called with the info from the server. Your JavaScript code uses the info to update something--like a GOOG stock chart.
Not to be confused with the cleaner, AJAX, the technology term, is really describing a framework or better stated as a technique for using XML and JavaScript to make asynchronous calls to server side code...
Here are some good code samples. And some more.
While many of these samples above show how to create all of the XML Request objects, if you look into the AJAX Control Toolkit from Microsoft for ASP.NET applications or jQuery, you'll find these easier to work with.
jQuery Sample (from jQuery site):
when code is hit, the some.php file is hit passing the name and location values in.
<script type="javascript">
function saveDataAjax(){
$.ajax({
type: "POST",
url: "some.php",
data: "name=John&location=Boston",
success: function(msg){
alert( "Data Saved: " + msg );
}
});
}
</script>
<input type="submit" onClick="saveDataAjax();" value="submit" />
It's a buzzword, the essence of it is:
Using Javascript to make an asynchronous HTTP request (in the background).
When the content arrives, an action is taken, usually performing some logic then updating the appearance of the page by manipulating the DOM tree; meaning, inserting new HTML elements, deleting some html elements, etc.
The X in AJAX stands for XML, but it's irrelevant. XML is just one of many ways to format the data that's sent by the server. JSON is a much better alternative (IMNSHO). Also, the server can send plain text or just regular html.
The keyword here is asynchronous request. A request that happens in the background, without the browser having to reload the page.
From the Pragmatic Ajax book:
What Is Ajax?
Ajax is a hard beast to distill into a
one-liner. The reason it is so hard is
because it has two sides to it:
Ajax can be viewed as a set of
technologies.
Ajax can be viewed
as an architecture.
Ajax: Asynchronous JavaScript and
XML
The name Ajax came from the bundling
of its enabling technologies: an
asynchronous communication channel
between the browser and server,
JavaScript, and XML. When it was
defined, it was envisioned as the
following:
Standards-based presentation using XHTML and CSS
Dynamic display and interaction using the browser’s DocumentObject
Model (DOM)
Data interchange and manipulation using XML and XSLT
Asynchronous data retrieval using XMLHttpRequest or XMLHTTP (from
Microsoft)
JavaScript binding everything together
Although it is common to develop using
these enabling technologies, it can
quickly become more trouble than
reward.
It is for these reasons that the more
important definition for Ajax is...
Ajax: The Architecture
The exciting evolution that is Ajax is
in how you architect web applications.
Let’s look first at the conventional
web architecture:
Define a page for every event in the application: view items, purchase
items, check out, and so on.
Each event, or action, returns a full page back to the browser.
That page is rendered to the user.
This seems natural to us now. It made
sense at the beginning of the Web, as
the Web wasn’t really about
applications. The Web started off as
more of a document repository; it was
a world in which you could simply link
between documents in an ad hoc way. It
was about document and data sharing,
not interactivity in any meaningful
sense.
Picture a rich desktop application for
a moment. Imagine what you would think
if, on every click, all of the
components on the application screen
redrew from scratch. Seems a little
nuts, doesn’t it? On the Web, that was
the world we inhabited until Ajax came
along.
Ajax is a new architecture. The
important parts of this architecture
are:
Small server-side events: Now components in a web application can
make small requests back to a server,
get some information, and tweak the
page that is viewed by changing the
DOM. No full page refresh.
Asynchronous: Requests posted back to the server don’t cause the
browser to block. The user can
continue to use other parts of the
application, and the UI can be updated
to alert the user that a request is
taking place.
onAnything: We can interact with the server based on almost anything
the user does. Modern browsers trap
most of the same user events as the
operating system: mouseovers, mouse
clicks, keypresses, etc. Any user
event can cause an asynchronous
request.
This all sounds great, doesn’t it?
With this change we have to be
careful, though. One of the greatest
things about the Web is that anybody
can use it. Having simple semantics
helps that happen. If we go overboard,
we might begin surprising the users
with new UI abstractions. This is a
common complaint with Flash UIs, where
users are confronted with new symbols,
metaphors, and required actions to
achieve useful results.
Most commonly, it refers to the use of the XMLHttpRequest object via JavaScript* in a browser.
Depending on who you ask, it could be used to describe almost any type of client/server communication over HTTP other than just typing a URL into a browser.
*jQuery provides some nice wrapper code to handle cross-browser differences, etc.
Ajax is a bit of a misnomer. To quote the wiki article:
Despite the name, the use of
JavaScript and XML is not actually
required, nor do the requests need to
be asynchronous.
Whereas now most people call "ajax" any type of
web application that communicates
with a server in the background
http://www.w3schools.com/Ajax/Default.Asp
that is a good place to start. This should answer all of your questions.
From the man that coined the term - http://adaptivepath.com/ideas/essays/archives/000385.php
"Ajax" is the successfull marketing term introduced back in 2005 to replace the the older term "DHTML" that did not stick well. "Ajax" today is part of the history too as the new word - "HTML5" emerge. Still "HTML5" is pretty much what original "DHTML" used to be.
Ajax is also reffered to as "the new approach to the application development" where a web page is created on the server initially but later on, during its lifetime, the updates are being done on the client as the data or partial content gets communicated to the server in a background.
Hope this clarifies.
Just to add.. may be not relevant for the question ..
although, AJAX was made famous by Gmail in their browser emails ..the credit of AJAX goes to Microsoft .. they created the AJAX thing..
I believe the fastest and easiest way to get started is with jQuery:
http://jquery.com/
http://docs.jquery.com/Ajax/jQuery.ajax#examples
AJAX stands for asynchronous JavaScript and XML, though it doesn't always deal with XML data anymore. Essentially it boils down to using the XMLHttpRequest object through JavaScript running on the client to make a web request and retrieve some information that you use to update the state of your page without requiring a page refresh.
Start with a basic tutorial that shows you how to use bare bones Ajax to make asynchronous requests such as http://www.w3schools.com/Ajax/Default.asp before moving on to using it in a production level application.
When using it in an application you're far better off investigating one of the common JavaScript frameworks that abstract away the differences between the various browsers and make it easy to manipulate the page after the request returns. I personally recommend http://www.jquery.com/
I read Head First AJAX as my first AJAX reference and I found it to give a simple and practical overview of AJAX.
Creative use of previously known technology. Both the browser side scripting and programmatic access to data on the server have been known before. In AJAX it has been put together for innovative use anabling new applications of thechology known before. The REST comes to mind as similar type of advance...
AJAX is very simple : someone somewhere tought that it would be cool to be able to send something to the server and receive something from it without reload a page.
AJAX is not a revolution, it's just a name for something simple : a web page can send a request to the server without being reloader - just some asynch stuff here.
You can add AJAX controls on your web pages wihout any works - just drag them in with Visual Studio. You may have to add some manager for them, but it is simply a drag-and-drop task.
But be warned : rogue web browser usually don't speak the same AJAX language as IE...
:)
AJAX is really fancy term for giving the browser the ability to refresh parts of its content with the need to reload an entire page. Like many have said, it doesn't require XML, or even Javascript in order to implement it. In fact in its early days it was done with with VBScript and Jscript and just called DHTML. Jesse James Garrett may have invented the AJAX term, but it was really Microsoft that invented the concept behind it.
This source says Microsoft started it in 1999, but I would date the birth of this technology even further. This Wired article is probably more accurate on the date of this technology being in the late 90's, much of it coming from the old days of the MSDN DHTML Dude columns written by Michael Wallent at Microsoft which started back in 1997. Much of the story is also told in this great video here by Michael himself: http://channel9.msdn.com/posts/Charles/Michael-Wallent-Advent-and-Evolution-of-WPF/ Megan still works at Microsoft by the way working on the Silverlight team nowadays, Microsoft's replacement for ActiveX.
Back to the AJAX thingy...when Jesse James Garrett back in 2005 he was mostly talking about the use of XMLHTTPRequest within Javascript code, and a dash of salt. That later began a hip word that many people started using even though they had no idea what it was, and thought that is really something brand new and hip, when really it was just a remix of something old.....sort of like many hip-hop songs you hear nowadays.
It's not new, just a newer version of something old!
I'll give it a try and say that "it's the concept of having a W3C based (JavaScript, HTML and CSS) solution for building Rich Applications for running on the web in a browser"
Everything else is just "technical details" I guess ... ;)
PS! - AMAZING question ...!! ;)
AJAX (Asynchronous JavaScript and XML) is a newly coined term for two powerful browser features that have been around for years, but were overlooked by many web developers until recently when applications such as Gmail, Google Suggest, and Google Maps hit the streets.
To know more information about Ajax learn Ajax tutorial
AJAX = Asynchronous JavaScript and XML.
AJAX is a technique for creating fast and dynamic web pages.
AJAX allows web pages to be updated asynchronously by exchanging small amounts of data with the server behind the scenes. This means that it is possible to update parts of a web page, without reloading the whole page.
Classic web pages, (which do not use AJAX) must reload the entire page if the content should change.
Examples of applications using AJAX: Google Maps, Gmail, Youtube, and Facebook tabs.(FROM w3school). to understand simply: when we request for a link or submit form we request a synchronously to server for data. webpage destroy current page and regenerate new page. but with AJAX browser can send the same request without repainting the entire page.
It's JavaScript, but it works.

Reverse Engineer A Web Form

I have a web site which I download 2-3 MB of raw data from that then feeds into an ETL process to load it into my data mart. Unfortunately the data provider is the US Dept. of Ag (USDA) and they do not allow downloading via FTP. They require that I use a web form to select the elements I want, click through 2-3 screens and eventually click to download the file. I'd like to automate this download process. I am not a web developer but somehow it seems that I should be able to use some tool to tell me exactly what put/get/magic goes from the final request to the server. If I had a tool that said, "pass these parameters to this url and wait for a response" I could then hack something together in Perl to automate this process.
I realize that if I deconstructed all 5 of their pages and read through the JavaScript includes and tapped my heals together 3 times I could get this info from what I have access to. But I want a faster and more direct path that does not require me to manually parse all their JS.
Restatement of the final question: Is there a tool or method that will show clearly what the final request request sent from a web form was and how it was structured?
A tamperer's best friends (these are firefox extensions, you could also use something like Wireshark)
HTTPFox
Tamper Data
Best of luck
Use Fiddler2 as a proxy to see what is being passed back and forth. I've done this with success in other similar circumstances
Home page is here: http://www.fiddler2.com/fiddler2/
As with the other responses, except my tool of choice is Charles
What about using a web testing toolkit, like Watir and Ruby ?
Easy to fill in the forms.. just use the output..
Use WatiN and combine it with WatiN TestRecorder (Google for it)
It can "simulate" a user sitting in front of the browser punching in values which you can supply from your own C# code...

How is AJAX implemented, and how does it help web dev?

From http://en.wikipedia.org/wiki/AJAX, I get a fairly good grasp of what AJAX is. However, it looks like in order to learn it, I'd have to delve into multiple technologies at the same time to get any benefit out of it. So two questions:
What are resources that can help me understand/use AJAX?
What sort of website would benefit from AJAX?
If you aren't interested in the nitty gritty, you could use a higher-level library like JQuery or Prototype to create the underlying Javascript for you. The main benefit is a vastly more responsive user interface for web-based applications.
There are many libraries out there that can help you get benefit out of AJAX without learning about implementing callbacks, etc.
Are you using .NET? Look at http://ajax.asp.net. If you're not, then take a look at tools like qcodo for PHP, and learn about prototype.js, jquery, etc.
As far as websites that would benefit: Every web application ever. :) Anything you interact with by exchanging information, not just by clicking a link and reading an article.
Every website can benefit from AJAX, but in my opinion the biggest benefit to AJAX comes in data entry sections - forms basically. I have done entire sites where the front end - the part the user sees had almost no AJAX functionality in it. All the AJAX stuff was in the administration control panel for assisting in (correct!) data entry.
There is nothing worse than submitting a form and getting back an error, using AJAX you can pretty much prevent this for everything but file uploads.
I find it easiest to just stay away from all the frameworks and other helpers and just do basic Javascript. This not only lets you understand what's going on under the covers, it also lets you do it in the simplest way possible. There's really not much to it. User the JS XML DOM objects to create an xml document client side. Sent it to the server with XMLHTTPRequest, and then process the result, again using the JS XML DOM objects. Start with something simple. Just try sending one piece of information to the server, and getting a small piece of information back.
The Mozilla documentation is good. Sites that benefit from it the most are ones that behave almost like a desktop application and need high interactivity. You can usually improve usability on almost any site by using it, however.
Ajax should be thought of as a means to alter some content on a page without reloading the entire page.
So when do you need to do this? Really only when you have some user interactions or form information that you want to keep intact while you change some content on the page.

Resources