How do I take a deeper look at what's going on in the network level of my web browser than the developer tools network panel will provide? For example, I'm trying to debug an issue on a page right now in Firefox (18.0.1) where the loading "spinner" keeps going continuously long after the page has finished loading.
The firebug net panel shows all resources loaded ("200 OK") but the loading icon on the tab is still spinning and the text on the bottom of the window still shows "Transferring data..". How would I take a closer look and see what is supposedly still transferring? Note that this only seems to happen with Firefox and only on a fresh load (not when resources are cached).
After checking your link with Wireshark and with HTTPScoup (a different tool), my conclusion from both was that all the data arrives successfully and no connection remains open with your server's IP. The problem in FF I think is related to something else
Google reveals that in some cases, it is possible to confuse FF into this state. It doesn't have to be related to open network connections.
Take a look at this questions for example:
FireFox sits on "Transferring data from..." or "Read...."
The issue there was unclosed tags in the HTML body.. so try to find problems like this one too. You can try fixing some validation errors by running a validator such as this one. Since you have about 200 validation errors, I suggest you start by deleting half your HTML code until the problem goes away and this way zero-in on the problem location. Then go in and fix the tag mess.. looking a little in the HTML source gave me a headache :)
I would use a tool like Fiddler or Wireshark to do this. Both of these tools provide more low level packet information.
I would start with Fiddler. Fiddler is meant for http traffic analysis whereas Wireshark is a full-fledged packet sniffing tool for all layers of network traffic.
Another advantage to both of these tools is that you can use them with any browser. One disadvantage to fiddler is I believe it is only available for Windows.
I would recommend a very nice Firefox extension called HttpFox which is a bit easier to use than wireshark and friends
It lets you see the HTTP calls made by the page in much greater detail, and also the flow between pages (fast redirects and such). Here is what it looks like:
My problem involved specifically firefox defaulting to mobile sites (but only certain mobile sites). Here is the fix that worked for me.
go into the: "about:config" area of Firefox (type into browser url, ok through warning).
Checked a couple hundred lines.. changed some to default values then I saw one stand out.
network.http.accept.default = "text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8,text/vnd.wap.wml;q=0.6"
The second I saw that wap.wml I knew we had a problem.. turns out via the firefox knowledge base that the q value sets the preference (0.0 lowest, 1.0 highest). When I clicked the reset button the value shot back to:
network.http.accept.default = "text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
I didn't look at the source code of the pages that I had problems with.. but 2 of them were: www.cnn.com and www.paypal.com (both of which shot me to m. version of the page).
after just recently working w/ some htaccess files. I realized that the coding in the htaccess (or in the site's javascript, depending on how you do it) will detect what type of device your own. Some will work w/ your browsers/device. some will look for what your capabilities are. The problem I had as specified above is that my firefox said I was wap capable... some htaccess files say: if wap capable then go to mobile site. Others will say if browser/device = "" then go to mobile site. So really it's how the company codes their backend.. The problem being is that the htaccess isn't taking into consideration the weights of what type you prefer to accept.
anyways fixed and shared..
I'm currently working on a method to automatically identify certain types of spam. My research for this is causing me to run into a ton of phishing attempts and nasty, nasty porn. Adding a website I'm about to open to the images exception list in Firefox preferences is a lengthy process. Is there a quick way (perhaps through a plugin) for me to signify I want to open a page without loading any of its scripts or images?
You can use Adblock Plus and add a filter like *$domain=yahoo.com - any web page from yahoo.com won't be allowed to load anything then. If you want to automate this task, this filter can also be added by running the following code from a privileged context, e.g. from the Error Console:
Components.utils.import("chrome://adblockplus-modules/content/Public.jsm", null)
.AdblockPlus.addPatterns(["*$domain=yahoo.com"]);
And removed by running:
Components.utils.import("chrome://adblockplus-modules/content/Public.jsm", null)
.AdblockPlus.removePatterns(["*$domain=yahoo.com"]);
Documentation of the public API of Adblock Plus: https://adblockplus.org/en/IAdblockPlus (I simplified the way this API is accessed because I assume that compatibility with old Adblock Plus versions and Firefox versions before Firefox 4 is irrelevant here).
I've built a JAVA survey web application with 70 questions. All the questions have some html radio buttons to choose the answer. Those questions and answers are dynamically generated to adapt themselves to the user choices path.
After the user click on a radio-button my web application re-generates some HTML dynamically. It seems incredible to me, but when the user click for the 23rd time/item the browser causes IE8 to crash a tab and do a recovery.
I reduced the amount of generated html and this made the user reach a higher item: 40 (and of course a higher number of time my app can generate HTML code without crashing).
This error is absent on other browsers.
The best guess I made is that, after generating a certain amount of HTML, IE8 reach some sort of cache-size-limit and is unable to handle the page.
On my computer, where I have IE8 latest release and IE-web-development tools I get no errors.
Using Visual studio 2005 we found this (Sorry I translated It from my mother tongue) :
mshtml.dll
UnHandled Excpetion 0x3fdf1b8a on iexplore.exe: 0xC0000005:
Access violation reading on the path 0x00000000.
IE8 version is 8.0.6001.18702
Ho can I debug It?
Can I use JavaScript to reset the cache?
If yes, could you provide me a snippet of code to do this?
FOUND!
It's due to the SmartScreen filter. My code manipulate the DOM adding text by the innerHTML property. Under certain situations It fires the SmartScreen filter for phishing and malware protection.
This says that a lot of users have the same issue.
I will document any progress I have in the next week. I plan to avoid using innerHTML under IE8 going by the DOM appendChild properties.
What Firefox add-ons do you use that are useful for programmers?
I guess it's silly to mention Firebug -- doubt any of us could live without it. Other than that I use the following (only listing dev-related):
Console2: next-generation error console
DOM inspector: as the title might indicate, allows you to browse the DOM
Edit Cookies: change cookies on the fly
Execute JS: ad-hoc Javascript execution
IE Tab: render a page in IE
Inspect This: brings the selected object into the DOM inspector
JSView: display linked javascript and CSS
LORI (Life of Request Info): shows how long it takes to render a page
Measure IT: a popup ruler.
URL Params: shows GET and POST variables
Web Developer: a myriad of tools for the web developer
Here are mine (developer centric):
FireBug - a myriad of productivity enhancing tools, includes javascript debugger, DOM inspector, allows you to edit the CSS/HTML on the fly which is highly valuable for troubleshooing layout and display problems.
Web Developer - again another great developer productivity tool. I mostly use it for quickly validating pages, disabling javascript (yes I disable javascript sometimes, don't you?), viewing cookies, etc.
Tamper Data - lets you tamper with http headers, form values, cookies, etc. prior to posting back to a page, or getting a page. Incredibly valuable for poking and prodding your pages, and seeing how your web app responds when used with slightly malicious intent.
JavaScript Debugger - has a few more features than javascript debugger provided by firebug. Although I must admit, I sparingly use this one since firebug has largely won me over.
Live HTTP Headers - invaluable for troubleshooting, use it frequently. Lets you spy on all HTTP headers communicated back and forth between client and server. It has helped me track down nefarious problems, especially when debugging issues when deploying your web app between environments.
Header Spy - nice addon for the geeky types, shows you the web server and platform a web site runs on in the status bar.
MeasureIt - I don't use this all too frequently, but I've still found it valuable from time to time.
ColorZilla - again, not something I use all that frequently, but when I need it, I need it. Valuable when you want to know a color and you don't want to dig through a CSS file, or open up a graphics editing app to get a color embedded in some image.
Add N Edit Cookies - this has been a great debugging tool in web farms where the load balancer writes a cookie, and uses the cookie value to keep your session "sticky". It allowed me to switch at will between servers to track down problems on specific machine. Also a good tool if you want to try to mess with a site that uses cookies to track your login status/account, and you want to see how your code responds to malformed or hacked info.
Yellowpipe Lynx Viewer Tool - yeah I know what your thinking, lynx, who needs it, its so 1994. But if you are developing a site that needs to take web accessibility into account (meaning accessible to users with visual impairments who use screen readers), or if you need to get a sense of how a web spider/indexer "sees" your site, this tool is invaluable. Granted, you could always just go out and grab Lynx for yourselfhere's the windows xp port that I use.
I've got a handful of other addons that I've used from time to time that I'll just quickly mention: FireFTP (one I installed wasn't stable and I've not tried a newer release), Html Validator (also found this one unstable, least back when I installed like a year ago), IE Tab (I usually just have both IE and FireFox open concurrently, but that is just me, I know many others that find this addon useful).
I'd also recommend the Web Developer extension by Chris Pederick.
As far as web development, especially for javascript, I find Firebug to be invaluable. Web developer toolbar is also very useful.
The ones I have are...
Y-SLow
Live Headers
Firebug
Dom Inspector
One that wasn't mentioned yet is this HTML Validator extension that I found very useful.
#Flávio Amieiro
MeasureIt is an unnecessary extension to have if you install the Web Developer Toolbar. Web Developer Toolbar includes a ruler as one of its features. Under the "Miscellaneous" category for Web Developer click the option "Display Ruler" to use a ruler identical to the MeasureIt one.
That will allow you reduce the number of extensions needed by at least one.
Firefox addons:
FireBug:helps web developers and designers test and inspect front-end code. It provides us with many useful features such as a console panel for logging information, a DOM inspector, detailed information about page elements, and much, much more.
Web Developer-gives you the power disable CSS, edit CSS on the fly, measure certain areas of a page and much more.
ColorZilla
Just click on the icon, hover over the area you'd like to know the hex color for, and click.
Window Resizer
to make sure the layout is displayed properly in the standard resolutions of today.
Total Validator
validating websites much easier by checking HTML, links, CSS and doing a lot more.
Web Developer for web development. Scribefire if you're a blogger-progammer
For web developing I use the Web Developer Toolbar, CSS Viewer and MeasureIt.
But I'm really not one of those who has a thousand of extensions to do everything. I like to keep things simple.
EDIT: Thanks to Dan's answer I don't need MeasureIt anymore. Can't believe I've never seen that! I guess I'll just have to pay more atention to this WebDeveloper toolbar.
Adding to everyones lists, Tamper Data is quite useful, lets you intercept requests and change the data in them.
It can be used to bypass javascript validation and check whether the server side is doing its thing.
I use Web Developer, it's a real time saver.
+1 for LORI ("life-of-request-info"). It's a very convenient alternative for rough measurements of the load time of a particular web page -- the kind of thing that you might otherwise use an external stopwatch for.
New Tab Homepage. Combined with a "speed dial"-type homepage (a personal, fast-loading page of links that you use frequently), helps you get where you're going faster when you open a new browser tab.
LastTab. Changes the behavior of Ctrl+Tab to let you navigate back and forth between your most-recently-used tabs with repeated presses of Ctrl+Tab, the same way that Alt+Tab works in Windows. Also provides a nice view of all open tabs while Ctrl is still being held down for easy navigation. (The resultant behavior is very similar to the Ctrl+Tab behavior in recent releases of Visual Studio.)
FireFTP is good for grabbing/uploading any necessary files.
I find Hackbar to be quite useful. Very useful if you want to edit the querystring part of the url, to test for vulnerabilities, or just general other types of testing where you might end up with complicated query string values.
I was learning DOM inspector, but I've switched to Firebug.
Some of which has been missed above are here
Load Time Analyzer – View detailed graphs of the loading time of web pages in firefox. The graphs display events like page requests, image loading times etc.
Poster – A must have tool for web developers enabling them to interact with web services and other web resources.
Aardvark – A cool extension for web developers and designers, allows them to view CSS attributes, id, class by highlighting page element individually.
Fiddler is a really great debugging proxy. Think of it as a more powerful version of the "Net" panel in Firebug or the Live HTTP headers.
It used to be an IE-only extension, now it also has hooks into Firefox.
Groundspeed, is useful for testing server side code. It was created for input validation tests during pentest, but can be useful for any test that require manipulating input (similar to TamperData).
It lets you control the form elements in the page, you can change their type and other attributes (size, lenght, javascript event handlers, etc). So for example you can change a hidden field or a select to a textbox and then enter any value to test the server response and stuff like that.