I'm developing a screen scraping robot that uses Watir (ruby) to crawl specific web searches.
Watir is used as the search results are delivered in pages, only available via AJAX requests.
My issue is now that to perform a new search, the browser has to be shut down in order for the search session to be cleared - otherwise the initial search overrule the change in the GET parameters.
Is it somehow possible to force Firefox to clear sessions on every request made?
Additionally, does anyone have experience solving these kind of issues via Watir?
If the session is maintained via cookies in your firefox browser then it's possible.
All you have to remove the cookies from your firefox cookies repository before it starts.
Firefox stores its cookies at (as of in my ubuntu and mac)
~/.mozilla/firefox/12wwonrk.default/cookies.sqlite [in ubuntu]
or
~/Library/Application Support/Firefox/Profiles/eox4ghka.default/cookies.sqlite [in mac]
(prior Firefox 3 it was cookies.txt instead sqlite)
If you can truncate the sqlite (or the txt) then the cookies will no longer be there.
As you are running Watir you are most probably using ruby. So, if you can run these commands through system or %x[] (or compatible commands through sqlite gem/lib) before Watir::Browser.new statement, hopefully you'll be done.
./sqlite3 path/to/cookies.sqlite
DELETE FROM moz_cookies;
.quit
If you want to use Watir, you can mess with profiles as described at http://watirwebdriver.com/. Most browsers seem to get their own profile for each new instance by default.
Related
I am unable to clear cookies using watir-webdriver and browser.cookies.clear
Are there any other alternatives ?
This is as far as I know a browser based limitation, due to concerns of security and privacy.
Webdriver is interacting with the browser by javascript, and javascript is not allowed to clear all cookies (think how nasty that could be on a malicious site). In the non testing env, most JS that is executing came from the server of the site you are accessing. So the most it is allowed to do is clear the cookies for the 'current domain' e.g. the cookies belonging to the current URL. So if a web page wants to clear all its own cookies, that is OK, but it is not allowed to clear cookies belonging to other domains.
So if you want all your tests to start with fresh cookies, you will need something in the 'Before' section in env.rb that goes to the root of the site in question, and then clears the cookies
BTW the same limitation applies to setting cookies, if you want to create cookies for a specific site, you need to navigate to that site before trying to create them, or nothing gets created.
This is not an answer, but FYI only.
Suppose chrome is the chosen browser, when cucumber is running, from the output of ps -ef | grep chrome we'll be able to find the customized user data directory which is something like below.
--user-data-dir=/tmp/.org.chromium.Chromium.G2DgPo
And inside that directory, we'll be able to find the Cookies file stored under the Default folder.
Not sure directly deleting that database file could fulfill the needs or not. Cause in a normal browser session, such runtime data are stored at
~/.config/google-chrome/Default/Cookies
1) I have an external iframe, and I wondered how Selenium or Firebug managed to overcome the Same Origin Policy to retrieve source code (or even execute some JavaScript)
As for Firebug, I think this has to do with cd(frames[0]) command. In Selenium, you can access content in the iframe with an xpath link, but I have no idea how it internally works.
2) Is it possible to use similar methods outside of a Firefox plugin?
3) What about IE? (or Chrome, but it is way less important)
Thank you :-)
The way that Selenium and Firebug get around it is buy running in the browser chrome rather than in the JavaScript sandbox. For example, calling *firefox calls the slightly less secure version of the browser, by less secure I mean it is running in more of the browser chrome than your average user, and can do the relevant calls.
Addons and Extensions to Firefox run in this context too so can access things in the same way.
I need a way to request any site using Ajax. I mean ANY site, I don't want to have to use the workarounds that firefox offers that only apply to someone who's making page requests from the same domain. Is there ANY way to let this happen? I want this to occur as a local file.
Downgrade your Firefox to under version 3
Try http://dirolf.com/2007/06/enabling-cross-domain-ajax-in-firefox.html
Firefox 3 note
Versions of Firefox prior to Firefox 3 allowed you to set the preference capability.policy..XMLHttpRequest.open to allAccess to give specific sites cross-site access. This is no longer supported.
BTW, you can also save your web application(.html) as .hta, HTA application is allow cross site scripting.
We have a problem with our SAAS site. We sometimes have users kicked out because our authentication cookie is not there (or possibly corrupted). This happens rarely enough that it is hard to find, but often enough that I want to know why.
I want to install a monitor / sniffer for one of our support engineers. They get the problem every once and a while and can stop and call when it happens.
I am looking for something that will log page visits (with timestamp) and cookie changes (create/mod/delete).
Does anyone have a tool that will do this type of logging for FireFox? Maybe a Sqlite tool that will work for Firefox (which I think takes exclusive on the Sqlite db file).
I think Tamper Data can help you. Open the Tamper Data window. Do the requests. Right click -> Export as XML. You can view the cookies by double clicking on the Cookie header.
I would like to create a rake task or something to clear the browser cache. The issue is, I am running a Flash app, and if I change the data, I more often than not need to reset the browser cache so it removes the old swf and can see the new xml data.
How do you reset the browser cache with ruby? Or even more precisely, how can I only remove a select item from the browser cache?
Thanks for the help!
I see a few possible solutions:
Write some shell script that deletes the temporary files from disk out the cache (what browser are you using?). I'm am not sure deleting the files on disk will necessarily work if the browser has them cached in memory.
Use and HTTP header (No-Cache) to avoid caching in the browser, Adobe has documentation on No-Cache. You could set this header only in development mode, so that in production the swf is cached.
Depending on your browser, force a page and cache refresh (e.g. Crtl-F5 in Firefox)
I'm not sure how you're loading the xml data, but in the past, I've gotten around the issue by appending a random number to the path of the xml file:
xml.load("data.xml?"+Math.random());
Basically, Flash will always think the file is a different URL. It won't be able to find a match in your cache.
Again, I'm not sure how you're loading the XML data, so I'm not sure if this applies to your situation.
Hope it helps, though.
You cannot reset browser cache, even if you would sometimes it will not be sufficient because caching can occur not only on the server and/or client, but also on any number of nodes your response goes through on its way from your server to your client.
The only tool at your disposal is the caching headers.
You can set them to NoCache just keep in mind that it will be hitting the server every time
Since you're using Safari, here's an article describing how to use AppleScript to clear the cache. But you can probably just skip the AppleScript part and remove the files directly in the rake task. The only catch might be that you have to restart the browser for it to take affect, but that could be done with a kill on the process and an "open /Applications/Safari.app" (I'm assuming you're on a Mac; in Windows it would be something like start "c:\program files\Safari...").