We have a problem with our SAAS site. We sometimes have users kicked out because our authentication cookie is not there (or possibly corrupted). This happens rarely enough that it is hard to find, but often enough that I want to know why.
I want to install a monitor / sniffer for one of our support engineers. They get the problem every once and a while and can stop and call when it happens.
I am looking for something that will log page visits (with timestamp) and cookie changes (create/mod/delete).
Does anyone have a tool that will do this type of logging for FireFox? Maybe a Sqlite tool that will work for Firefox (which I think takes exclusive on the Sqlite db file).
I think Tamper Data can help you. Open the Tamper Data window. Do the requests. Right click -> Export as XML. You can view the cookies by double clicking on the Cookie header.
Related
I am unable to clear cookies using watir-webdriver and browser.cookies.clear
Are there any other alternatives ?
This is as far as I know a browser based limitation, due to concerns of security and privacy.
Webdriver is interacting with the browser by javascript, and javascript is not allowed to clear all cookies (think how nasty that could be on a malicious site). In the non testing env, most JS that is executing came from the server of the site you are accessing. So the most it is allowed to do is clear the cookies for the 'current domain' e.g. the cookies belonging to the current URL. So if a web page wants to clear all its own cookies, that is OK, but it is not allowed to clear cookies belonging to other domains.
So if you want all your tests to start with fresh cookies, you will need something in the 'Before' section in env.rb that goes to the root of the site in question, and then clears the cookies
BTW the same limitation applies to setting cookies, if you want to create cookies for a specific site, you need to navigate to that site before trying to create them, or nothing gets created.
This is not an answer, but FYI only.
Suppose chrome is the chosen browser, when cucumber is running, from the output of ps -ef | grep chrome we'll be able to find the customized user data directory which is something like below.
--user-data-dir=/tmp/.org.chromium.Chromium.G2DgPo
And inside that directory, we'll be able to find the Cookies file stored under the Default folder.
Not sure directly deleting that database file could fulfill the needs or not. Cause in a normal browser session, such runtime data are stored at
~/.config/google-chrome/Default/Cookies
We have a simple CCTV system in our office that shows a live image from each of our security cameras. The CCTV system doesn't have an API or any method of extracting the live images. You can however view the image from another browser by creating a basic HTML page with the image link:
http://192.168.1.6/media/getimage_sid.php?sid=a09c4ecb72bade3802e7bf563b0d0bd6&card=1&camera=1&width=384&height=288
This works perfectly, until the session expires and/or timesout. I don't know very much about cookies and sessions but when I inspected the page in Google Chrome I noticed the following cookie:
Name Value Domain Path Expires Size
PHPSESSID a09c4ecb72bade3802e7bf563b0d0bd6 192.168.1.6 / Session 41
there is also a HTTP column and a Secure column but both are empty.
What I am trying to figure out, is how do I keep that cookie alive or trigger it to recreate with the same value? I'm assuming that a rake task to log in to the system wouldn't work because that would reset the session ID every time.
The intranet is a Rails application, so one way would be to create a script that logs in and stores the current session ID to the database and then putting the last recorded sessions ID into the IMG links from the database. It's a bit of a long way around though, I'm hoping for a better solution.
I have read a few articles showing how to do this with AJAX but that would appear to rely on the intranet being viewed all the time. I need this to work if no-one has viewed the intranet for the weekend.
This project is so we can put a couple of live (when the page refreshes!) images on our intranet so we don't have to continuously go to the CCTV system, log in and find the right camera just to see who is at the garage door etc.
Any help would be appreciated.
It's a bit of hack but I've made a small script to pull in the latest session ID and then put it into the image links.
A random different approach: does the following URL get the right image, without having to worry about the session id?
http://192.168.1.6/media/getimage_sid.php?card=1&camera=1&width=384&height=288
The session ID used in the cookie seems to be the PHP generated one.
I don't think session ID should become stale if you 'notify' the server that you're still online.1 You should try to specify the Cookie: in your HTTP request headers. Specifying the SID via the URL is probably not be enough to indicate to the server that you're actually using it.2
If your web-pages are fetching the images directly (i.e. you have an <img src="http://192.168.1.6/..."> in the HTML page) you might work like this:
make an AJAX request (XMLHttpRequest) to a URL which returns a session ID.
any subsequent request to the server on that page should automatically include the session in the headers.3
Otherwise, if you can't specify a Cookie: header, you may choose to make the time before a session becomes stale longer. If you have access to the computer hosting the PHP interface (192.168.1.6) then you can configure PHP to do so (via the php.ini configuration file, I believe). Information about session configuration is available here, and specifically the gc-maxlifetime options seems useful:
session.gc_maxlifetime specifies the number of seconds after which data will be seen as 'garbage' and potentially cleaned up. Garbage collection may occur during session start (depending on session.gc_probability and session.gc_divisor).
Alternatively, if none of the above appeal to you, your solution to fetch (GET) a page to obtain a valid, fresh session ID seems logical and good. You could optimize this by measuring how long it takes before session IDs become stale and fetching new session IDs only at that interval.
1 I looked for a valid reference for this but couldn't find one.
2 specifically PHP uses a PHPSESSID= token in the URL whereas in your example it looks like sid=. It is also generally considered bad practice security-wise I believe (this article explains how it might be used for XSS), since you're exposing user information in the URL, though I think this has little to no effect in this case
3 according to the XMLHttpRequest spec of the send() method:
If the user agent supports HTTP State Management it should persist, discard and send cookies
I've got a web service that, like most others, uses js and css files. I use the old trick of appending a version number to the js and css file like; ?v=123 and that gets changed every time we update the service on production.
Now, this works fine on all browsers, except for Chrome. Chrome seems to prefer it's cached version over getting the new one and therefor seems to ignore the appended variable. In some cases, forcing it to refresh cache (cmd+r / ctrl+f5) wasn't enough so I had to go into options and clear out the cache for it to load up the new content.
Has anyone experienced this issue with Chrome? And if so, what was the resolution to the problem?
Chrome should certainly treat requests with varying query strings as different requests; a cached result for style.css?v=123 should never be used for style.css?v=124. If you're seeing different behavior, please file a bug at http://new.crbug.com/ and post the bug ID here.
That said, I'd first check to see whether the page was cached longer than you expected. If a new version of the page itself wasn't downloaded, then it would still be requesting ?v=123 as the HTML wouldn't have changed. If you're sending long-lived cache headers with the page, it's certainly possible that Chrome is caching it more aggressively than you expected. If that's the behavior you're seeing, please star http://crbug.com/8742 for updates.
I had also same experience
You can user Ctrl + Shift + R for cache free browsing in both Chrome + Mozilla.
I have had this experience as well.
I run a membership site which displays content such as "You must be logged in as a Gold member in order to see this content" if they are not logged in or are trying to view content not allowed by their membership level. But even if the user is logged in, the user would still see "You need to log in", due to Google Chrome's aggressive caching. In Firefox, however, it works fine as I test logging in and out of all 5 levels of membership - each displaying the proper content.
While Chrome's caching problem can be solved by clearing the cache every time the user logs in and out, it would be really annoying to take that approach.
I would like to create a rake task or something to clear the browser cache. The issue is, I am running a Flash app, and if I change the data, I more often than not need to reset the browser cache so it removes the old swf and can see the new xml data.
How do you reset the browser cache with ruby? Or even more precisely, how can I only remove a select item from the browser cache?
Thanks for the help!
I see a few possible solutions:
Write some shell script that deletes the temporary files from disk out the cache (what browser are you using?). I'm am not sure deleting the files on disk will necessarily work if the browser has them cached in memory.
Use and HTTP header (No-Cache) to avoid caching in the browser, Adobe has documentation on No-Cache. You could set this header only in development mode, so that in production the swf is cached.
Depending on your browser, force a page and cache refresh (e.g. Crtl-F5 in Firefox)
I'm not sure how you're loading the xml data, but in the past, I've gotten around the issue by appending a random number to the path of the xml file:
xml.load("data.xml?"+Math.random());
Basically, Flash will always think the file is a different URL. It won't be able to find a match in your cache.
Again, I'm not sure how you're loading the XML data, so I'm not sure if this applies to your situation.
Hope it helps, though.
You cannot reset browser cache, even if you would sometimes it will not be sufficient because caching can occur not only on the server and/or client, but also on any number of nodes your response goes through on its way from your server to your client.
The only tool at your disposal is the caching headers.
You can set them to NoCache just keep in mind that it will be hitting the server every time
Since you're using Safari, here's an article describing how to use AppleScript to clear the cache. But you can probably just skip the AppleScript part and remove the files directly in the rake task. The only catch might be that you have to restart the browser for it to take affect, but that could be done with a kill on the process and an "open /Applications/Safari.app" (I'm assuming you're on a Mac; in Windows it would be something like start "c:\program files\Safari...").
I am using RUBY to screen scrap a web page (created in asp.net) which uses gridview to display data. I am successfully able to read the data displayed on page-1 of the grid but unable to figure out how I can move to the next page in the grid to read all the data.
Problem is the page number hyperlinks are not normal hyperlinks (with URL) but instead are javascript hyperlink which causes postback to the same page..
An example of the hyperlink:-
6
I recommend using Watir, a ruby library designed for browser testing, if you're already using ruby for processing. For one thing, it gives you a much nicer interface to the DOM elements on the page, and it makes clicking links like this easier:
ie.link(:text, '6').click
Then, of course you have easier methods for navigating the table as well. It's easy enough to automate this process:
1..total_number_of_pages.each do |next_page|
ie.link(:text, next_page).click
# table processing goes here
end
I don't know your use case, but this approach has its advantages and disadvantages. For one thing, it actually runs a browser instance, so if this is something you need to frequently run quietly in the background in completely automated way, this may not be the best approach. On the other hand, if it's ok to launch a browser instance, then you don't have to worry about all that postback nonsense, and you can just click the link as if you were a user.
Watir: http://wtr.rubyforge.org/
You'll need to figure out the actual URL.
Option 1a: Open the page in a browser with good developer support (e.g. firefox with the web development tools) and look through the source to find where _doPostBack is defined. Figure out what URL it's constructing. Note that it might not be in the main page source, but instead in something that the page loads.
Option 1b: Ditto, but have ruby do it. If you're fetching the page with Net:HTTP you've got the tools to find the definition of __doPostBack already (the body as a string, ruby's grep, and the ability to request additional files, such as those in script tags).
Option 2: Monitor the traffic between a browser and the page (e.g. with a logging proxy) to find out what the URL is.
Option 3: Ask the owner of the web page.
Option 4: Guess. This may not be as bad as it sounds (e.g. if the original URL ends with "...?page=1" or something) but in general this is the least likely to work.
Edit (in response to your comment on the other question):
Assuming you're using the Net:HTTP library, you can do a postback by just replacing your get with a post, e.g. my_http.post(my_url) instead of my_http.get(my_url)
Edit (in response to danieltalsky's answer):
watir may be a really good solution for you (I'm kicking myself for not having thought of it), but be aware that you may have to manually fire the event or go through other hoops to get what you want. As a specific gotcha, with any asynchronous fetch like this you need to make sure that the full response has come back before you scrape it; that isn't a problem when you're doing the request inline yourself.
You will have to perform the postback. The data is pass with a form POST back to the server. Like Markus said use something like FireBug or the Developer Tools in IE 8 and fiddler to watch the traffic. But honestly this is a web form using the bloated GridView and you will be in for a fun adventure. ;)
You'll need to do some investigation in order to figure out what HTTP request the javascript execution is performing. I've used the Mozilla browser with the Firebug plugin and also the "Live HTTP Headers" plugin to help determine what is going on. It will likely become clear to you which requests you will need to make in order to traverse to the next page. Make sure you pay attention to any cookies getting set.
I've had really good success using Mechanize for scraping. It wraps all of the HTTP communication, html parsing and searching(using Nokogiri), redirection, and holding onto cookies. But it doesn't know how to execute Javascript, which is why you will need to figure out what http request to perform on your own.