can I get the correct data/text that is displayed via AJAX using mechanize in ruby?
Or is there any other scripting gem that would allow me to do so?
Mechanized cannot read data displayed by JavaScript, because it does not implement a JavaScript engine (in other words, it can't run it). You'll need a browser to do that, or a program that automates a browser to do it for you. WATIR is one such program.
You can use WATIR with webdriver which is a console only, headless browser.
Related
I'm having some problem accessing the form element on a page I'm getting using Mechanize.
username_page = agent.get 'https://member.carefirst.com/mos/#/home'
username_form = username_page.form_with(name: 'soloLoginForm')
username_form is nil. (username_page does have the page). The page definitely has a form and the field is #soloLoginForm, but username_page.body has no form element.
I'm guessing this is some async or dynamic issue. I'm able to grab the form with poltergeist, and I'm looking into doing all my form filling with capybara/poltergeist, but I wonder if there's something simple I'm missing that will allow me to use mechanize, as I'd planned.
It seems to be that 'https://member.carefirst.com/mos/#/home' uses Angular to render elements of the page and AngularJS requires Javascript support in the browser or in your case Capybara needs a driver with Javascript support.
Mechanize doesn't support Javascript, check this old SO thread. This is probably the reason why it works when you try with poltergeist.
Check: https://github.com/teamcapybara/capybara#drivers
As stated in the answer by #hernanvicente the page is using Angular and requires JS (which mechanize does not support). However, you really want to be using selenium with headless Chrome rather than Poltergeist nowadays. Poltergeist is equivalent to about a 7 year old version of Safari (due to PhantomJS, which it uses for rendering, being abandoned) so it doesn't support a lot of JS and CSS used in modern sites. Another advantage of using Selenium with Chrome is that you can easily swap between headless and headed to see what's happening when you need to debug things.
How I can handle the sign-up form. It appears every time when I hit the url. I want dismiss it globally and also I want to access elements of it. How can I do that? using ruby, watir-webdriver, rspec or cucumber.
Check this out (that is a line from watir-webdriver code):
browser.goto 'http://login:password#www.yoursite.com/index.html'
In other words you can send keys for basic HTTP authorisation right in the url like http://login:password#www.yoursite.com/index.html
I hope it will help you.
Is there any way in Watir to check if a ajax call is made?
I mean when testing manually the tester would open Firebug or an equivalent program and check under the network tab and see that the request appears there.
How could this be automated using Watir?
I wrote simple console script for scraping with capybara (selenium driver) but doesn't want browser to appear. I just wan't text to be written on console.
Is possible to run capybara wihout opening browser?
There is a headless-webkit driver for Capybara that would avoid opening a browser window.
If you are not tied to the Capybara API and don't need to worry about JavaScript then mechanize would probably be a simpler way to interact with Web sites.
I'm trying to download some data from a webpage that is dynamically generated, so using wget doesn't work. The page is http://gaceta.diputados.gob.mx/SIL/Legislaturas/Listados.html I want to download the list shown for each of the options that can be selected in the field "Legislatura" once downloaded I can process the data in ruby.
Just wanted to know what is the best way to download this, and if posible to select each of the options and download.
You can use the Web Inspector in Safari or Chrome or the Firebug extension in Firefox to look at how the data is loaded. The page is doing an AJAX POST request to a Perl script for this website, and the data is return as XML.
I would use cURL to grab the data.
You could use http://watir.com/ or webrat to simulate what you would do to view the data then use Nokogiri to parse the HTML.