How I can handle the sign-up form. It appears every time when I hit the url. I want dismiss it globally and also I want to access elements of it. How can I do that? using ruby, watir-webdriver, rspec or cucumber.
Check this out (that is a line from watir-webdriver code):
browser.goto 'http://login:password#www.yoursite.com/index.html'
In other words you can send keys for basic HTTP authorisation right in the url like http://login:password#www.yoursite.com/index.html
I hope it will help you.
Related
I'm having some problem accessing the form element on a page I'm getting using Mechanize.
username_page = agent.get 'https://member.carefirst.com/mos/#/home'
username_form = username_page.form_with(name: 'soloLoginForm')
username_form is nil. (username_page does have the page). The page definitely has a form and the field is #soloLoginForm, but username_page.body has no form element.
I'm guessing this is some async or dynamic issue. I'm able to grab the form with poltergeist, and I'm looking into doing all my form filling with capybara/poltergeist, but I wonder if there's something simple I'm missing that will allow me to use mechanize, as I'd planned.
It seems to be that 'https://member.carefirst.com/mos/#/home' uses Angular to render elements of the page and AngularJS requires Javascript support in the browser or in your case Capybara needs a driver with Javascript support.
Mechanize doesn't support Javascript, check this old SO thread. This is probably the reason why it works when you try with poltergeist.
Check: https://github.com/teamcapybara/capybara#drivers
As stated in the answer by #hernanvicente the page is using Angular and requires JS (which mechanize does not support). However, you really want to be using selenium with headless Chrome rather than Poltergeist nowadays. Poltergeist is equivalent to about a 7 year old version of Safari (due to PhantomJS, which it uses for rendering, being abandoned) so it doesn't support a lot of JS and CSS used in modern sites. Another advantage of using Selenium with Chrome is that you can easily swap between headless and headed to see what's happening when you need to debug things.
I am using Watir to log into an application, push some buttons, etc... Basically the normal stuff that a person would use Watir for.
However, my problem is that there is one particular page that I need to test. It's actually a dynamically-generated PDF and I need to get the actual binary data from it, so that I can load it using a certain gem that we're using. This normally works with static PDF files because we can just use:
open("http://site.com/something.pdf")
This works for static PDFs. However, for a dynamically generated one it doesn't work because we are using Ruby to send the HTTP request and it is not aware of the headers/cookies/session that Watir is using. So instead of getting the actual PDF we get a login page.
Another thing we tried was to use Watir to get the PDF:
#browser.goto "http://site.com/dynamic/thepdffile"
#browser.text
#browser.html
We tried getting the text or html from the page, but no luck because firefox creates a DOM when loading a pdf so the text is an empty string and the html is the DOM that firefox creates when viewing a pdf page. We need the raw HTTP response and there doesn't seem to be a way to extract that.
So we need a solution for this and in my opinion we have these options:
Figure out a way to use "open" or similar method in Ruby, using the session from Watir.
Figure out how to use watir to get the binary http response from the PDF page.
Disable the pdf plugin (which doesn't seem possible) such that the "save as" dialog appears.
Or if you have some other idea please share! Thanks in advance!
I figured out a solution.
In the profile for firefox you can set the plugin.scan.Acrobat to "999" which will effectively disable the PDF plugin.
profile = Selenium::WebDriver::Firefox::Profile.new
profile['plugin.scan.Acrobat'] = "999"
b = Watir::Browser.new :firefox, :profile => profile
I was trying to parse some HTML content from a site. Nokogiri works perfectly for content loaded the first time.
Now the issue is how to fetch that content which is loaded using AJAX. For example, there is a "see more" link and more items are fetched using AJAX, or consider a case for AJAX-based tabs.
How can I fetch that content?
You won't be able to parse anything that requires a JavaScript runtime to produce that content using Nokogiri. Nokogiri is a HTML/XML parser, not a web browser.
PhantomJS on the other hand is a web browser, albeit a special kind of browser ;) Take a look at that and have a play.
It isn't completely clear what you want to do, but if you are trying to get access to additional HTML that is loaded by AJAX, then you will need to study the code, figure out what URL is being used for the AJAX request, whether any session IDs or cookies have been set, then create a new URL that reproduces what AJAX is using. Request that, and you should get the new content back.
That can be difficult to do though. As #Nuby said, Mechanize could be good help, as it is designed to manage cookies and sessions for you in the background. Mechanize uses Nokogiri internally so if you request a page from Mechanize, you can use Nokogiri searches against it to drill down and extract any particular JavaScript strings. They'll be present as text, so then you can use regex or substring matches to get at the particular parameters you need, then construct the new URL and ask Mechanize to get it.
can I get the correct data/text that is displayed via AJAX using mechanize in ruby?
Or is there any other scripting gem that would allow me to do so?
Mechanized cannot read data displayed by JavaScript, because it does not implement a JavaScript engine (in other words, it can't run it). You'll need a browser to do that, or a program that automates a browser to do it for you. WATIR is one such program.
You can use WATIR with webdriver which is a console only, headless browser.
Alright, so here's the dealio: I'm working on a Ruby app that'll take data from a website, and aggregate that data into an XML file.
The website I need to take data from does not have any APIs I can make use of, so the only thing I can think of is to login to the website, sequentially load the pages that have the data I need (in this case, PMs; I want to archive them), and then parse the returned HTML.
The problem, though, is that I don't know of any ways to programatically simulate a login session.
Would anyone have any advice, or know of any proven methods that I could use to successfully login to an https page, and then programatically load pages from the site using a temporary cookie session from the login? It doesn't have to be a Ruby-only solution -- I just wanna know how I can actually do this. And if it helps, the website in question is one that uses Microsoft's .NET Passport service as its login/session mechanism.
Any input on the matter is welcome. Thanks.
Mechanize
Mechanize is ruby library which imititates the behaviour of a web browser. You can click links, fill out forms und submit them. It even has a history and remebers cookies. It seems your problem could be easily solved with the help of mechanize.
The following example is taken from http://docs.seattlerb.org/mechanize/EXAMPLES_rdoc.html:
require 'rubygems'
require 'mechanize'
a = Mechanize.new
a.get('http://rubyforge.org/') do |page|
# Click the login link
login_page = a.click(page.link_with(:text => /Log In/))
# Submit the login form
my_page = login_page.form_with(:action => '/account/login.php') do |f|
f.form_loginname = ARGV[0]
f.form_pw = ARGV[1]
end.click_button
my_page.links.each do |link|
text = link.text.strip
next unless text.length > 0
puts text
end
end
You can try use wget to fetch the page. You can analyse login process with this app www.portswigger.net/proxy/.
For what it's worth, you could check out Webrat. It is meant to be used a tool for automated acceptance tests, but I think you could use it to simulate filling out the login fields, then click through links by their names, and grab the needed HTML as a string. Haven't tried doing anything like it, tho.