I am trying to get hidden field with mechanize in ruby and trying to click on it.
agent = Mechanize.new
agent.get('http://www.example.com/')
agent.page.link_with(:text => "More Links...")
But this gives me:
=> nil
Actually, I want to click on it:
agent.page.link_with(:text => "More Links...").click
But this is an error:
undefined method `click' for nil:NilClass
And here is my HTML code:
<div id="rld-4" class="results_links_more highlight_d links_deep" style="display: none;">
<a class="large" href="javascript:;">More Links...</a>
</div>
Mechanize currently doesn't support javascript. I'd suggest you try and figure
out what the server expects the user-agent to send and then replicate this with
Mechanize. You can use a tool like HTTPFox which is a Firefox addon that monitors the traffic between a web server and your browser. Once you have this, you can easily replicate it with mechanize. Something like this;
agent = Mechanize.new
# Doesn't work
# home_page = agent.get('http://requestb.in/')
# agent.click(home_page.link_with(:text => "Create a RequestBin"))
# => undefined method `[]' for nil:NilClass (NoMethodError)
# Works
# The javascript code just makes a POST request with one parameter
request_bin = agent.post("http://requestb.in/api/v1/bins", { "private" => "false" })
puts request_bin.body
That should probably find the link if it's really on the page, but the bigger problem is that clicking on a link with a href of 'javascript:;' doesn't do what you think it does. That's because mechanize is not a full browser with a javascript interpreter, etc.
Related
I am trying to automate an online survey on a website but I get this error each time:
Selenium::WebDriver::Error::UnknownError: unknown error: Element is not clickable at
point (561, 864). Other element would receive the click: a id="habla_oplink_a"
class="habla_oplink_a_normal hbl_pal_header_font_size hbl_pal_title_fg "
What I need to understand is how I can scroll to a certain point of the page so that my script can resume filling out the survey on the page.
This is my code that manages to fill out a portion of the survey but fails when it reaches a row which is not in view inside the browser (a row that requires the user to scroll down to):
buttons = browser.elements(:class => "assessment-choice")
buttons.each do |button|
button.click
end
I would also like to be able to change my code so that it only selects a specific option but the HTML on the page is not very friendly.
This is the webpage I am looking at: https://staging2.clearfit.com/assessment/assessment/95867fb272df436352a0bd5fbdd
The HTML of one of the options on the survey:
<a id="answers_79_0" class="assessment-choice" onmouseover="answerOver(this)" onmouseout="answerOut(this)" onclick="setAssessmentAnswer(this, 3, '0', '79', '#answers_49839163')">Strongly<br>Agree</a>
Using execute_script
To scroll to an element, you will need to execute javascript:
browser.execute_script('arguments[0].scrollIntoView();', button)
This can be seen to be working in the following script. Without the line to scroll, a chat tab overlays one of the buttons causing an exception.
require 'watir-webdriver'
browser = Watir::Browser.new :chrome
browser.goto 'https://staging2.clearfit.com/assessment/assessment/95867fb272df436352a0bd5fbdd'
buttons = browser.elements(:class => "assessment-choice")
buttons.each do |button|
browser.execute_script('arguments[0].scrollIntoView();', button)
button.click
end
Using the watir-scroll gem
Note that you can install the watir-scroll gem to make the scrolling line nicer. The gem allows the line to simply be:
browser.scroll.to button
The script would then look like:
require 'watir-webdriver'
require 'watir-scroll'
browser = Watir::Browser.new :chrome
browser.goto 'https://staging2.clearfit.com/assessment/assessment/95867fb272df436352a0bd5fbdd'
buttons = browser.elements(:class => "assessment-choice")
buttons.each do |button|
browser.scroll.to button
button.click
end
Firstly, this should be unnecessary. According to the spec, all element interactions require implicit scrolling to the element. If something does prevent this from happening, though, you can use this Selenium method instead of a javascript implementation:
buttons = browser.elements(:class => "assessment-choice")
buttons.each do |button|
button.wd.location_once_scrolled_into_view
button.click
end
public
def scroll_to(param)
args = case param
when :top, :start
'window.scrollTo(0, 0);'
when :center
'window.scrollTo(window.outerWidth / 2, window.outerHeight / 2);'
when :bottom, :end
'window.scrollTo(0, document.body.scrollHeight);'
when Array
['window.scrollTo(arguments[0], arguments[1]);', Integer(param[0]), Integer(param[1])]
else
raise ArgumentError, "Don't know how to scroll to: #{param}!"
end
#browser.execute_script(*args)
end
public
# This method pulls the object on the page you want to interact with, then it 'jumps to it'.
def jump_to(param)
# Leveraging the scroll_to(param) logic, this grabs the cooridnates,
# and then makes them an array that is able to be located and moved to.
# This is helpful when pages are getting too long and you need to click a button
# or interact with the browser, but the page 'Cannot locate element'.
location = param.wd.location
location = location.to_a
$helper.scroll_to(location)
end
Then you just call jump_to(element) and it "Jumps" to it.
This is how I got around it- not sure if that is a normal way. The problem is it goes to point (0,0); working on a version that moves to it to center screen.
I have logged in to Linkedin and reached my groups page using Ruby Mechanize. I am also able to retrieve the list of questions on the page. However, I am unable to click the "Show more" link at the bottom so that I can the entire page and hence all the questions:
require 'rubygems'
require 'mechanize'
require 'open-uri'
a = Mechanize.new { |agent|
# LinkedIn probably refreshes after login
agent.follow_meta_refresh = true
}
a.get('http://linkedin.com/') do |home_page|
my_page = home_page.form_with(:name => 'login') do |form|
form.session_key = '********' #put you email ID
form.session_password = '********' #put your password here
end.submit
mygroups_page = a.click(my_page.link_with(:text => /Groups/))
#puts mygroups_page.links
link_to_analyse = a.click(mygroups_page.link_with(:text => 'Semantic Web'))
link_to_test = link_to_analyse.link_with(:text => 'Show more...')
puts link_to_test.class
# link_to_analyse.search(".user-contributed .groups a").each do |item|
# puts item['href']
# end
end
Although a link exists with text 'Show more...' in the page, I am somehow not able to click it.the link_to_test.class shows NilClass What is the possible problem?
The part of the page I need to reach is:
<div id="inline-pagination">
<span class="running-count">20</span>
<span class="total-count">1134</span>
<a href="groups?mostPopularList=&gid=49970&split_page=2&ajax=ajax" class="btn-quaternary show-more-comments" title="Show more...">
<span>Show more...</span>
<img src="http://static01.linkedin.com/scds/common/u/img/anim/anim_loading_16x16.gif" width="16" height="16" alt="">
</a>
</div>
I need to click the show more... I can use links_with(:href => ..) but doesnt seem to work.
NEW ANSWER:
I just inspected the page source of the group and it seems that for the "Show more" link they actually use the three full stop characters and not an ellipsis.
Have you tried targeting the link by it's title attribute?
link_to_analyse.link_with(:title => 'Show more...')
If that's still not working, have you tried dumping the text of all the links on the page with
link_to_analyse.links.each do |link|
puts link.text
end
---- OLD ANSWER INCORRECT ----
LinkedIn use the "Horizontal Ellipsis" Unicode character (code U+2026) for their links that "look" like they have "..." at the end. So your code is not actually finding the link.
Character you need: http://www.fileformat.info/info/unicode/char/2026/index.htm
Sneaky :)
EDIT: and to get the link ofcourse you need to insert an appropriate Unicode character in your link text like so:
link_to_analyse.link_with(:text => 'Show more\u2026')
The tags inside the anchor will create some white space around the anchor text. You can account for that with:
link_to_analyse.link_with :text => /\A\s*Show more...\s*\Z/
But it's probably good enough to just do:
link_to_analyse.link_with :text => /Show more.../
Is there a straightforward way to set custom headers with Mechanize 2.3?
I tried a former solution but get:
$agent = Mechanize.new
$agent.pre_connect_hooks << lambda { |p|
p[:request]['Referer'] = 'https://wwws.mysite.com/cgi-bin/apps/Main'
}
# ./mech.rb:30:in `<main>': undefined method `pre_connect_hooks' for nil:NilClass (NoMethodError)
The docs say:
get(uri, parameters = [], referer = nil, headers = {}) { |page| ... }
so for example:
agent.get 'http://www.google.com/', [], agent.page.uri, {'foo' => 'bar'}
alternatively you might like:
agent.request_headers = {'foo' => 'bar'}
agent.get url
You misunderstood the code you were copying. There was a newline in the example, but it disappeared in the formatting as it wasn't tagged as code. $agent contains nil since you're trying to use it before it has been initialized. You must initialize the object and then use it. Just try this:
$agent = Mechanize.new
$agent.pre_connect_hooks << lambda { |p| p[:request]['Referer'] = 'https://wwws.mysite.com/cgi-bin/apps/Main' }
For this question I noticed people seem to use:
page = agent.get("http://www.you.com/index_login/", :referer => "http://www.you.com/")
As an aside, now that I tested this answer, it seems this was not the issue behind my actual problem: that every visit to a site I'm scraping requires going through the login sequence pages again, even seconds later after the first logged-in visit, despite that I'm always loading and saving the complete cookie jar in yaml format. But that would lead to another question of course.
I know is a very simple question but I've been stuck for an hour and I just can't understand how this works.
I need to scrape some stuff from my school's library so I need to insert 'CE' to a text field and then click on a link with text 'Clasificación'. The output is what I am going to use to work. So here is my code.
require 'rubygems'
require 'open-uri'
require 'nokogiri'
require 'mechanize'
url = 'http://biblio02.eld.edu.mx/janium-bin/busqueda_rapida.pl?Id=20110720161008#'
searchStr = 'CE'
agent = Mechanize.new
page = agent.get(url)
searchForm = page.form_with(:method => 'post')
searchForm['buscar'] = searchStr
clasificacionLink = page.link_with(:href => "javascript:onClick=set_index_and_submit(\'51\');").click
page = agent.submit(searchForm,clasificacionLink)
When I run it, it gives me this error
janium.rb:31: undefined method `[]=' for nil:NilClass (NoMethodError)
Thanks!
I think your problem is actually on line 13, not 31, and I'll even tell why I think that. Not only does your script not have 31 lines but, from the fine manual:
form_with(criteria)
Find a single form matching criteria.
There are several forms on that page that have method="post". Apparently Mechanize returns nil when it can't exactly match the form_with criteria including the single part mentioned in the documentation; so, if your criteria matches more than one thing, form_with returns nil instead of choosing one of the options and you end up trying to do this:
nil['buscar'] = searchStr
But nil doesn't have a []= method so you get your NoMethodError.
If you use this:
searchForm = page.form_with(:name => 'forma')
you'll get past the first part as there is exactly one form with name="forma" on that page. Then you'll have trouble with this:
clasificacionLink = page.link_with(:href => "javascript:onClick=set_index_and_submit(\'51\');").click
page = agent.submit(searchForm, clasificacionLink)
as Mechanize doesn't know what to do with JavaScript (at least mine doesn't). But if you use just this:
page = agent.submit(searchForm)
you'll get a page and then you can continue building and debugging your script.
mu's answer sounds reasonable. I am not sure if this is strictly necessary, but you might also try to put braces around searchStr.
searchForm['buscar'] = [searchStr]
Just for fun, I wrote a very small rails blog (just a hello world).
Now I want to create a post using mechanize.
So I created a Ruby Prog and started coding.
Here is my problem:
Rails creates my form element including all inputs.
In HTML my inputs look like this:
<input type="text" size="30" name="post[title]" id="post_title">
or
<textarea rows="20" name="post[description]" id="post_description" cols="40"></textarea>
Well...
Here is my Ruby Prog using Mechanize:
require 'rubygems'
require 'mechanize'
agent = WWW::Mechanize.new
page = agent.get('http://localhost:3000/posts/new')
target_form = page.form_with(:class => 'new_post')
target_form.post[title] = "test"
target_form.post[description] = "test"
page = agent.submit(target_form)
puts "end"
I know where my error is but I don't know how to fix it.
At target_form.post[title] = "test" it crashes, cause of
undefined method `name' for nil:NilClass (NoMethodError)
I think (please correct me), it's because of the input name, cause it is post[title] instead of only post right?
How can I fix it?
How about
target_form.field_with(:name => "post[title]").value = "test"
target_form.field_with(:name => "post[description]").value = "test"