There's a website that I want to scrape (FWIW it's svpply) and there's a button that shows up sometimes saying "Show All", it's an HTML <button> element. Is there any way to use Ruby to emulate a click of this button, and get the content of the full page that results after clicking that button, since the button reveals more content?
The "Show All" button triggers a javascript ajax request. The only way to automate that is to use a library that can execute javascript. Libraries such as Mechanize and ScrAPI will not work.
What will work are tools that drive an actual browser such as watir and selenium. I installed watir-webdriver and successfully got it to click the button and show additional products.
require 'watir-webdriver'
b = Watir::Browser.new
b.goto 'svpply.com/editors_pick'
#count products
puts b.elements(:xpath => '//li[#data-class="Product"]').count
#=> 30
#Now click button
show_all = b.button(:id => "btn_all")
show_all.click
sleep 4
#count products again
puts b.elements(:xpath => '//li[#data-class="Product"]').count
#=>60
Mechanize can do this nicely for you.
If the page is reloaded when you press the button, you can, otherwise you need something that can parse javascript. If you understand where the website redirects you after pressing the button (even on the same page but with some parameters set, use firebug for this purpose), you can eventually read what you need.
Related
I'm looking for a way to programmatically click the drop-down arrow that appears when we open a specific FB post such as https://www.facebook.com/userxxx/posts/10155300116786742
Can anyone please provide me the code for Visual Basic? Thank you.
Drop-down arrow
in your webbrowser control you can invoke javascript code inside of the requested page by using
WebBrowser.Document.InvokeScript("eval", "some javascrtip here")
that leaves us with the problem of doing the dropdown click with javascript. That you can do with:
document.querySelectorAll("[data-testid=post_chevron_button]")[0].click()
so, all together it end up being
Dim micode As Object() = {"document.querySelectorAll('[data-testid=post_chevron_button]')[0].click()
"}
WebBrowser.Document.InvokeScript("eval", micode)
notice the js code provided might change. You should decide what code is better yourself. I recommend open facebook or any page, in chrome and press F12 ... that is the developer console. You can run and debug javascript there.... have fun!
I am using firebug and firepath to see xpath or html code to use in my Selenium tests. But for one popup window I cannot see any html code or firepath...
I also tried to right click to see xpath but I do not get any menu when right clicking on the popup.
It works fine on the rest of the wesite it is only this popup.. Is there any setting or something I have to do?
This is a java script alert. You will have to switch to the alert and then Use alert.accept() to mimic clicking on OK button of the alert.
clicking a text agent.click(page.link_with(:text => 'some_text') with mechainze is piece of cake. How to click an image with mechanize?
Clicking on a pure HTML image will typically have no effect. If the image has an onclick handler, you will not be able to click on it with Mechanize as it does not support javascript.
You may want to use something like Capybara with the Webkit, PhantomJS or Selenium driver instead.
It is rather similar. You just need to grab one of the attributes of your image. have a look below..:
agent.click(page.image_with(:alt=> 'your image')
Hi All I am having problems accessing a form with in a popup window that is opened after I click on a link. I appears once I have clicked the link it causes the script to hang and will not even time out. I need to be able to access the form, set some text fields and the click the submit button.
Link code:
<a id="ctl00_ContentPlaceHolder2_ctrlPageHeader1_aFilter" class="RightTextHeading" onclick="javascript:openMdlWindow('InvestmentDetailOptions.aspx?IDAssetType=','620','600');if(window.document.RetValue == '2'){window.parent.LoadinIframe('InvestmentDetail.aspx?FromMenu=N&IDAssetType=','Investment Details > Full View','false');}" style="text-decoration:none;">Filter</a>
I have tried everything but nothing seems to work. Has anyone ever came across this before and have a solution?
Thanks
Did you try this:
browser.window(:title => "annoying popup").use do
browser.button(:id => "close").click
end
More information: http://watirwebdriver.com/browser-popups/
I am trying to using watir in ruby, I am able open a browser,
enter some values to a username/password form, but then I press enter, then I click the submit button (actually it is pressing an enter, it is easiler to code), it would pop up a new browser windows for our application, then I found I have no control to the new browser. What can I do?
In addition to my problem, the new browser window got no menubar, no toolbar, no navigation bar, thus I cannot open the IE developer toolbar to find the name of the element in the webpage in the new browser.
By the way, my app can only support IE.
I tried the attach method, but it does work:
C:/Ruby187/lib/ruby/gems/1.8/gems/watir-1.8.1/lib/watir/ie-class.rb:302:
in `attach_browser_window': Unable to locate a window with title of (?-mix:New browser title) (Watir::Exception::NoMatchingWindowFoundException)
from C:/Ruby187/lib/ruby/gems/1.8/gems/watir-1.8.1/lib/watir/ie-class.rb
:150:in `_attach_init'
from C:/Ruby187/lib/ruby/gems/1.8/gems/watir-1.8.1/lib/watir/ie-class.rb
:144:in `attach'
from test1.rb:36
Attach method is able to handle new window.
http://rdoc.info/gems/watir/1.8.1/Watir/IE.attach
for example
browser = Watir::Browser.new
browser.text_field(:id,'username').set 'username')
browser.button(:index,1).click
# popped up the new window
popup = Watir::Browser.attach(:title,'Foobar')
If you will use only the popped up window, you can overwrite browser variable instead.
You could try to authenticate like so:
examplehost.com/authentication?user=foo&password=bar
You could also try to remove "target" attribute (which is probably equal to "_blank") of that authentication form ( i'm pretty much sure you should be able to achive this with watir )
EDIT:
Also, there's this option:
ie2 = Watir::IE.attach(:title, ‘New Ie window, opened upon form submit’)
it can also attach by :url instead of :title
ok, there's possibly this solution that might work:
b = Watir::Browser.start "http://yourpage"
b.execute_script("document.getElementById('your-image-id').onClick = function() {}")
so, the idea is to get that image and overwrite it's onclick event