Is there a way of accessing this dialog box to get the file name or to save this file somewhere so i can access it later. I am using Ruby mechanize to navigate through the website to get to this screen.
There is no dialog with mechanize. You submit the form, that returns a Mechanize::File object, and you can then save that like so:
file = form.submit
File.open('myfile','w'){|f| f << file.body}
I would do it this way.
Use nokogiri to open the page:
#doc = Nokogiri::HTML(open(url))
go through the doc page and find that link for download.
then you can use something link this:
require 'net/http'
Net::HTTP.start('theserver.com') { |http|
resp = http.get('/xx/the_file_to_downlaod.csv')
open('the_downlaod.csv', 'wb') { |file|
file.write(resp.body)
}
}
Related
I want to get taobao's list of URL of products on search result page without taobao API.
I tried following Ruby script.
require "open-uri"
require "rubygems"
require "nokogiri"
url='https://world.taobao.com/search/search.htm?_ksTS=1517338530524_300&spm=a21bp.7806943.20151106.1&search_type=0&_input_charset=utf-8&navigator=all&json=on&q=%E6%99%BA%E8%83%BD%E6%89%8B%E8%A1%A8&cna=htqfEgp0pnwCATyQWEDB%2FRCE&callback=__jsonp_cb&abtest=_AB-LR517-LR854-LR895-PR517-PR854-PR895'
charset = nil
html = open(url) do |f|
charset = f.charset
f.read
end
doc = Nokogiri::HTML.parse(html, nil, charset)
p doc.xpath('//*[#id="list-itemList"]/div/div/ul/li[1]/div/div[1]/div/a/#href').each{|i| puts i.text}
# => 0
I want to get list of URL like https://click.simba.taobao.com/cc_im?p=%D6%C7%C4%DC%CA%D6%B1%ED&s=328917633&k=525&e=lDs3%2BStGrhmNjUyxd8vQgTvfT37ERKUkJtUYVk0Fu%2FVZc0vyfhbmm9J7EYm6FR5sh%2BLS%2FyzVVWDh7%2FfsE6tfNMMXhI%2B0UDC%2FWUl0TVvvELm1aVClOoSyIIt8ABsLj0Cfp5je%2FwbwaEz8tmCoZFXvwyPz%2F%2ByQnqo1aHsxssXTFVCsSHkx4WMF4kAJ56h9nOp2im5c3WXYS4sLWfJKNVUNrw%2BpEPOoEyjgc%2Fum8LOuDJdaryOqOtghPVQXDFcIJ70E1c5A%2F3bFCO7mlhhsIlyS%2F6JgcI%2BCdFFR%2BwwAwPq4J5149i5fG90xFC36H%2B6u9EBPvn2ws%2F3%2BHHXRqztKxB9a0FyA0nyd%2BlQX%2FeDu0eNS7syyliXsttpfoRv3qrkLwaIIuERgjVDODL9nFyPftrSrn0UKrE5HoJxUtEjsZNeQxqovgnMsw6Jeaosp7zbesM2QBfpp6NMvKM5e5s1buUV%2F1AkICwRxH7wrUN4%2BFn%2FJ0%2FIDJa4fQd4KNO7J5gQRFseQ9Z1SEPDHzgw%3D however I am getting 0
What should I do?
I don't know taobao.com but the page seems like its running lots of javascript. So perhaps the content can actually not be retrieved with a client without javascript capabilities. So instead of open-uri, you could try the gem selenium-webdriver:
https://rubygems.org/gems/selenium-webdriver/versions/2.53.4
I'm using watir-webdriver to do my GUI smoke tests, and one area I'd like to test out is redirecting from a dynamic url
Is it possible to save the url to a file, then load it for use?
What I'm thinking of in pseudo code:
#browser.goto 'google.com'
#browser.url.save
in another test
#browser.load url
continue testing....
Is there a way to do this?
To write a string to a file you can just do that:
File.open('path/to/yourfile', 'w') { |file| file.write(#browser.url) }
You can use it in the other test like this:
File.open('path/to/yourfile', "rb") { |file| #browser.goto(file.read) }
I am trying to scrape a website and store data in XML using Mechanize and Nokogiri. I didn't set up a Rails project and I am only using Ruby and IRB.
I wrote this method:
def mechanize_club
agent = Mechanize.new
agent.get("http://www.rechercheclub.applipub-fft.fr/rechercheclub/")
form = agent.page.forms.first
form.field_with(:name => 'codeLigue').options[0].select
form.submit
page2 = agent.get('http://www.rechercheclub.applipub-fft.fr/rechercheclub/club.do?codeClub=01670001&millesime=2015')
body = page2.body
html_body = Nokogiri::HTML(body)
codeclub = html_body.search('.form').children("tr:first").children("th:first").to_i
#codeclubs << codeclub
filepath = '/davidgeismar/Documents/codeclubs.xml'
builder = Nokogiri::XML::Builder.new(encoding: 'UTF-8') do |xml|
xml.root {
xml.codeclubs {
#codeclubss.each do |c|
xml.codeclub {
xml.code_ c.code
}
end
}
}
end
puts builder.to_xml
end
My first problem is that I don't know how to test my code.
I call ruby webscraper.rb in my console, the file is treated I think, but it doesn't create an XML file in the specified path.
Then, more specifically I am quite sure this code is wrong as I didn't get a chance to test it.
Basically what I am trying to do is to submit a form several times:
agent = Mechanize.new
agent.get("http://www.rechercheclub.applipub-fft.fr/rechercheclub/")
form = agent.page.forms.first
form.field_with(:name => 'codeLigue').options[0].select
form.submit
I think this code is ok, but I dont want it to only select options[0], I want it to select an option, then scrape all the data I need, then go back to page, then select options[1]... until there are no more options (an iteration I guess).
the file is treated I think, but it doesnt create an xml file in the specified path.
There is nothing in your code that creates a file. You print some output, but don't do anything to open or write a file.
Perhaps you should read the IO and File documentation and review how you are using your filepath variable?
The second problem is that you don't call your method anywhere. Though it's defined and Ruby will see it and parse the method, it has no idea what you want to do with it unless you invoke the method:
def mechanize_club
...
end
mechanize_club()
I'm having a trouble with Mechanize gem, how to convert Mechanize::File into Mechanize::Page,
here's my piece of code:
**link** = page.link_with(:href => %r{/en/users}).click
when users link clicked it goes to the page with the list of users, now i want to click the first user, but i can't achieve this, because link return Mechanize::File object
Any help, suggestions 'd be great, thanks
Mechanize uses Content-Type to determine how the resource should be handled. Occasionally websites will not set the mime-types for their resources. Mechanize::File is the default for unset Content-Type.
If you are only dealing with 'text/html' you can following Jimm Stout's suggestion of using post_connect_hooks
agent = Mechanize.new do |a|
a.post_connect_hooks << ->(_,_,response,_) do
if response.content_type.empty?
response.content_type = 'text/html'
end
end
end
Just parse the body with nokogiri:
link = page.link_with(:href => %r{/en/users}).click
doc = Nokogiri::HTML link.body
agent.get doc.at('a')[:href]
I'm trying to call resources (images, for example.) from my website to avoid constant updates. Thus far, I've tried just using this:
#sprite.bitmap = Bitmap.new("http://www.minscandboo.com/minscgame/001-Title01.jpg")
But, this just gives "File not found error". What is the correct method for achieving this?
Try using Net::HTTP to get a local file first:
require 'net/http'
Net::HTTP.start("minscandboo.com") { |http|
resp = http.get("/miscgame/001-Title01.jpg")
open("local-game-image.jpg", "wb") { |file|
file.write(resp.body)
}
}
# ...
#sprite.bitmap = Bitmap.new("local-game-image.jpg")