I have some 'generic' methods that extract data based on css selectors that usually are the same in many websites. However I have another method that accept as argument the css selector for a given website.
I need to call the get_title method if title_selector argument is nos passed. How can I do that?
Scrape that accept css selectors as arguments
def scrape(urls, item_selector, title_selector, price_selector, image_selector)
collection = []
urls.each do |url|
doc = Nokogiri::HTML(open(url).read) # Opens URL
#items = doc.css(item_selector)[0..1].map {|item| item['href']} # Sets items
#items.each do |item| # Donwload each link and parse
page = Nokogiri::HTML(open(item).read)
collection << {
:title => page.css(title_selector).text, # I guess I need conditional here
:price => page.css(price_selector).text
}
end
#collection = collection
end
end
Generic title extractor
def get_title(doc)
if doc.at_css("meta[property='og:title']")
title = doc.css("meta[property='og:title']")
else doc.css('title')
title = doc.at_css('title').text
end
end
Use an or operator inside your page.css call. It will call get_title if title_selector is falsey (nil).
:title => page.css(title_selector || get_title(doc)).text,
I'm not sure what doc should actually be in this context, though.
EDIT
Given your comment below, I think you can just refactor get_title to handle all of the logic. Allow get_title to take an optional title_selector parameter and add this line to the top of your method:
return doc.css(title_selector).text if title_selector
Then, my original line becomes:
:title => get_title(page, title_selector)
Related
I am working on a CLI Project and trying to open up a web page by using url variable declared in another method.
def self.open_deal_page(input)
index = input.to_i - 1
#deals = PopularDeals::NewDeals.new_deals
#deals.each do |info|
d = info[index]
#product_url = "#{d.url}"
end
#product_url.to_s
puts "They got me!"
end
def self.deal_page(product_url)
#self.open_deal_page(input)
deal = {}
html = Nokogiri::HTML(open(#product_url))
doc = Nokogiri::HTML(html)
deal[:name] = doc.css(".dealTitle h1").text.strip
deal[:discription] = doc.css(".textDescription").text.strip
deal[:purchase] = doc.css("div a.button").attribute("href")
deal
#binding.pry
end
but I am receiving this error.
`open': no implicit conversion of nil into String (TypeError)
any possible solution? Thank you so much in advance.
Try returning your #product_url within your open_deal_page method, because now you're returning puts "They got me!", and also note that your product_url is being created inside your each block, so, it won't be accessible then, try creating it before as an empty string and then you can return it.
def open_deal_page(input)
...
# Create the variable
product_url = ''
# Assign it the value
deals.each do |info|
product_url = "#{info[index].url}"
end
# And return it
product_url
end
In your deal_page method tell to Nokogiri to open the product_url that you're passing as argument.
def deal_page(product_url)
...
html = Nokogiri::HTML(open(product_url))
...
end
I am trying to do a post and run some if statement. What I want to do is:
check all fields are filled
if all fields are filled move on to next step, or else reload page
check if already in data base
add if not already in data base
post "/movies/new" do
title = params[:title]
year = params[:year]
gross = params[:gross]
poster = params[:poster]
trailer = params[:trailer]
if title && year && gross && poster && trailer
movie = Movie.find_by(title: title, year: year, gross: gross)
if movie
redirect "/movies/#{movie.id}"
else
movie = Movie.new(title: title, year: year, gross: gross, poster: poster, trailer: trailer)
if movie.save
redirect "/movies/#{movie.id}"
else
erb :'movies/new'
end
end
else
erb :'movies/new'
end
end
I don't think my if statement is correct. It works even if all my fields are not filled
Your code is doing a lot of work in one single method. I would suggest to restructure it into smaller chunks to make it easier to manage. I mostly code for Rails, so apologies if parts of these do not apply to your framework.
post "/movies/new" do
movie = find_movie || create_movie
if movie
redirect "/movies/#{movie.id}"
else
erb :'movies/new'
end
end
def find_movie
# guard condition to ensure that the required parameters are there
required_params = [:title, :year, :gross]
return nil unless params_present?(required_params)
Movie.find_by(params_from_keys(required_params))
end
def create_movie
required_params = [:title, :year, :gross, :poster, :trailer]
return nil unless params_present?(required_params)
movie = Movie.new(params_from_keys(required_params))
movie.save ? movie : nil # only return the movie if it is successfully saved
end
# utility method to check whether all provided params are present
def params_present?(keys)
keys.each {|key| return false if params[key].blank? }
true
end
# utility method to convert params into the hash format required to create / find a record
def params_from_keys(keys)
paras = {}
keys.each { |key| paras.merge!(key: params[key]) }
paras
end
Even if you type nothing in the HTML fields, they will still be submitted as empty strings.
You can avoid having empty parameters by, for example, filtering them:
post '/movies/new' do
params.reject! { |key, value| value.empty? }
# rest of your code
end
Also I would rather post to /movies rather than to /movies/new, that's more REST-wise.
Try if condition to check fields are blank like below -
unless [title, year, gross, poster, trailer].any?(&:blank?)
This will check any of the field should not be nil or blank("").
I have been trying to use Minitest to test my code (full repo) but am having trouble with one method which downloads a SHA1 hash from a .txt file on a website and returns the value.
Method:
def download_remote_sha1
#log.info('Downloading Elasticsearch SHA1.')
#remote_sha1 = ''
Kernel.open(#verify_url) do |file|
#remote_sha1 = file.read
end
#remote_sha1 = #remote_sha1.split(/\s\s/)[0]
#remote_sha1
end
You can see that I log what is occurring to the command line, create an object to hold my SHA1 value, open the url (e.g. https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.deb.sha1.txt)
I then split the string so that I only have the SHA1 value.
The problem is that during a test, I want to stub the Kernel.open which uses OpenURI to open the URL. I would like to ensure that I'm not actually reaching out to download any file, but rather I'm just passing the block my own mock IO object testing just that it correctly splits stuff.
I attempted it like the block below but when #remote_sha1 = file.read occurs the file item is nil.
#mock_file = Minitest::Mock.new
#mock_file.expect(:read, 'd377e39343e5cc277104beee349e1578dc50f7f8 elasticsearch-1.4.2.deb')
Kernel.stub :open, #mock_file do
#downloader = ElasticsearchUpdate::Downloader.new(hash, true)
#downloader.download_remote_sha1.must_equal 'd377e39343e5cc277104beee349e1578dc50f7f8'
end
I was working on this question too, but matt figured it out first. To add to what matt posted:
When you write:
Kernel.stub(:open, #mock_file) do
#block code
end
...that means when Kernel.open() is called--in any code, anywhere before the stub() block ends--the return value of Kernel.open() will be #mock_file. However, you never use the return value of Kernel.open() in your code:
Kernel.open(#verify_url) do |f|
#remote_sha1 = f.read
end
If you wanted to use the return value of Kernel.open(), you would have to write:
return_val = Kernel.open(#verify_url) do |f|
#remote_sha1 = f.read
end
#do something with return_val
Therefore, the return value of Kernel.open() is irrelevant in your code--which means the second argument of stub() is irrelevant.
A careful examination of the source code for stub() reveals that stub() takes a third argument--an argument which will be passed to a block specified after the stubbed method call. You, in fact, have specified a block after your stubbed Kernel.open() method call:
stubbed method call -+ +- start of block
| | |
V V V
Kernel.open(#verify_url) do |f|
#remote_sha1 = f.read
end
^
|
end of block
So, in order to pass #mockfile to the block you need to specify it as the third argument to Kernel.stub():
Kernel.stub(:open, 'irrelevant', #mock_file) do
end
Here is a full example for future searchers:
require 'minitest/autorun'
class Dog
def initialize
#verify_url = 'http://www.google.com'
end
def download_remote_sha1
#remote_sha1 = ''
Kernel.open(#verify_url) do |f|
#remote_sha1 = f.read
end
#puts #remote_sha1[0..300]
#remote_sha1 = #remote_sha1.split(" ")[0] #Using a single space for the split() pattern will split on contiguous whitespace.
end
end
#Dog.new.download_remote_sha1
describe 'downloaded file' do
it 'should be an sha1 code' do
#mock_file = Minitest::Mock.new
#mock_file.expect(:read, 'd377e39343e5cc277104beee349e1578dc50f7f8 elasticsearch-1.4.2.deb')
Kernel.stub(:open, 'irrelevant', #mock_file) do
#downloader = Dog.new
#downloader.download_remote_sha1.must_equal 'd377e39343e5cc277104beee349e1578dc50f7f8'
end
end
end
xxx
The second argument to stub is what you want the return value to be for the duration of your test, but the way Kernel.open is used here requires the value it yields to the block to be changed instead.
You can achieve this by providing a third argument. Try changing the call to Kernel.stub to
Kernel.stub :open, true, #mock_file do
#...
Note the extra argument true, so that #mock_file is now the third argument and will be yielded to the block. The actual value of the second argument doesn’t really matter in this case, you might want to use #mock_file there too to more closely correspond to how open behaves.
Here's my problem:
Camping is splitting urls at question marks.
So if we have some code like this:
Camping.goes :CodeLine
module CodeLine::Controllers
class Index
def get
render :index
end
end
class TextEntered < R '/(.*)'
def get(textStringEntered)
"#{textStringEntered}"
end
end
end
module CodeLine::Views
def index
html do
head do
title "Uh Oh"
end
body do
text "Looks like you got to the index"
br
br
form :name => "input" do
input :type => "text", :name => "text"
input :type => "submit", :value => "Submit"
end
end
end
end
end
Run camping path/to/file
After going to localhost:3301 in your browser and entering some text in the text field and hitting submit, you should see everything after the slash, but instead it splits the url at the question mark and because it thinks there is nothing after the slash, it takes you to the index.
Question:
Is it possible to set up input so it does not use a question mark, or can I make camping not split at the question mark?
Appendix A
Tested in
1. Google Chrome
2. Firefox
3. Safari
The route only matches the path of the URL:
https://example.com/hello/world?a=this&b=hello&c=world#nice
^ ^ ^ ^ ^
Schema Host Path Query parameters Fragment
In Camping you get access to the query parameters through #input:
#input.a # => "this"
#input.b # => "hello"
#input.c # => "world"
Query parameters are more like "options" that you can pass to the controller. For example, you don't want to have a separate controller to handle "sorting by name" and "sorting by date", so instead you use query parameters:
class Search
def get
query = #input.q || "*"
page = (#input.page || 1).to_i
sort = #input.sort || "name"
#results = fetch_results_from_database_or_something(query, page, sort)
render :search
end
end
That way, all of these works:
/search?query=hello # Page 1, sort by name
/search?page=5 # Page 5, sort by name, search for everything
/search?query=cars&page=4&sort=date
I want to parse a Markdown document so I get a tree structure that I am able to manipulate. Afterwards I want the output to be Markdown again.
Example:
# This is a title
And a short paragraph...
m = SomeLib.parse("# This is a tit...")
m.insert(1, "Here is a new paragraph") # or something simmilar
m.to_md
Should become
# This is a title
Here is a new paragraph
And a short paragraph...
As I want to heavily change the document I do not want to use REGEX or simillar techniques.
I looked into Maruku and BlueCloth but somehow I cannot generate Markdown again.
Probably not out of the box, but using redcarpet you could write a custom renderer to build your tree and then manipulate it.
Though beware in this case you can't reuse the Markdown and Renderer instance and all methods in the custom Renderer subclass are supposed to return a string. Something like this could be a starting point:
class StackRenderer < Redcarpet::Render::Base
attr_reader :items
def initialize
super
#items = []
end
def header(title, level)
items << { :text => title, :level => level, :type => :header }
"#{'#' * level} #{title}\n\n"
end
def paragraph(text)
items << { :text => text, :type => :paragraph }
"#{text}\n\n"
end
end
# example...
sr = StackRenderer.new
md = Redcarpet::Markdown.new(sr)
text = <<-EOF
# This is a title
And a short paragraph...
EOF
md.render(text) # => "# This is a title\n\nAnd a short paragraph...\n\n"
sr.items # => [{:type=>:header, :level=>1, :text=>"This is a title"},
# {:type=>:paragraph, :text=>"And a short paragraph..."}]