I know I'm being an idiot here, but I can't think of how this is done. I am creating an app with certain interests and am using a a Wikipedia scrape set up using Nokogiri. I have two inputs: Title and Wikipedia, but want to fill Summary and Content in the data model using the scrape. I want to use the Wikipedia attribute as a variable in a url within a method, but keep getting the error dynamic constant assignment PAGE_URL = "http://en.wikipedia.org/w/i....
I thought that the methods should go in the model, with reference to them in the Create definition under the controller, but this doesn't seem to work.
EDIT
I've just tried taking the constants out of the methods as suggested, but I am still getting a dynamic constant assignment error. My model currently looks like this:
PAGE_URL1 = "http://en.wikipedia.org/w/index.php?title="
PAGE_URL2 = "&printable=yes"
def get_PAGE_URL
PAGE_URL = PAGE_URL1 + self.wikipedia + PAGE_URL2
end
def get_page
page = Nokogiri::HTML(open(PAGE_URL))
end
def get_summary
get_PAGE_URL
self.summary = page.css("p")[0].text
end
def get_full_page
get_PAGE_URL
puts page.css('div#content.mw-body div#bodyContent div#mw-content-text.mw-content-ltr p').each do |p|
self.content = puts p.text
end
end
Constants can't go inside of methods, they must be defined inside of the class' direct scope.
Edit:
For example:
class WikiScraper
PAGE_URL = "http://www.wikipedia.org/"
def scrape
page_num = '5'
my_url = PAGE_URL + page_num
end
end
Related
Solved. I didn't have the initialization right, which gave the noMethodError. Then I was changing an array, but checking a variable that referred to a position in the array, and that variable had not been reassigned.
Edited to initialize bookPagesInfo, bookChaptersInfo, editPagesInfo, and editChapterInfo as suggested. Still gives the same NoMethod Error.
I have a book with page and chapter info, and want to be able to apply edits that change the number of pages, introPages, chapters, and povs.
class Book
attr_accessor :pages, :chapters, :bookPagesInfo, :bookChaptersInfo, :introPages, :povs
def initialize(bookPagesInfo, bookChaptersInfo)
#bookPagesInfo = bookPagesInfo
#bookChaptersInfo = bookChaptersInfo
#pages = bookPagesInfo[0]
#introPages = bookPagesInfo[1]
#chapters = bookChaptersInfo[0]
#povs = bookChaptersInfo[1]
end
def applyEdit(edit)
#pages += edit.new_pages
end
end
class Edit
attr_accessor :new_pages, :new_chapters, :editPagesInfo, :editChaptersInfo, :new_intro_pages, :new_povs
def initialize(editPagesInfo, editChaptersInfo)
#editPagesInfo = editPagesInfo
#editChaptersInfo = editChaptersInfo
#new_pages = editPagesInfo[0]
#new_intro_pages = editPagesInfo[1]
#new_chapters = editChaptersInfo[0]
#new_povs = editChaptersInfo[1]
end
end
The above code works for editing just number of pages. However, if I change my applyEdit method to iterate over the bookPagesInfo array, I can't get it to work. Running applyEdit below gives a nonfatal error.
def applyEdit(edit)
#bookPagesInfo.each_with_index do {|stat, idx| stat += edit.bookPagesInfo[idx]}
end
## gives undefined method `each_with_index' for nil:NilClass (NoMethodError), but
## my understanding is as long as bookPagesInfo was initialized as an array, it
## should be an array, not nilClass
I'm pretty new to classes (and this website, sorry for formatting). Thanks for the help.
You've got attr_accessors defined for :bookPagesInfo and :bookChaptersInfo, which will give you reader and writer methods, but it won't set #bookPagesInfo and #bookChaptersInfo for you in the initialize method - you need to do that yourself. So, when you try to read from the instance variable in applyEdit, you're reading nil.
Try adding
#bookPagesInfo = bookPagesInfo
#bookChaptersInfo = bookChaptersInfo
in Book#initialize.
I'm building a small web scraper using Ruby and now I'm trying to refactor my code. Unfortunately, I'm encountering some errors while I'm refactoring my code. This is one of the errors.
Basically, I'm calling two separate methods in the first method which is whey_scrapper. Each of these two methods are basically responsible of scraping a specific item on the webpage. When I run and debug this code with byebug, I basically try to display the products or prices I've scraped but I get an error message saying that 'products' or 'prices' is undefined. This is my current code:
require 'open-uri'
require 'nokogiri'
require 'httparty'
require 'byebug'
require 'csv'
class Wheyscrapper
def whey_scrapper
company = 'Body+%26+fit'
url = "https://www.bodyenfitshop.nl/afslanken/afslank-toppers/?manufacturer=#{company}"
unparsed_page = open(url).read
parsed_page = Nokogiri::HTML(unparsed_page)
product_scrapper
prices_scrapper
# csv = CSV.open('wheyprotein.csv', 'wb')
end
def product_scrapper
products = Array.new
product_names = parsed_page.css('div.product-primary')
product_names.each do |product_name|
product = {
name: product_name.css('h2.product-name').text
}
products << product
end
end
def prices_scrapper
prices = Array.new
product_prices = parsed_page.css('div.price-box')
product_prices.each do |product_price|
price = {
amount: product_price.css('span.price').text
}
prices << price
end
end
byebug
whey_scrapper
end
There's a lot going on here, but to make it more Ruby you'd consider making those lazy-initialized and giving them names that reflect that:
class Wheyscrapper
URL = "https://www.bodyenfitshop.nl/afslanken/afslank-toppers/?%s"
def initialize(company:)
#company = company
# Use encode_www_form to encode query-string parameters
#url = URL % URI.encode_www_form(manufacturer: company)
end
def document
# Lazy-initialize a parsd version of the page
#document ||= Nokogiri::HTML(open(url).read)
end
def products
document.css('div.product-primary').map do |product_name|
{
name: product_name.css('h2.product-name').text
}
end
end
def prices
document.css('div.price-box').map do |product_price|
{
amount: product_price.css('span.price').text
}
end
end
end
This fixes a lot of the data propagation problems you had in your original. When you declare a variable it's a local variable, meaning it doesn't exist outside of that particular call of that particular method. If you want to persist it for longer you need to use instance variables, as in #products, or you need to define methods that return the data you need.
The above approach combines that, using a lazy-initialized instance variable to persist the parsed document, and exposes that as a method the other methods can use.
Now you can spin this up:
scraper = WheyScraper.new(company: "Body & Fit")
Where that should enable everything to be available directly:
scraper.prices
scraper.products
When you learn how to use Ruby effectively you'll often find solutions to your problems that are really minimal. Usually a lot of Ruby code is a sign that it's not being used properly.
This should be refactored in a better way but this should at least work without refactor, based on my comments above
require 'open-uri'
require 'nokogiri'
require 'httparty'
require 'csv'
class Wheyscrapper
def whey_scrapper
company = 'Body+%26+fit'
url = "https://www.bodyenfitshop.nl/afslanken/afslank-toppers/?manufacturer=#{company}"
unparsed_page = open(url).read
#parsed_page = Nokogiri::HTML(unparsed_page)
product_scrapper
prices_scrapper
# csv = CSV.open('wheyprotein.csv', 'wb')
end
def product_scrapper
#products = Array.new
product_names = #parsed_page.css('div.product-primary')
product_names.each do |product_name|
product = {
name: product_name.css('h2.product-name').text
}
#products << product
end
end
def prices_scrapper
#prices = Array.new
#product_prices = #parsed_page.css('div.price-box')
#product_prices.each do |product_price|
price = {
amount: product_price.css('span.price').text
}
#prices << price
end
end
end
w = Wheyscrapper.new.whey_scrapper
I'm new to Ruby and having a little trouble json. I have inherited my classes with custom made JSONable class, as explained HERE in this answer. I have customized it according to my need, but I couldn't figure out how to make it work with custom nested (complex) objects, according to my requirement. I have following scenario.
First Class:
class Option < JSONable
def IncludeAll=(includeAll) #bool
#includeAll = includeAll
end
def IncludeAddress=(includeAddress) #bool
#includeAddress= includeAddress
end
......
Second Class:
class Search < JSONable
def CustomerId=(customerId)
#customerId = customerId
end
def identifier=(identifier)
#identifier = identifier
end
def Options=(options) #This is expected to be of Class Option, declared above
#options = options
end
Third Class:
class Request < JSONable
def DateTimeStamp=(dateTimeStamp)
#dateTimeStamp = dateTimeStamp
end
def SDKVersion=(sDKVersion)
#sDKVersion = sDKVersion
end
def RequestMessage=(requestMessage) #This is of type Search, declared above
#requestMessage = requestMessage
end
I call it as:
search = Search.new
searchOpts = Options.new
request = Request.new
search.identifier = identifier
searchOpts.IncludeAll = false
searchOpts.IncludeAddress = true
search.Options = searchOpts #setting nested level2 property here
//THE MOST OUTER CLASS OBJECT
request.SDKVersion = "xyz"
request.RequestMessage = search #setting nested level1
My ultimate goal is to send this request object to an API, after converting it to JSON. so i call to_json on request object as:
request.to_json
But here, suggested solution in that post (JSONable) fails in this case, as it can't convert the nested complex objects request.search and request.search.Options to Json.
(gives error: in 'to_json': wrong number of arguments (1 for 0) (ArgumentError)')
What I tried:
class JSONable
def to_json
hash = {}
self.instance_variables.each do |var|
#hash[var] = self.instance_variable_get var #tried to apply following check
if((self.instance_variable_get var).instance_of? Options ||((varVal).instance_of? Search))
varVal = self.instance_variable_get var
hash[var] = varVal.to_json #convert inner object to json
else
hash[var] = self.instance_variable_get var
end
end
hash.to_json
end
.....
This converts the nested model without any problem, but it messes up the 3rd level json. The result is as following:
{"DateTimeStamp":"121212","SDKVersion":"1.5","Culture":"en","RequestMessage":"{\"identifier\":\"851848913\",\"Options\":\"{\\\"IncludeAll\\\":true,\\\"IncludeAssociatedEntities\\\":true,\\\"IncludeAddress\\\":true,\\\"IncludePaymentInstructions\\\":true}\"}"}
And API doesn't respond. It seems as it messes up the boolean variables, which should be something like:
"SearchOption":"{\"IncludeAll\":true,\"IncludeAssociatedEntities\":true,\...
but it gives:
"SearchOption\":\"{\\\"IncludeAll\\\":true,\\\"IncludeAssociatedEntities\\\":true,\\\"Includ...
So the API logic can't cast it to corresponding bool objects anymore. JSON validator also fails to validate this result, i checked online
Questions:
How can I avoid this, and produce valid JSON in this case?
How can I apply generic check to in my JSONable class to check if the object is of some custom class / complex object.
(currently i have checked only for specific classes as:)
if((self.instance_variable_get var).instance_of? Options ||((varVal).instance_of? Search))
Other Info:
It works fine for all complex objects, having no nested objects
API is developed in .NET
I'm not using Rails, its a Ruby console app (I'm new to Ruby)
The answer you referred is dated “Dec 2010.” JSON library is included in ruby stdlib for years already and it perfectly converts Hash instances to json. That said, you just need to construct hashes out of your objects and then call JSON.dump on the resulting hash. I have no idea what JSONable is and you definitely do not need it. Introduce some base class, let’s call it Base:
class Base
def to_h
instance_variables.map do |iv|
value = instance_variable_get(:"##{iv}")
[
iv.to_s[1..-1], # name without leading `#`
case value
when Base then value.to_h # Base instance? convert deeply
when Array # Array? convert elements
value.map do |e|
e.respond_to?(:to_h) ? e.to_h : e
end
else value # seems to be non-convertable, put as is
end
]
end.to_h
end
end
Now just derive your classes from Base to make them respond to to_h, define all your instance variables as you did, and call:
require 'json'
JSON.dump request.to_h # request.to_h.to_json should work as well
The above should produce the nested JSON, hashes are happily converted to json by this library automagically.
I'm writing a wrapper for an XML API using Nokogiri to build the XML for submission.
In order to keep my code DRY, I'm using custom blocks for the first time and just getting to grips with how to pass variables back and forth and how that works.
What I'm doing at the moment is this:
# Generic action
def action(xml, action_title, test=false)
xml.request do
xml.login do
xml.username("my_user")
xml.password("my_pass")
end
xml.action(action_title)
xml.params do
yield
end
end
end
# Specific action
def get_users(city = "", gender = "")
build = Nokogiri::XML::Builder.new do |xml|
action(xml, "getusers") do
xml.city(city) unless city.blank?
xml.gender(gender) unless gender.blank?
end
end
do_stuff_to(build)
end
Ideally, I'd like to the specific action method to look like this:
def get_users(city = "", gender = "")
action("getusers") do |xml|
xml.city(city) unless city.blank?
xml.gender(gender) unless gender.blank?
end
end
In doing so, I'd want the other logic currently in the specific action method to be moved to the generic action method with the generic action method returning the results of do_stuff_to(build).
What I'm struggling with is how to pass the xml object from action() back to get_users(). What should action() look like in order to achieve this?
Turns out this was quite simple. The action method needs to be changed so it looks like this:
def action(action_title)
build = Nokogiri::XML::Builder.new do |xml|
xml.request do
xml.login do
xml.username("my_user")
xml.password("my_pass")
end
xml.action(action_title)
xml.params do
yield xml
end
end
end
do_stuff_to(build)
end
That meant the specific action method could be called like this to the same effect:
def get_users(city = "", gender = "")
action("getusers") do |xml|
xml.city(city) unless city.blank?
xml.gender(gender) unless gender.blank?
end
end
I'm sure this has to do with the intricacies mentionned in Shoes > Manual > Rules but I just don't get it. If someone would care to explain why #var == nil in the following code ...
I thought I could use visit to switch between different views in my application but that won't work if I lose all state.
class MyShoe < Shoes
url '/', :index
url '/somewhere', :somewhere
def index
#var = para link( "go somewhere", :click => "/somewhere" )
end
def somewhere
para "var = #{#var.inspect}"
end
end
Shoes.app
_why himself has answered this issue, and I'll get to that in a minute. First, the simplest way to pass data (specifically, strings) between different urls is like this:
class MyShoe < Shoes
url '/', :index
url '/somewhere/(\d+)', :somewhere
def index
#var = para link( "What is 2 + 2?", :click => "/somewhere/4" )
end
def somewhere(value)
para "2 + 2 = #{value}"
end
end
Shoes.app
It will match the subgroups of the regex and pass the matching strings as parameters to the method. Occasionally useful, but it gets unwieldy in a hurry. The other solution is to use constants or class variables, as _why explains here:
OK, so fooling around further it looks like all instance variables get
wiped at the beginning of every method within a Shoes subclass.
That's OK I guess. So what's the preferred way to have some data
that's shared from one Shoes URL to another? Passing it from one page
to the next in the URL itself works -- if it's a string. If it's not
a string, what should you use -- ##class_variables?
Sure you could use a class var. Those
are guaranteed to persist throughout
the life of the app. Or a constant.
Also, Shoes comes with SQLite3, data
can be passed there.
In your example, it would look like this:
class MyShoe < Shoes
url '/', :index
url '/somewhere', :somewhere
def index
##var = para link( "go somewhere", :click => "/somewhere" )
end
def somewhere
para "var = #{##var.inspect}"
end
end
Shoes.app