I will go ahead and apologize upfront as I am new to ruby and rails and I cannot for the life of me figure out how to implement using hashids in my project. The project is a simple image host. I have it already working using Base58 to encode the sql ID and then decode it in the controller. However I wanted to make the URLs more random hence switching to hashids.
I have placed the hashids.rb file in my lib directory from here: https://github.com/peterhellberg/hashids.rb
Now some of the confusion starts here. Do I need to initialize hashids on every page that uses hashids.encode and hashids.decode via
hashids = Hashids.new("mysalt")
I found this post (http://zogovic.com/post/75234760043/youtube-like-ids-for-your-activerecord-models) which leads me to believe I can put it into an initializer however after doing that I am still getting NameError (undefined local variable or method `hashids' for ImageManager:Class)
so in my ImageManager.rb class I have
require 'hashids'
class ImageManager
class << self
def save_image(imgpath, name)
mime = %x(/usr/bin/exiftool -MIMEType #{imgpath})[34..-1].rstrip
if mime.nil? || !VALID_MIME.include?(mime)
return { status: 'failure', message: "#{name} uses an invalid format." }
end
hash = Digest::MD5.file(imgpath).hexdigest
image = Image.find_by_imghash(hash)
if image.nil?
image = Image.new
image.mimetype = mime
image.imghash = hash
unless image.save!
return { status: 'failure', message: "Failed to save #{name}." }
end
unless File.directory?(Rails.root.join('uploads'))
Dir.mkdir(Rails.root.join('uploads'))
end
#File.open(Rails.root.join('uploads', "#{Base58.encode(image.id)}.png"), 'wb') { |f| f.write(File.open(imgpath, 'rb').read) }
File.open(Rails.root.join('uploads', "#{hashids.encode(image.id)}.png"), 'wb') { |f| f.write(File.open(imgpath, 'rb').read) }
end
link = ImageLink.new
link.image = image
link.save
#return { status: 'success', message: Base58.encode(link.id) }
return { status: 'success', message: hashids.encode(link.id) }
end
private
VALID_MIME = %w(image/png image/jpeg image/gif)
end
end
And in my controller I have:
require 'hashids'
class MainController < ApplicationController
MAX_FILE_SIZE = 10 * 1024 * 1024
MAX_CACHE_SIZE = 128 * 1024 * 1024
#links = Hash.new
#files = Hash.new
#tstamps = Hash.new
#sizes = Hash.new
#cache_size = 0
class << self
attr_accessor :links
attr_accessor :files
attr_accessor :tstamps
attr_accessor :sizes
attr_accessor :cache_size
attr_accessor :hashids
end
def index
end
def transparency
end
def image
##imglist = params[:id].split(',').map{ |id| ImageLink.find(Base58.decode(id)) }
#imglist = params[:id].split(',').map{ |id| ImageLink.find(hashids.decode(id)) }
end
def image_direct
#linkid = Base58.decode(params[:id])
linkid = hashids.decode(params[:id])
file =
if Rails.env.production?
puts "#{Base58.encode(ImageLink.find(linkid).image.id)}.png"
File.open(Rails.root.join('uploads', "#{Base58.encode(ImageLink.find(linkid).image.id)}.png"), 'rb') { |f| f.read }
else
puts "#{hashids.encode(ImageLink.find(linkid).image.id)}.png"
File.open(Rails.root.join('uploads', "#{hashids.encode(ImageLink.find(linkid).image.id)}.png"), 'rb') { |f| f.read }
end
send_data(file, type: ImageLink.find(linkid).image.mimetype, disposition: 'inline')
end
def upload
imgparam = params[:image]
if imgparam.is_a?(String)
name = File.basename(imgparam)
imgpath = save_to_tempfile(imgparam).path
else
name = imgparam.original_filename
imgpath = imgparam.tempfile.path
end
File.chmod(0666, imgpath)
%x(/usr/bin/exiftool -all= -overwrite_original #{imgpath})
logger.debug %x(which exiftool)
render json: ImageManager.save_image(imgpath, name)
end
private
def save_to_tempfile(url)
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = uri.scheme == 'https'
http.start do
resp = http.get(uri.path)
file = Tempfile.new('urlupload', Dir.tmpdir, :encoding => 'ascii-8bit')
file.write(resp.body)
file.flush
return file
end
end
end
Then in my image.html.erb view I have this:
<%
#imglist.each_with_index { |link, i|
id = hashids.encode(link.id)
ext = link.image.mimetype.split('/')[1]
if ext == 'jpeg'
ext = 'jpg'
end
puts id + '.' + ext
%>
Now if I add
hashids = Hashids.new("mysalt")
in ImageManager.rb main_controller.rb and in my image.html.erb I am getting this error:
ActionView::Template::Error (undefined method `id' for #<Array:0x000000062f69c0>)
So all in all implementing hashids.encode/decode is not as easy as implementing Base58.encode/decode and I am confused on how to get it working... Any help would be greatly appreciated.
I would suggest loading it as a gem by including it into your Gemfile and running bundle install. It will save you the hassle of requiring it in every file and allow you to manage updates using Bundler.
Yes, you do need to initialize it wherever it is going to be used with the same salt. Would suggest that you define the salt as a constant, perhaps in application.rb.
The link you provided injects hashids into ActiveRecord, which means it will not work anywhere else. I would not recommend the same approach as it will require a high level of familiarity with Rails.
You might want to spend some time understanding ActiveRecord and ActiveModel. Will save you a lot of reinventing the wheel. :)
Before everythink you should just to test if Hashlib is included in your project, you can run command rails c in your project folder and make just a small test :
>> my_id = ImageLink.last.id
>> puts Hashids.new(my_id)
If not working, add the gem in gemfile (that anyway make a lot more sence).
Then, I think you should add a getter for your hash_id in your ImageLink model.
Even you don't want to save your hash in the database, this hash have it's pllace in your model. See virtual property for more info.
Remember "Skinny Controller, Fat Model".
class ImageLink < ActiveRecord::Base
def hash_id()
# cache the hash
#hash_id ||= Hashids.new(id)
end
def extension()
# you could add the logic of extension here also.
ext = image.mimetype.split('/')[1]
if ext == 'jpeg'
'jpg'
else
ext
end
end
end
Change the return in your ImageManager#save_image
link = ImageLink.new
link.image = image
# Be sure your image have been saved (validation errors, etc.)
if link.save
{ status: 'success', message: link.hash_id }
else
{status: 'failure', message: link.errors.join(", ")}
end
In your template
<%
#imglist.each_with_index do |link, i|
puts link.hash_id + '.' + link.extension
end # <- I prefer the do..end to not forgot the ending parenthesis
%>
All this code is not tested...
I was looking for something similar where I can disguise the ids of my records. I came across act_as_hashids.
https://github.com/dtaniwaki/acts_as_hashids
This little gem integrates seamlessly. You can still find your records through the ids. Or with the hash. On nested records you can use the method with_hashids.
To get the hash you use to_param on the object itself which result in a string similar to this ePQgabdg.
Since I just implemented this I can't tell how useful this gem will be. So far I just had to adjust my code a little bit.
I also gave the records a virtual attribute hashid so I can access it easily.
attr_accessor :hashid
after_find :set_hashid
private
def set_hashid
self.hashid = self.to_param
end
Related
I'm building a small web scraper using Ruby and now I'm trying to refactor my code. Unfortunately, I'm encountering some errors while I'm refactoring my code. This is one of the errors.
Basically, I'm calling two separate methods in the first method which is whey_scrapper. Each of these two methods are basically responsible of scraping a specific item on the webpage. When I run and debug this code with byebug, I basically try to display the products or prices I've scraped but I get an error message saying that 'products' or 'prices' is undefined. This is my current code:
require 'open-uri'
require 'nokogiri'
require 'httparty'
require 'byebug'
require 'csv'
class Wheyscrapper
def whey_scrapper
company = 'Body+%26+fit'
url = "https://www.bodyenfitshop.nl/afslanken/afslank-toppers/?manufacturer=#{company}"
unparsed_page = open(url).read
parsed_page = Nokogiri::HTML(unparsed_page)
product_scrapper
prices_scrapper
# csv = CSV.open('wheyprotein.csv', 'wb')
end
def product_scrapper
products = Array.new
product_names = parsed_page.css('div.product-primary')
product_names.each do |product_name|
product = {
name: product_name.css('h2.product-name').text
}
products << product
end
end
def prices_scrapper
prices = Array.new
product_prices = parsed_page.css('div.price-box')
product_prices.each do |product_price|
price = {
amount: product_price.css('span.price').text
}
prices << price
end
end
byebug
whey_scrapper
end
There's a lot going on here, but to make it more Ruby you'd consider making those lazy-initialized and giving them names that reflect that:
class Wheyscrapper
URL = "https://www.bodyenfitshop.nl/afslanken/afslank-toppers/?%s"
def initialize(company:)
#company = company
# Use encode_www_form to encode query-string parameters
#url = URL % URI.encode_www_form(manufacturer: company)
end
def document
# Lazy-initialize a parsd version of the page
#document ||= Nokogiri::HTML(open(url).read)
end
def products
document.css('div.product-primary').map do |product_name|
{
name: product_name.css('h2.product-name').text
}
end
end
def prices
document.css('div.price-box').map do |product_price|
{
amount: product_price.css('span.price').text
}
end
end
end
This fixes a lot of the data propagation problems you had in your original. When you declare a variable it's a local variable, meaning it doesn't exist outside of that particular call of that particular method. If you want to persist it for longer you need to use instance variables, as in #products, or you need to define methods that return the data you need.
The above approach combines that, using a lazy-initialized instance variable to persist the parsed document, and exposes that as a method the other methods can use.
Now you can spin this up:
scraper = WheyScraper.new(company: "Body & Fit")
Where that should enable everything to be available directly:
scraper.prices
scraper.products
When you learn how to use Ruby effectively you'll often find solutions to your problems that are really minimal. Usually a lot of Ruby code is a sign that it's not being used properly.
This should be refactored in a better way but this should at least work without refactor, based on my comments above
require 'open-uri'
require 'nokogiri'
require 'httparty'
require 'csv'
class Wheyscrapper
def whey_scrapper
company = 'Body+%26+fit'
url = "https://www.bodyenfitshop.nl/afslanken/afslank-toppers/?manufacturer=#{company}"
unparsed_page = open(url).read
#parsed_page = Nokogiri::HTML(unparsed_page)
product_scrapper
prices_scrapper
# csv = CSV.open('wheyprotein.csv', 'wb')
end
def product_scrapper
#products = Array.new
product_names = #parsed_page.css('div.product-primary')
product_names.each do |product_name|
product = {
name: product_name.css('h2.product-name').text
}
#products << product
end
end
def prices_scrapper
#prices = Array.new
#product_prices = #parsed_page.css('div.price-box')
#product_prices.each do |product_price|
price = {
amount: product_price.css('span.price').text
}
#prices << price
end
end
end
w = Wheyscrapper.new.whey_scrapper
I am new to ruby and this site so please bear with me! I have googled endlessly to no fruition.
I am trying to pass in a college object to my class method scrape_college_info that I created in the previous class method scrape_illinois_index_page, so that I may scrape the next level of information for the specific college the user selects using Pry and Nokogiri. Unfortunately, I keep getting an argument error.
I know it isn't the prettiest, but this is my code right now:
class College
attr_accessor :name, :location, :size, :type, :url
BASE_PATH = "https://www.collegesimply.com/colleges/illinois/"
def self.college
self.scrape_colleges
end
def self.scrape_colleges
colleges = self.scrape_illinois_index_page
colleges
end
def self.scrape_illinois_index_page
doc = Nokogiri::HTML(open(BASE_PATH))
# binding.pry
colleges = []
doc.xpath("//tr").each do |doc|
college = self.new
if doc.css("td")[0] != nil
college.name = doc.css("td")[0].text.strip
end
if doc.css("td")[1] != nil
college.location = doc.css("td")[1].text.strip
end
if doc.css('table.table tbody tr td:nth-child(1) a')[0] != nil
college.link = doc.css('table.table tbody tr td:nth-child(1) a')[0]['href']
end
colleges << college
end
colleges
end
def self.scrape_college_info(college)
doc = Nokogiri::HTML(open(BASE_PATH + "#{college.link}"))
end
end
Try below code to get college.link.
if doc.css("td")[0] != nil
college.name = doc.css("td")[0].text.strip
college.link = doc.css("td")[0].css("a").map{|a| a['href']}[0]
end
Now you can pass college link like :
def self.scrape_college_info(college)
doc = Nokogiri::HTML(open("https://www.collegesimply.com" + "#{college.link}"))
end
Hope this will solve your problem. Please let me know, if it works for you.
Try using URI.join:
new_url = URI.join(BASE_PATH, college.link).to_s
I am trying to create a database using mongoid but it fails to find the create method. I am trying to create 2 databases based on csv files:
extract_data class:
class ExtractData
include Mongoid::Document
include Mongoid::Timestamps
def self.create_all_databases
#cbsa2msa = DbForCsv.import!('./share/private/csv/cbsa_to_msa.csv')
#zip2cbsa = DbForCsv.import!('./share/private/csv/zip_to_cbsa.csv')
end
def self.show_all_database
ap #cbsa2msa.all.to_a
ap #zip2cbsa.all.to_a
end
end
the class DbForCSV works as below:
class DbForCsv
include Mongoid::Document
include Mongoid::Timestamps
include Mongoid::Attributes::Dynamic
def self.import!(file_path)
columns = []
instances = []
CSV.foreach(file_path, encoding: 'iso-8859-1:UTF-8') do |row|
if columns.empty?
# We dont want attributes with whitespaces
columns = row.collect { |c| c.downcase.gsub(' ', '_') }
next
end
instances << create!(build_attributes(row, columns))
end
instances
end
private
def self.build_attributes(row, columns)
attrs = {}
columns.each_with_index do |column, index|
attrs[column] = row[index]
end
ap attrs
attrs
end
end
I am not aware of all fields and it may change in time. that's why I have create database and generic mehtods.
I have also another issue after having fixed the 'create!' issue.
I am using the encoding to make sure only UTF8 char are handled but I still see:
{
"zip" => "71964",
"cbsa" => "31680",
"res_ratio" => "0.086511098",
"bus_ratio" => "0.012048193",
"oth_ratio" => "0.000000000",
"tot_ratio" => "0.082435345"
}
when doing 'ap attrs' in the code. how to make sure that 'zip' -> 'zip'
Thanks
create! is a class method but you're trying to call it as an instance method. Your import! method shouldn't be an instance method either, it should be a class method since it produces instances of your class:
def self.import!(file_path)
#-^^^^
# everything else would be the same...
end
You'd also make build_attributes a class method since it is just a helper method for another class method:
def self.build_attributes
#...
end
And then you don't need that odd looking new call when using import!:
def self.create_all_databases
#cbsa2msa = DbForCsv.import!('./share/private/csv/cbsa_to_msa.csv')
#zip2cbsa = DbForCsv.import!('./share/private/csv/zip_to_cbsa.csv')
end
I'm trying to make a simple rss generator. My initialize method works fine and the update method runs without an error too but the new item in the update method never get added to the rss feed. I think it has something to do with how i'm accessing the variable 'maker' but i'm not sure.
require "rss"
class RSS_Engine
def initialize
#rss = RSS::Maker.make("atom") do |maker|
maker.channel.author = "Jamie"
maker.channel.updated = Time.now.to_s
maker.channel.about = "http://www.ruby-lang.org/en/feeds/news.rss"
maker.channel.title = "Example Feed"
#maker = maker
end
end
def update
#maker.items.new_item do |item|
item.title = "Test"
item.updated = Time.now.to_s
end
end
def print_rss
puts #rss
end
end
rss = RSS_Engine.new
rss.update
rss.print_rss
I got the original code from this example:
rss = RSS::Maker.make("atom") do |maker|
maker.channel.author = "matz"
maker.channel.updated = Time.now.to_s
maker.channel.about = "http://www.ruby-lang.org/en/feeds/news.rss"
maker.channel.title = "Example Feed"
maker.items.new_item do |item|
item.link = "http://www.ruby-lang.org/en/news/2010/12/25/ruby-1-9-2-p136-is-released/"
item.title = "Ruby 1.9.2-p136 is released"
item.updated = Time.now.to_s
end
This code works fine but i want to be able to add new posts to the rss feed over time so i'm trying to put the 'new.item' bit into it's own method.
The problem is not #maker variable, you have to invoke to_feed method to regenerate the feed after you modify it out of the code block.
So you need to add #rss = #maker.to_feed at the end of your update method.
One more thing about creating a new feed entry, link or id attribute need to be set.
Below code will work for you:
def update
#maker.items.new_item do |item|
item.link = "http://test.com"
item.title = "Test"
item.updated = Time.now.to_s
end
#rss = #maker.to_feed
end
If you are interested about why, you can take a look ruby rss source code. And below code(under rss/maker/base.rb) is the root cause why you need to invoke to_feed method if you modify feed out of the block:
def make(*args, &block)
new(*args).make(&block)
end
def make
yield(self)
to_feed
end
For some reason in my current controller I am getting ActiveModel::ForbiddenAttributesError even though I believe I am using strong parameters just fine. Albeit I am using permit! for the time being to permit all model attributes. See code below, what am I missing
class HeuristicsController < ApplicationController
def index
#heuristics = Heuristic.order(:name).page params[:page]
#heuristic = Heuristic.new
end
def create
#heuristic = Heuristic.new(params[:heuristic])
if #heuristic.save
redirect_to action: 'index', :flash => {:success => "New heuristic created!" }
else
render 'new'
end
end
def new
#title = "Heuristic"
#heuristic = Heuristic.new
end
private
def heuristic_params
params.require(:heuristic).permit!
end
end
i think you did not fully understand the way that strong-params work...
you have a method
def heuristic_params
params.require(:heuristic).permit!
end
and you are not using it
Heuristic.new(params[:heuristic])