Why is Mechanize returning "undefined method 'value=' for nil:NilClass" when trying to set a password? - ruby

I wrote a script with Mechanize to scrape some links, which later I will write code to put into an Excel file.
For now I can't authenticate past the first page. I keep getting an undefined method value= for nil:NilClass when attempting to set the password in the form and haven't been able to find any information on it.
I don't even have the method value= in my code so I don't understand what is going on. The code runs fine for the username, but once I enter the password and hit enter I get the error:
users.rb:11:in `block (2 levels) in <main>': undefined method `value=' for nil:NilClass (NoMethodError)
from (eval):23:in `form_with'
from formity_users.rb:7:in `block in <main>'
from /home/codelitt/.rvm/gems/ruby-2.0.0-p247/gems/mechanize-2.7.1/lib/mechanize.rb:433:in `get'
from formity_users.rb:5:in `<main>'
This is my users.rb script:
require 'rubygems'
require 'mechanize'
a = Mechanize.new
a.get('https://www.example.com') do |page|
#Enter information into forms
logged_in = page.form_with(:id => 'frmLogin') do |f|
puts "Username?"
f.field_with(:name => "LoginCommand.EmailAddress").value = gets.chomp
puts "Password?"
f.field_with(:name => "Login.Password").value = gets.chomp
end.click_button
#Click drop down
admin_page = logged_in.click.link_with(:text => /Admin/)
#Click Users and enter user admin section
user_admin = admin_page.click.link_with(:text => /Users/)
#Scrape and print links for now
user_admin.links.each do |link|
text = link.text.strip
next unless text.length > 0
puts text
end
end

I think your error is coming from
f.field_with(:name => "Login.Password")
which seems to be nil. For username, I see that you have specified input name LoginCommand.EmailAddress and for password input name is Login.Password.
I'd expect anybody who has written this markup to use consistent names. Maybe you should look that the underlying html to see you're using correct field names in your code.

Related

Ruby ArgumentError when actually providing correct arguments

Ruby complains i am not providing enough arguments to my script, which is:
#!/usr/bin/ruby
require 'mail'
def send(file,recipients_csv)
recipients=recipients_csv.split(",")
recipients.each do |recipient|
Mail.defaults do
delivery_method :smtp,{ :address => 'localhost', :port => 25,:openssl_verify_mode => OpenSSL::SSL::VERIFY_NONE}
end
mail = Mail.new do
from 'noreply#mydomain.com'
to "#{recipient}"
subject "test"
body "test"
add_file :filename => "a_testfile.tsv", :content => File.read(file.path)
end
mail.deliver!
end
end
testfile=File.new("newfile.tsv","w")
send(testfile,"name#mydomain.com")
What i get back is:
Mailer.rb:4:in `send': wrong number of arguments (1 for 2) (ArgumentError)
from /usr/lib64/ruby/gems/1.9.1/gems/treetop-1.4.15/lib/treetop/runtime/compiled_parser.rb:18:in `parse'
from /usr/lib64/ruby/gems/1.9.1/gems/mail-2.5.4/lib/mail/elements/address_list.rb:26:in `initialize'
from /usr/lib64/ruby/gems/1.9.1/gems/mail-2.5.4/lib/mail/fields/common/common_address.rb:9:in `new'
I dont get this, the arguments i provide are obviously 2
This might be conflicting with the Ruby base send method. Try renaming send to send_mail (or something), to avoid overwriting the send method
This error isn't coming from when you are running the script yourself on line 22, you are clearly passing it two arguments. It is actually coming from one of the three files you see in your error stack.
from /usr/lib64/ruby/gems/1.9.1/gems/treetop-1.4.15/lib/treetop/runtime/compiled_parser.rb:18:in `parse'
from /usr/lib64/ruby/gems/1.9.1/gems/mail-2.5.4/lib/mail/elements/address_list.rb:26:in `initialize'
from /usr/lib64/ruby/gems/1.9.1/gems/mail-2.5.4/lib/mail/fields/common/common_address.rb:9:in `new'
If you go into those files send is being called with only one argument as opposed to two.

How do I use Mechanize to go through each link?

Am trying to go through a series of links with a css class title and click those links and then get the product title. But i keep getting the error undefined method each for #<Mechanize::Page::Link:0x007fbfe2524410> (NoMethodError)? I Don't understand what am doning wrong?
heres my code:
require 'mechanize'
file = File.new("outputscrape.txt", 'w')
agent = Mechanize.new { |agent|
agent.user_agent_alias = 'Windows Chrome'}
page = agent.get('http://www.amazon.com/s/ref=sr_nr_n_0?rh=n%3A283155%2Cn%3A%211000%2Cn%3A5%2Cn%3A15377001%2Cn%3A6133979011%2Cn%3A6133980011&bbn=6133979011&ie=UTF8&qid=1412193262&rnid=6133979011')
title_link = page.link_with(:dom_class => "title")
title_link.each do |link|
link.click
file.write(link.at('#productTitle').text.strip)
end
From the mechanize docs:
link_with(criteria)
Find a single link matching criteria.
You need to use:
links_with(criteria)
Find all links matching criteria.
The object mentioned in your error message, Page::Link:
undefined method each for #<Mechanize::Page::Link:0x007fbfe2524410>
(NoMethodError)
doesn't sound like more than one thing, does it? More than one thing would be more like Page::Links, or Page::Link::Group, or Page::LinkSet. You are doing the equivalent of:
10.each do |number|
puts number
end
However, numbers do not have an each() method, so that produces the error:
undefined method `each' for 10:Fixnum (NoMethodError)
Compare that to your error:
undefined method each for #<Mechanize::Page::Link:0x007fbfe2524410>
On the other hand an Array does have an each() method, so you can do this:
[10, 20, 30].each do |number|
puts number
end

Login form with Mechanize?

I've been trying to write a webscraper in Ruby to scrape from a corporate events data website, and I'm referring to the Flickr example on the Mechanize docs page:
When I run corp_act_scrape.rb:
require 'rubygems'
require 'mechanize'
agent = Mechanize.new
home_page = agent.get("http://www.eventsdata.com/main.php#")
mypage = home_page.form_with(:name => 'loginForm') do |form|
form.myusrname = ARGV[0]
form.mypasswrd = ARGV[1]
end.submit
rows = page.css('#recentEventsDiv > div.RecentEventsDisplay > table > tbody > tr')
nextLink = page.link_with(:text => 'Next')
hasNextLink = nextLink?
while page.hasNextLink do
puts rows
page = agent.click(page.nextLink)
end
I receive the error:
corp_act_scrape.rb:9:in `block in <main>': undefined method `myusrname=' for nil:
NilClass (NoMethodError)
from (eval):23:in `form_with'
from corp_act_scrape.rb:7:in `<main>'
Copying the Flickr example, it seems that I should be able to enter my username and password as methods, but it doesn't seem to work in practice. Also, that section of the code is pretty confusing to me. If you have an alternative method of submitting the form, please also let me know.

Proper way of reading a page and saving it into a html file?

I have the following:
require 'rubygems'
require 'anemone'
require 'nokogiri'
require 'open-uri'
Anemone.crawl("http://www.findbrowsenodes.com/", :delay => 3) do |anemone|
anemone.on_pages_like(/http:\/\/www.findbrowsenodes.com\/us\/.+\/[\d]*/) do | page |
doc = Nokogiri::HTML(open(page.url))
id = doc.at_css("#n_info #clipnode").text unless doc.at_css("#n_info #clipnode").nil?
File.open("#{node_id}.html", "wb") do |f|
f.write(open(page).read)
end
end
end
So I'm trying to save each URL as a html file with this:
File.open("#{id}.html", "wb") do |f|
f.write(open(page).read)
end
But I get this error:
alex#alex-K43U:~/rails/anemone$ ruby anemone.rb
/home/alex/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/open-uri.rb:35:in
open': can't convert Anemone::Page into String (TypeError) from
/home/alex/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/open-uri.rb:35:in
open' from anemone.rb:27:in block (3 levels) in <main>' from
anemone.rb:26:inopen' from anemone.rb:26:in `block (2 levels) in
'
What's the right way of doing this?
There are several problems / confusions:
As the error says, the open methods expects a String (i.e. the url), but you're providing an Anemone::Page object.
This object has a url method, which you already use on line 9.
On line 9: open(page.url)
You're already opening the page, so you could reuse that. But:
According to the docs http://anemone.rubyforge.org/doc/classes/Anemone/Page.html Anemone::Page contains a body method that may already contain the content (I'm just guessing, haven't use or tried that library). If that's the case, there's no need to use open.
As I see it, the following untested code may be more like what you're looking for:
doc = Nokogiri::HTML(page.body)
# [snip]
File.open("#{node_id}.html", "wb") do |f|
f.write(page.body)
end

Trying to get list of friends from twitter using httparty(ruby)

I am trying to get the list of specific user`s friends from twitter.
This is my code -
require 'rubygems'
require 'httparty'
class TwitterData
include HTTParty
base_uri 'http://api.twitter.com/1/'
default_params :output => 'json'
format :json
def self.get_username_data(username)
get('statuses/friends.json' , :query => { :screen_name => username })
end
end
puts "Please your twitter username - "
twitter_username = gets
puts TwitterData.get_username_data(twitter_username).inspect
This is the error I am getting -
Please your twitter username -
twitter
C:/Ruby192/lib/ruby/gems/1.9.1/gems/crack-0.1.8/lib/crack/json.rb:14:in `rescue in parse': Invalid JSON string (Crack::ParseError)
from C:/Ruby192/lib/ruby/gems/1.9.1/gems/crack-0.1.8/lib/crack/json.rb:12:in `parse'
from C:/Ruby192/lib/ruby/gems/1.9.1/gems/httparty-0.6.1/lib/httparty/parser.rb:116:in `json'
from C:/Ruby192/lib/ruby/gems/1.9.1/gems/httparty-0.6.1/lib/httparty/parser.rb:136:in `parse_supported_format'
from C:/Ruby192/lib/ruby/gems/1.9.1/gems/httparty-0.6.1/lib/httparty/parser.rb:103:in `parse'
from C:/Ruby192/lib/ruby/gems/1.9.1/gems/httparty-0.6.1/lib/httparty/parser.rb:66:in `call'
from C:/Ruby192/lib/ruby/gems/1.9.1/gems/httparty-0.6.1/lib/httparty/request.rb:180:in `parse_response'
from C:/Ruby192/lib/ruby/gems/1.9.1/gems/httparty-0.6.1/lib/httparty/request.rb:164:in `handle_response'
from C:/Ruby192/lib/ruby/gems/1.9.1/gems/httparty-0.6.1/lib/httparty/request.rb:57:in `perform'
from C:/Ruby192/lib/ruby/gems/1.9.1/gems/httparty-0.6.1/lib/httparty.rb:280:in `perform_request'
from C:/Ruby192/lib/ruby/gems/1.9.1/gems/httparty-0.6.1/lib/httparty.rb:232:in `get'
from twitter_friends_2.rb:11:in `get_username_data'
from twitter_friends_2.rb:17:in `<main>'
Replace your method with this
def self.get_username_data(username)
get("/statuses/friends.json?screen_name=#{username}")
end
OR
def self.get_username_data(username)
get('/statuses/friends.json' , :query => { :screen_name => username.strip })
end
You need to strip the username because from command line when user enters the twitter username and hits enter key a "\n" gets appended to the username and the same username is sent as a parameter which causes the problem. Any of the above given code snippets should work.

Resources