I am writing a program that uses Mechanize to scrape a student's grades and classes from edline.net using a student account and return the data I need. However, after logging in, from the homepage I have to access a link (called 'Private Reports') which will then dynamically return a page of links to each of the student's classes and respective grades.
When testing I create a new object my_account that has several instance variables including the homepage. I pointed new variables to the instance variables for this to be more simple to read):
result_page = agent.page.link_with(:text => 'Private Reports').click
I get:
Mechanize::UnsupportedSchemeError
But if I were to replace :click with :text it responds correctly and result_page will equal the link's text "Private Reports"
Why does it respond to :text correctly but give an error for :click? Is there a way to get around this or should I rethink my solution to this problem?
Here's the code:
class AccountFetcher
EDLINE_LOGIN_URL = 'http://edline.net/Index.page'
def self.fetch_form(agent)
# uses #agent of an Account object
page = agent.get(EDLINE_LOGIN_URL)
form = page.form('authenticationEntryForm')
end
# edline's login form has to have several pre-conditions met before submitting the form
def self.initialize_form(agent)
form = AccountFetcher.fetch_form(agent)
submit_event = form.field_with(:name => 'submitEvent')
enter_clicked = form.field_with(:name => 'enterClicked')
ajax_support = form.field_with(:name => 'ajaxSupported')
ajax_support.value = 'yes'
enter_clicked.value = true
submit_event.value = 1
return form
end
# logs the user in and returns the homepage
def self.fetch_homepage(u_username, u_password, agent)
form = AccountFetcher.initialize_form(agent)
username = form.field_with(:name => 'screenName')
password = form.field_with(:name => 'kclq')
username.value = u_username
password.value = u_password
form.submit
end
end
# class Account will be expanded later on but here are the bare bones for a user to log in to their account
class Account
attr_accessor :report, :agent, :username, :password
def initialize(u_username, u_password)
#agent = Mechanize.new
#username = u_username
#password = u_password
end
def login
page = AccountFetcher.fetch_homepage(self.username, self.password, self.agent)
#report = page
end
end
my_account = Account.new('ex_username', 'ex_password')
my_account.login
page = my_account.report
agent = my_account.agent
page = agent.page.link_with(:text => 'Private Reports').click
Where does the link point to? Ie, what's the href?
I ask because "scheme" usually refers to something like http or https or ftp, so maybe the link has a weird scheme that mechanize doesn't know how to handle, hence Mechanize::UnsupportedSchemeError
Related
So i have a bit of a pickle.
I have an encrypted data bag to store LDAP passwords. In my node run list, one of my recipes installs the secret key onto my client machine.
In my problematic cookbook, i have a helper (in /libraries) that pulls data from AD (using LDAP). Problem is, i can't find a way to delay the assignment of my node attribute after initial compile phase.
Take this line of code as example :
node.override['yp_chefserver']['osAdminUser'] = node['yp_chefserver']['osAdminUser'] + get_sam("#{data_bag_item('yp_chefserver', 'ldap', IO.read('/etc/chef/secret/yp_chefserver'))['ldap_password']}")
Im trying to override an attribute by adding an array returned by my helper function "get_sam" which returns an array, but it needs to run AFTER the compile phase since the file "/etc/chef/secret/yp_chefserver" doesnt exist before the convergence of my runlist.
So my question : Is there a way to assign node attributes via data_bag_items during the execution phase?
Some things i've tried :
ruby_block 'attribution' do
only_if { File.exist?('/etc/chef/secret/yp_chefserver')}
block do
node.override['yp_chefserver']['osAdminUser'] = node['yp_chefserver']['osAdminUser'] + get_sam("#{data_bag_item('yp_chefserver', 'ldap', IO.read('/etc/chef/secret/yp_chefserver'))['ldap_password']}")
Chef::Log.warn("content of osAdminUser : #{node['yp_chefserver']['osAdminUser']}")
end
end
This doesn't work because the custom resource ruby_block doesn't have the method "data_bag_item". I've tried using lazy attributes in my "chef_server" custom resource, but same problem.
I also tried having the attribution done directly in my helper module, but since the helper module compiles before the exec phase, the file doesn't exist when it assigns the variable.
Here is the helper function in question should anyone wonder, it pulls the SamAccountName from LDAP to assign admin users to my chef server. :
module YpChefserver
module LDAP
require 'net-ldap'
#ldap
def get_ldap(ldap_password)
if #ldap.nil?
#ldap = Net::LDAP.new :host => "ADSERVER",
:port => 389,
:auth => {
:method => :simple,
:username => "CN=USERNAME,OU=East Service Accounts,OU=System Accounts,DC=ad,DC=ypg,DC=com",
:password => "#{ldap_password}"
}
end
#ldap
end
def get_ldap_users(ldap_password)
filter = Net::LDAP::Filter.eq("cn", "DevOps")
treebase = "dc=ad, dc=ypg, dc=com"
get_ldap(ldap_password).search(:base => treebase, :filter => filter) do |entry|
#puts "DN: #{entry.dn}"
entry.each do |attribute, values|
return values if attribute == :member
end
end
end
def get_sam(ldap_password)
samacc = Array.new
get_ldap_users(ldap_password).entries.each{ |elem|
y = elem.to_s.split(/[,=]/)
filter = Net::LDAP::Filter.eq("cn", y[1])
treebase = "OU=Support Users and Groups,OU=CGI Support,DC=ad,DC=ypg,DC=com"
get_ldap(ldap_password).search(:base => treebase, :filter => filter, :attributes => "SamAccountName") do |entry|
samacc << entry.samaccountname
end
}
return samacc
end
end
end
Turns out you can actually call it inside a ruby block, just by using the actual Chef call instead of the resource name, as follow :
ruby_block 'attributes' do
only_if {File.exist?('/etc/chef/secret/yp_chefserver')}
block do
dtbg = Chef::EncryptedDataBagItem.load('yp_chefserver','ldap',"IO.read('/etc/chef/secret/yp_chefserver')")
end
end
Leaving this here for those who might need it
EDIT :
Here is final function using the code mentionned above to pull accounts from AD, using encrypted data bags to provide the password and to then pass those results to my node attributes, all during the execution phase :
ruby_block 'attributes' do
extend YpChefserver::LDAP
only_if {File.exist?('/etc/chef/secret/yp_chefserver')}
block do
# Chef::Config[:encrypted_data_bag_secret] = '/etc/chef/secret/yp_chefserver'
dtbg = Chef::EncryptedDataBagItem.load('yp_chefserver','ldap')
node.override['yp_chefserver']['ldap_pw'] = dtbg['ldap_password']
userarray = Array.new
userarray.push("#{node['yp_chefserver']['osAdminUser']}")
get_sam("#{node['yp_chefserver']['ldap_pw']}").each { |i| userarray.push(i[0]) }
node.override['yp_chefserver']['authorized_users'] = userarray
node.override['yp_chefserver']['local_admin_pw'] = dtbg['local_admin_pw']
end
end
I'm trying to get to google play developer console using ruby. But first I have to login. I'm trying like this:
def try_post(url, body = {}, headers = {})
unless #agent #This just creates a new mechanize instance
setup
end
puts 'Logging in'
# Hardcoded for testing purposes
#agent.get 'https://accounts.google.com/ServiceLogin?service=androiddeveloper&passive=1209600&continue=https://play.google.com/apps/publish/%23&followup=https://play.google.com/apps/publish/#identifier'
form = #agent.page.forms.find {|f| f.form_node['id'] == "gaia_loginform"}
unless form
raise 'No login form'
end
form.field_with(:id => "Email").value = #config.email
form.click_button
form = #agent.page.forms.find {|f| f.form_node['id'] == "gaia_loginform"}
unless form
raise 'No login form'
end
form.field_with(:name => "Passwd").value = #config.password
form.click_button
if #agent.page.uri.host != "play.google.com"
STDERR.puts "login failed? : uri = " + #agent.page.uri.to_s
raise 'Google login failed'
end
# #agent.post(url, body)
end
However this fails spectacularly. I tried a few other ways (trying to populate Passwd-hidden, finding field by id and so on) but no luck. I think that the password does not get entered since when I try to puts #agent.page.body after the final click_button I see enter password text somewhere in HTML.
What am I doing wrong and how can I fix it?
I've been digging around a bit more and found out that it's not that simple and I could not login with mechanize in any way.
So I ended up with using watir which was fairly simple and straightforward. Here's an example:
browser.goto LOGIN_URL
browser.text_field(:id, 'Email').set #config.email
browser.button(:id, 'next').click
browser.text_field(:id, 'Passwd').wait_until_present
browser.text_field(:id, 'Passwd').set #config.password
browser.button(:id, 'signIn').click
# Here I wait until an element on my target page is visible and then continue
browser.link(:href, '#SOMETHING').wait_until_present
Hope it helps.
I am trying to implement a Ruby script that will take in a username and password, then proceed to fill in the account details on a login form on another website and return the then follow a link and retrieve the account history. To do this I am using the Mechanize gem.
I have been following the examples here
but still I cant seem to get it to work. I have simplified this down greatly to try get it to work in parts but a supposedly simple filling in a form is holding me up.
Here is my code:
# script gets called with a username and password for the site
require 'mechanize'
#create a mechanize instant
agent = Mechanize.new
agent.get('https://mysite/Login.aspx') do |login_page|
#fill in the login form on the login page
loggedin_page = login_page.form_with(:id => 'form1') do |form|
username_field = form.field_with(:id => 'ContentPlaceHolder1_UserName')
username_field.value = ARGV[0]
password_field = form.field_with(:id => 'ContentPlaceHolder1_Password')
password_field.value = ARGV[1]
button = form.button_with(:id => 'ContentPlaceHolder1_btnlogin')
end.submit(form , button)
#click the View my history link
#account_history_page = loggedin_page.click(home_page.link_with(:text => "View My History"))
####TEST to see if i am actually making it past the login page
#### and that the View My History link is now visible amongst the other links on the page
loggedin_page.links.each do |link|
text = link.text.strip
next unless text.length > 0
puts text if text == "View My History"
end
##TEST
end
Terminal error message:
stackqv2.rb:19:in `block in <main>': undefined local variable or method `form' for main:Object (NameError)
from /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize.rb:409:in `get'
from stackqv2.rb:8:in `<main>'
You don't need to pass form as an argument to submit. The button is also optional. Try using the following:
loggedin_page = login_page.form_with(:id => 'form1') do |form|
username_field = form.field_with(:id => 'ContentPlaceHolder1_UserName')
username_field.value = ARGV[0]
password_field = form.field_with(:id => 'ContentPlaceHolder1_Password')
password_field.value = ARGV[1]
end.submit
If you really do need to specify which button is used to submit the form, try this:
form = login_page.form_with(:id => 'form1')
username_field = form.field_with(:id => 'ContentPlaceHolder1_UserName')
username_field.value = ARGV[0]
password_field = form.field_with(:id => 'ContentPlaceHolder1_Password')
password_field.value = ARGV[1]
button = form.button_with(:id => 'ContentPlaceHolder1_btnlogin')
loggedin_page = form.submit(button)
It's a matter of scope:
page.form do |form|
# this block has its own scope
form['foo'] = 'bar' # <- ok, form is defined inside this block
end
puts form # <- error, form is not defined here
ramblex's suggestion is to not use a block with your form and I agree, it's less confusing that way.
Need to post to existing web page (no login required) and post parameters for submit where multiple submit forms tags exist and contains identical tags with the same NAME and VALUE tags; for example, on the same page this INPUT Submit is repeated 3 times under different FORM tags:
< INPUT TYPE='Submit' NAME='submit_button' VALUE='Submit Query' >
My Ruby code runs ok for identifying the fields on the form tags, but fails on the page.forms[x].action post with 405 HTTPMethodNotAllowed for https://pdb.nipr.com/html/PacNpnSearch -- unhandled response.
Ruby code:
class PostNIPR2
def post(url)
button_count = 0
agent = Mechanize.new
page = agent.get(url)
page.forms.each do |form|
form.buttons.each do |button|
if(button.value == 'Submit Query')
button_count = button_count + 1
if (button_count == 3)
btn_submit_license = button.name
puts button
puts btn_submit_license
puts button.value
end
end
end
end
begin
uform = page.forms[1]
uform.license = "0H20649"
uform.state = "CA"
uform.action = 'https://pdb.nipr.com/html/PacNpnSearch'
rescue Exception => e
error_page = e.page
end
page = agent.submit(uform)
end
url = "https://pdb.nipr.com/html/PacNpnSearch.html"
p = PostNIPR2.new
p.post(url)
end
Is your question how to select that button? If so:
form.button_with(:name => 'submit_button')
or submit the form like this:
next_page = form.submit form.button_with(:name => 'submit_button')
Also you are changing the form's action for some reason and that will explain the 405s
You are correct, sorry about the comment code - the question was to have the form.license and form.state updated with the input params then have the form.submit post the form.button_with(:name => 'Submit Query' - I did this and received the 405 HTTPMethodNotAllowed, while for https://pdb.nipr.com/html/PacNpnSearch -- unhandled response. But now I have changed the code to agent.page.form_with(:name => 'license_form') which now correctly finds the form I need to post to; then I get the form.button_with(:value => 'Submit Query') and then utilize the agent.submit(form, button). Now I get the correct result.
I want login to vk.com or m.vk.com without Ruby. But my code dosen't work.
require 'net/http'
email = "qweqweqwe#gmail.com"
pass = "qeqqweqwe"
userUri = URI('m.vk.com/index.html')
Net::HTTP.get(userUri)
res = Net::HTTP.post_form(userUri, 'email' => email, 'pass' => pass)
puts res.body
First of all, you need to change userUri to the following:
userUri = URI('https://login.vk.com/?act=login')
Which is where the vk site expects your login parameters.
I'm not very faimilar with vk, but you probably need a way to handle the session cookie. Both receiving it, and providing it for future requests. Can you elaborate on what you're doing after login?
Here is the net/http info for cookie handling:
# Headers
res['Set-Cookie'] # => String
res.get_fields('set-cookie') # => Array
res.to_hash['set-cookie'] # => Array
puts "Headers: #{res.to_hash.inspect}"
This kind of task is exactly what Mechanize is for. Mechanize handles redirects and cookies automatically. You can do something like this:
require 'mechanize'
agent = Mechanize.new
url = "http://m.vk.com/login/"
page = agent.get(url)
form = page.forms[0]
form['email'] = "qweqweqwe#gmail.com"
form['pass'] = "qeqqweqwe"
form.submit
puts agent.page.body