Google login with mechanize on ruby - ruby

I'm trying to get to google play developer console using ruby. But first I have to login. I'm trying like this:
def try_post(url, body = {}, headers = {})
unless #agent #This just creates a new mechanize instance
setup
end
puts 'Logging in'
# Hardcoded for testing purposes
#agent.get 'https://accounts.google.com/ServiceLogin?service=androiddeveloper&passive=1209600&continue=https://play.google.com/apps/publish/%23&followup=https://play.google.com/apps/publish/#identifier'
form = #agent.page.forms.find {|f| f.form_node['id'] == "gaia_loginform"}
unless form
raise 'No login form'
end
form.field_with(:id => "Email").value = #config.email
form.click_button
form = #agent.page.forms.find {|f| f.form_node['id'] == "gaia_loginform"}
unless form
raise 'No login form'
end
form.field_with(:name => "Passwd").value = #config.password
form.click_button
if #agent.page.uri.host != "play.google.com"
STDERR.puts "login failed? : uri = " + #agent.page.uri.to_s
raise 'Google login failed'
end
# #agent.post(url, body)
end
However this fails spectacularly. I tried a few other ways (trying to populate Passwd-hidden, finding field by id and so on) but no luck. I think that the password does not get entered since when I try to puts #agent.page.body after the final click_button I see enter password text somewhere in HTML.
What am I doing wrong and how can I fix it?

I've been digging around a bit more and found out that it's not that simple and I could not login with mechanize in any way.
So I ended up with using watir which was fairly simple and straightforward. Here's an example:
browser.goto LOGIN_URL
browser.text_field(:id, 'Email').set #config.email
browser.button(:id, 'next').click
browser.text_field(:id, 'Passwd').wait_until_present
browser.text_field(:id, 'Passwd').set #config.password
browser.button(:id, 'signIn').click
# Here I wait until an element on my target page is visible and then continue
browser.link(:href, '#SOMETHING').wait_until_present
Hope it helps.

Related

Ruby 2.2.0 Google Drive OAuth refresh

I am trying to setup a command line backup app in ruby which accesses Google Drive using Oauth. I have set everything up in the account, created my Client ID and Secret. I seem to remember doing this before but cannot find the code. I used this answer before I think: Ruby google_drive gem oAuth2 saving
I have made a class to handle the Google Drive stuff then there is the applications main file which if given "hard" as an argument will do the initial setup where you have to copy and paste the link into the web browser in order to get a code which you can then paste into the CLI to get the initial access token and refresh token. This works and following these steps my list method (when not commented out) correctly lists everything in Google Drive. When I do the initial setup I am manually copying the access and refresh tokens to .access_token and .refresh_token, these are loading in the code fine.
What is not working, is refreshing the token which I understand I need to do otherwise it will expire, meaning I will have to go through the initial setup again which is obviously a pain (and not suitable for a CRON job). I am getting the following error when I run #auth.refresh!
/home/user/.rvm/gems/ruby-2.2.0/gems/signet-0.6.0/lib/signet/oauth_2/client.rb:947:in `fetch_access_token': Authorization failed. Server message: (Signet::AuthorizationError)
{
"error" : "invalid_grant"
}
from /home/user/.rvm/gems/ruby-2.2.0/gems/signet-0.6.0/lib/signet/oauth_2/client.rb:964:in `fetch_access_token!'
from /home/user/.rvm/gems/ruby-2.2.0/gems/signet-0.6.0/lib/signet/oauth_2/client.rb:981:in `refresh!'
from /home/user/Development/BackupBadger/Sources/Mechanisms/GoogleDriveMechanism.rb:62:in `connect'
from BackupBadger.rb:9:in `<main>'
I have had a look to see what it might be but am for the moment stuck on why this error is triggering when I can seemingly authenticate (since I can list all files on the drive)
My main file
$root=File.join('/home/user/Development/BackupBadger')
$sources=File.join($root,'Sources')
require File.join($sources,'Mechanisms','GoogleDriveMechanism.rb')
badger = BackupBadger::GoogleDriveMechanism.new
if ARGV[0] == "hard" then
badger.hard_setup
else
badger.connect
#badger.list
end
My class Google Drive
module BackupBadger
require 'google/api_client'
require 'google_drive'
require 'pp'
require File.join($sources,'Mechanism.rb')
class GoogleDriveMechanism
def initialize()
#client = Google::APIClient.new(
:application_name => 'Backup Badger',
:application_version => '0.0.1'
)
#access_token_path = File.join($root,'.access_token')
#refresh_token_path = File.join($root,'.refresh_token')
#auth = nil
#access_token = File.open(#access_token_path, "rb").read
#refresh_token = File.open(#refresh_token_path, "rb").read
#session = nil
#client_id = 'CLIENT_ID'
#client_secret = 'CLIENT_SECRET'
#redirect_uri = 'urn:ietf:wg:oauth:2.0:oob'
#scope = "https://www.googleapis.com/auth/drive " +
"https://spreadsheets.google.com/feeds/"
end
# Call this to do the initial setup, which requires pasting a url into a web broswer
def hard_setup
#auth = #client.authorization
#auth.client_id = #client_id
#auth.client_secret = #client_secret
#auth.scope = #scope
#auth.redirect_uri = #redirect_uri
print("1. Open this page:\n%s\n\n" % #auth.authorization_uri)
print("2. Enter the authorization code shown in the page: ")
#auth.code = $stdin.gets.chomp
#auth.fetch_access_token!
#access_token = #auth.access_token
system'clear'
print "Save your access token\n\n"
print #access_token
print "\nSave your refresh token\n\n"
print #auth.refresh_token
end
def connect
#auth = #client.authorization
#auth.client_id = #client_id
#auth.client_secret = #client_secret
#auth.scope = #scope
#auth.redirect_uri = #redirect_uri
#auth.refresh_token = #refresh_token
puts #access_token
puts #refresh_token
# Error is here
#auth.refresh!
#refresh_token = #auth.refresh_token
#access_token = #auth.access_token
File.write(#refresh_token_path, #refresh_token) if #refresh_token
File.write(#access_token_path, #access_token) if #access_token
puts #access_token
puts #refresh_token
end
def list
#session = GoogleDrive.login_with_oauth(#access_token)
for file in #session.files
p file.title
end
end
end
end
If the tl;dr is simply "how do I use a refresh token to get a new access token", then the answer is https://developers.google.com/accounts/docs/OAuth2WebServer#refresh
I won't paste the code coz it's likely to change, but in essence you simply POST to a Google URL and the JSON response is a shiny new access token.

How do I follow URL redirection?

I have a URL and I need to retrieve the URL it redirects to (the number of redirections is arbitrary).
One real example I'm working on is:
https://www.google.com/url?q=http://m.zynga.com/about/privacy-center/privacy-policy&sa=D&usg=AFQjCNESJyXBeZenALhKWb52N1vHouAd5Q
which will eventually redirect to:
http://company.zynga.com/privacy/policy
which is the URL I'm interested in.
I tried with open-uri as follows:
privacy_url = "https://www.google.com/url?q=http://m.zynga.com/about/privacy-center/privacy-policy&sa=D&usg=AFQjCNESJyXBeZenALhKWb52N1vHouAd5Q"
final_url = nil
open(privacy_url) do |h|
puts "Redirecting to #{h.base_uri}"
final_url = h.base_uri
end
but I keep getting the original URL back, meaning that final_url is equal to privacy_url.
Is there any way to follow this kind of redirection and programmatically access the resulting URL?
I finally made it, using the Mechanize gem. They key is to enable the follow_meta_refresh options, which is disabled by default.
Here's how
require 'mechanize'
browser = Mechanize.new
browser.follow_meta_refresh = true
start_url = "https://www.google.com/url?q=http://m.zynga.com/about/privacy-center/privacy-policy&sa=D&usg=AFQjCNESJyXBeZenALhKWb52N1vHouAd5Q"
final_url = nil
browser.get(start_url) do |page|
final_url = page.uri.to_s
end
puts final_url # => http://company.zynga.com/privacy/policy

Using Ruby and Mechanize to fill in a remote login form mystery

I am trying to implement a Ruby script that will take in a username and password, then proceed to fill in the account details on a login form on another website and return the then follow a link and retrieve the account history. To do this I am using the Mechanize gem.
I have been following the examples here
but still I cant seem to get it to work. I have simplified this down greatly to try get it to work in parts but a supposedly simple filling in a form is holding me up.
Here is my code:
# script gets called with a username and password for the site
require 'mechanize'
#create a mechanize instant
agent = Mechanize.new
agent.get('https://mysite/Login.aspx') do |login_page|
#fill in the login form on the login page
loggedin_page = login_page.form_with(:id => 'form1') do |form|
username_field = form.field_with(:id => 'ContentPlaceHolder1_UserName')
username_field.value = ARGV[0]
password_field = form.field_with(:id => 'ContentPlaceHolder1_Password')
password_field.value = ARGV[1]
button = form.button_with(:id => 'ContentPlaceHolder1_btnlogin')
end.submit(form , button)
#click the View my history link
#account_history_page = loggedin_page.click(home_page.link_with(:text => "View My History"))
####TEST to see if i am actually making it past the login page
#### and that the View My History link is now visible amongst the other links on the page
loggedin_page.links.each do |link|
text = link.text.strip
next unless text.length > 0
puts text if text == "View My History"
end
##TEST
end
Terminal error message:
stackqv2.rb:19:in `block in <main>': undefined local variable or method `form' for main:Object (NameError)
from /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize.rb:409:in `get'
from stackqv2.rb:8:in `<main>'
You don't need to pass form as an argument to submit. The button is also optional. Try using the following:
loggedin_page = login_page.form_with(:id => 'form1') do |form|
username_field = form.field_with(:id => 'ContentPlaceHolder1_UserName')
username_field.value = ARGV[0]
password_field = form.field_with(:id => 'ContentPlaceHolder1_Password')
password_field.value = ARGV[1]
end.submit
If you really do need to specify which button is used to submit the form, try this:
form = login_page.form_with(:id => 'form1')
username_field = form.field_with(:id => 'ContentPlaceHolder1_UserName')
username_field.value = ARGV[0]
password_field = form.field_with(:id => 'ContentPlaceHolder1_Password')
password_field.value = ARGV[1]
button = form.button_with(:id => 'ContentPlaceHolder1_btnlogin')
loggedin_page = form.submit(button)
It's a matter of scope:
page.form do |form|
# this block has its own scope
form['foo'] = 'bar' # <- ok, form is defined inside this block
end
puts form # <- error, form is not defined here
ramblex's suggestion is to not use a block with your form and I agree, it's less confusing that way.

How to submit formstack form using ruby?

I have a form similiar to THIS and want to be submit data to it from a CSV file using ruby. Here is what I have been trying to do:
require 'uri'
require 'net/http'
params = {
'field15157482-first' => 'bip',
'field15157482-last' => 'bop',
'field15157485' => 'bip#bob.com',
'field15157487' => 'option1'
'fsSubmitButton1196962' => 'Submit'
}
x = Net::HTTP.post_form(URI.parse('http://www.formstack.com/forms/?1196833-GxMTxR20GK'), params)
I keep getting A valid form ID was not supplied. I have a hunch I am using the wrong URL but I don't know what to replace it with.
I would use the the API but I don't have access to the token hence my stone age approach. Any suggestions would be much appreciated.
The form uses hidden variables and cookies to attempt to maintain a "unique session". Fortunately, Mechanize makes handling 'sneaky' forms quite easy.
require "mechanize"
form_uri = "http://www.formstack.com/forms/?1196962-617Z6Foyif"
#agent = Mechanize.new
page = #agent.get form_uri
form = page.forms[0]
form.fields_with(:class => /fsField/).each do |field|
field.value = case field.name
when /first/ then "First Name"
when /last/ then "Last Name"
else "email#address.com"
end
end
page = form.submit form.buttons.first
puts
puts "=== Response Header"
puts
puts page.header
puts
puts "=== Response Body"
puts
puts page.body
Looking at the source on http://www.formstack.com/forms/?1196833-GxMTxR20GK and the example in your link, it appears that formstack forms post to index.php, and require a form id to be passed in to identify which form is being submitted.. Looking at the forms in both examples, you'll see a field similar to this:
<input type="hidden" name="form" value="1196833" />
Try adding the following to your params hash:
'form' => '1196883' # or other appropriate form value
You may also need to include the other hidden fields for a valid submit.

post form parameters difference between Firefox and Ruby Mechanize

I am trying to figure out if mechanize sends correct post query.
I want to log in to a forum (please see html source, mechanize log in my other question) but I get only the login page again. When looking into it I can see that firefox sends out post with parameters like
auth_username=myusername&auth_password=mypassword&auth_login=Login but my script sends
auth_username=radek&auth_password=mypassword is that ok or the &auth_login=Login part must be present?
When I tried to add it using login_form['auth_login'] = 'Login' I got an error gems/mechanize-0.9.3/lib/www/mechanize/page.rb:13 inmeta': undefined method search' for nil:NilClass (NoMethodError)
It seems to me that auth_login is a form button not a field (I don't know if it matters)
[#<WWW::Mechanize::Form
{name nil}
{method "POST"}
{action
"http://www.somedomain.com/login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1"}
{fields
#<WWW::Mechanize::Form::Field:0x36946c0 #name="auth_username", #value="">
#<WWW::Mechanize::Form::Field:0x369451c #name="auth_password", #value="">}
{radiobuttons}
{checkboxes}
{file_uploads}
{buttons
#<WWW::Mechanize::Form::Button:0x36943b4
#name="auth_login",
#value="Login">}>
]
My script is as follow
require 'rubygems'
require 'mechanize'
require 'logger'
agent = WWW::Mechanize.new {|a| a.log = Logger.new("loginYOTA.log") }
agent.follow_meta_refresh = true #Mechanize does not follow meta refreshes by default, we need to set that option.
page = agent.get("http://www.somedomain.com/login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1")
login_form = page.form_with(:method => 'POST') #works
puts login_form.buttons.inspect
puts page.forms.inspect
STDIN.gets
login_form.fields.each { |f| puts "#{f.name} : #{f.value}" }
#STDIN.gets
login_form['auth_username'] = 'myusername'
login_form['auth_password'] = 'mypassword'
login_form['auth_login'] = 'Login'
STDIN.gets
page = agent.submit login_form
#Display message if logged in
puts page.parser.xpath("/html/body/div/div/div/table/tr/td[2]/div/strong").xpath('text()').to_s.strip
puts
puts page.parser.xpath("/html/body/div/div/div/table/tr/td[2]/div").xpath('text()').to_s.strip
output = File.open("login.html", "w") {|f| f.write(page.parser.to_html) }
You can find more code, html, log in my other related question log in with browser and then ruby/mechanize takes it over?
the absence of one parameter compare to firefox in POST caused mechanize not to log in. Adding new parameter solved this problem. So it seems to me that the web server requires &auth_login=Login parameter to be in POST.
You can read how to add new field to mechanize form in another question.

Resources