log in with browser and then ruby/mechanize takes it over? - ruby

Question

Is that even possible? what I need to pass to mechanize? With what url I can start then?
I cannot manage (so far) to log into one website using mechanize so I was thinking if I can do this little workaround. I believe I can capture all cookies and everything else and then pass it to ruby/mechanize to do the rest ...
screenshots below are made using firebug ( Firebug logs the POST or GET request, the response headers )
login that works = just one line
and html for login that works
<script type="text/javascript" src="clientscript/vbulletin_md5.js?v=3612"></script>
<table cellpadding="0" cellspacing="1" border="0">
<tr>
<td class="smallfont" align="left"><label for="navbar_username">User Name</label></td>
<td class="smallfont" align="left" colspan="2"><label for="navbar_password">Password</label></td>
</tr>
<tr>
<td><input type="text" class="button" name="vb_login_username" id="navbar_username" size="10" accesskey="u" tabindex="101" value="User Name" onfocus="if (this.value == 'User Name') this.value = '';" /></td>
<td><input type="password" class="button" name="vb_login_password" id="navbar_password" size="10" accesskey="p" tabindex="102" /></td>
<td class="smallfont" align="left" valign="middle"><input type="submit" class="button" value="Log in" tabindex="103" title="Enter your username and password in the boxes provided to login, or click the 'register' button to create a profile for yourself." accesskey="s" />
<label for="cb_cookieuser_navbar">
<input type="checkbox" name="cookieuser" value="1" tabindex="103" id="cb_cookieuser_navbar" accesskey="c" />Save?</label>
<input type="hidden" name="s" value="" />
<input type="hidden" name="securitytoken" value="1cbc0286417d97b4eb43ee0b0c2b54e7c615e5b8" />
<input type="hidden" name="do" value="login" />
<input type="hidden" name="vb_login_md5password" />
<input type="hidden" name="vb_login_md5password_utf" /></td>
</tr>
</table>
</form>
<!-- / login form -->
login that doesn't work for me
and it's html
<form class="login" method="post"> <fieldset>
<legend>Members Login</legend>
<div>
<label for="auth_username">Username</label> <input id="auth_username" name="auth_username">
</div>
<div>
<label for="auth_password">Password</label> <input id="auth_password" name="auth_password" type="password">
</div>
</fieldset>
<div class="buttons">
<input name="auth_login" type="submit" value="Login"><p class="note">Forgot your password?</p>
</div>
</form>
my script is almost the same in both cases.
require 'rubygems'
require 'mechanize'
#agent = WWW::Mechanize.new
agent = WWW::Mechanize.new
page = agent.get("http://www.vbulletin.org/forum/index.php")
login_form = page.form_with(:action => 'login.php?do=login')
puts
login_form.fields.each { |f| puts "#{f.name} : #{f.value}" }
login_form['vb_login_username'] = 'user name'
login_form['vb_login_password'] = ''
page = agent.submit login_form
output = File.open("login.html", "w") {|f| f.write(page.parser.to_html) }
mechanize log for the login that doesn't work
INFO -- : Net::HTTP::Get: /login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-language => en-us,en;q=0.5
DEBUG -- : request-header: connection => keep-alive
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: user-agent => WWW-Mechanize/0.9.3 (http://rubyforge.org/projects/mechanize/)
DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
DEBUG -- : request-header: host => www.somedomain.com
DEBUG -- : request-header: keep-alive => 300
DEBUG -- : Read 400 bytes
DEBUG -- : Read 1424 bytes
DEBUG -- : Read 2448 bytes
DEBUG -- : Read 3211 bytes
DEBUG -- : response-header: vary => Accept-Encoding
DEBUG -- : response-header: cache-control => no-store, no-cache, must-revalidate, post-check=0, pre-check=0
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: expires => Thu, 19 Nov 1981 08:52:00 GMT
DEBUG -- : response-header: content-type => text/html; charset=utf-8
DEBUG -- : response-header: date => Fri, 29 Jan 2010 23:43:12 GMT
DEBUG -- : response-header: content-encoding => gzip
DEBUG -- : response-header: server => Apache/2.2.3 (CentOS)
DEBUG -- : response-header: content-length => 3211
DEBUG -- : response-header: set-cookie => PHPSESSID=7cfilg86ju2ldcgso22246hpu4; path=/, WebStats:visitorId=lSMkcwuSWEE%3D; expires=Mon, 27-Jan-2020 23:43:12 GMT; path=/, WebStats:sessionId=%2B2HHK296t%2BQ%3D; expires=Mon, 27-Jan-2020 23:43:12 GMT; path=/
DEBUG -- : response-header: accept-ranges => bytes
DEBUG -- : response-header: pragma => no-cache
DEBUG -- : gunzip body
DEBUG -- : saved cookie: PHPSESSID=7cfilg86ju2ldcgso22246hpu4
DEBUG -- : saved cookie: WebStats:visitorId=lSMkcwuSWEE%3D
DEBUG -- : saved cookie: WebStats:sessionId=%2B2HHK296t%2BQ%3D
INFO -- : status: 200
DEBUG -- : query: "auth_username=radek&auth_password=mypassword"
INFO -- : Net::HTTP::Post: /login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-language => en-us,en;q=0.5
DEBUG -- : request-header: connection => keep-alive
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: content-type => application/x-www-form-urlencoded
DEBUG -- : request-header: user-agent => WWW-Mechanize/0.9.3 (http://rubyforge.org/projects/mechanize/)
DEBUG -- : request-header: cookie => WebStats:sessionId=%2B2HHK296t%2BQ%3D; WebStats:visitorId=lSMkcwuSWEE%3D; PHPSESSID=7cfilg86ju2ldcgso22246hpu4
DEBUG -- : request-header: referer => http://www.somedomain.com/login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
DEBUG -- : request-header: content-length => 43
DEBUG -- : request-header: host => www.somedomain.com
DEBUG -- : request-header: keep-alive => 300
DEBUG -- : Read 650 bytes
DEBUG -- : Read 1674 bytes
DEBUG -- : Read 2698 bytes
DEBUG -- : Read 3211 bytes
DEBUG -- : response-header: vary => Accept-Encoding
DEBUG -- : response-header: cache-control => no-store, no-cache, must-revalidate, post-check=0, pre-check=0
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: expires => Thu, 19 Nov 1981 08:52:00 GMT
DEBUG -- : response-header: content-type => text/html; charset=utf-8
DEBUG -- : response-header: date => Fri, 29 Jan 2010 23:43:13 GMT
DEBUG -- : response-header: content-encoding => gzip
DEBUG -- : response-header: server => Apache/2.2.3 (CentOS)
DEBUG -- : response-header: content-length => 3211
DEBUG -- : response-header: accept-ranges => bytes
DEBUG -- : response-header: pragma => no-cache
DEBUG -- : gunzip body
INFO -- : status: 200

Answer 1

Your problem is very likely from the fact that Mechanize only tracks cookies created with the Set-Cookie HTTP header. It cannot handle cookies created by JavaScript.

Answer 2

Yes, capturing cookies (eg, through FireCookies Plugin in Firefox), and manually passing it to mechanize could work for most cases.