Why does Mechanize HTML content print <!-- Session expired -->? - ruby
I want to login to my bank account with Mechanize (2.7.3) and print out the current balance.
But after successful login something's not quite right because Mechanize is not showing the correct information that's supposed to be on the page.
The output is interesting. Especially the <!-- Session expired --> part.
Here's the code I'm using:
require 'mechanize'
require 'logger'
# Instantiate a new Mechanize object
a = Mechanize.new do |agent|
agent.user_agent_alias = 'Mac Safari'
agent.follow_meta_refresh = true
agent.log = Logger.new "mech.log"
end
# Fetch URL with Mechanize
a.get('https://mijn.ing.nl/internetbankieren/SesamLoginServlet') do |page|
login_form = page.forms.first
# Store the randomly generated input names into variables
username_input_name = login_form.fields[0].name
password_input_name = login_form.fields[1].name
# Fill in the username and password form
login_form.field_with(:name => username_input_name).value = 'username'
login_form.field_with(:name => password_input_name).value = 'password'
# Login
dashboard_page = login_form.submit(nil, {'Cookie' => HTTP::Cookie.cookie_value(a.cookie_jar.cookies)})
puts dashboard_page.content
# Check if the login was successfull
puts "=================="
puts check_1 = dashboard_page.title == 'Mijn ING Overzicht - Mijn ING' ? "CHECK 1 LOGIN SUCCESS" : "CHECK 1 LOGIN FAIL"
puts "=================="
end
Here's the console output with a.follow_meta_refresh = true:
<html>
<body>
<!-- Session expired -->
<script>
<!-- Hide script from old browsers
function urlencode(str) {
return escape(str).replace('%3A', ':').replace('+', '%2B').replace('%20', '+').replace('*', '%2A').replace('/', '%2F').replace('#', '%40').replace('%2F', '/');
}
var url_encoded_referrer = urlencode(document.location);
var url = 'https://' + document.location.hostname + '/ssm/sso/login?Target=' + url_encoded_referrer;
window.location = url;
//-- Stop hiding script -->
</script>
</body>
</html>
==================
CHECK 1 LOGIN FAIL
==================
This is the logfile:
# Logfile created on 2013-12-27 09:16:17 +0100 by logger.rb/41954
I, [2013-12-27T09:16:17.112862 #978] INFO -- : Net::HTTP::Get: /internetbankieren/SesamLoginServlet
D, [2013-12-27T09:16:17.112910 #978] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2013-12-27T09:16:17.112935 #978] DEBUG -- : request-header: accept => */*
D, [2013-12-27T09:16:17.112957 #978] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22
D, [2013-12-27T09:16:17.112978 #978] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2013-12-27T09:16:17.112999 #978] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2013-12-27T09:16:17.113019 #978] DEBUG -- : request-header: host => mijn.ing.nl
I, [2013-12-27T09:16:17.678728 #978] INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2013-12-27T09:16:17.678787 #978] DEBUG -- : response-header: date => Fri, 27 Dec 2013 08:16:17 GMT
D, [2013-12-27T09:16:17.678813 #978] DEBUG -- : response-header: cache-control => no-cache, no-store
D, [2013-12-27T09:16:17.678834 #978] DEBUG -- : response-header: pragma => no-cache
D, [2013-12-27T09:16:17.678855 #978] DEBUG -- : response-header: expires => Thu, 01 Jan 1970 00:00:00 GMT
D, [2013-12-27T09:16:17.678877 #978] DEBUG -- : response-header: set-cookie => sessiontype=mpb; Secure, aac=332016DB0A85C1245596211F7D403A78; Expires=Sat, 27 Dec 2014 08:16:17 GMT; Domain=.ing.nl; Secure, internetbankierenmi=1575004352.20480.0000; path=/, TS765584=312694932a75764058a09928f7990ca23a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7; Path=/
D, [2013-12-27T09:16:17.678900 #978] DEBUG -- : response-header: vary => Accept-Encoding,User-Agent
D, [2013-12-27T09:16:17.678921 #978] DEBUG -- : response-header: content-encoding => gzip
D, [2013-12-27T09:16:17.678942 #978] DEBUG -- : response-header: keep-alive => timeout=90, max=1000
D, [2013-12-27T09:16:17.678962 #978] DEBUG -- : response-header: connection => Keep-Alive
D, [2013-12-27T09:16:17.678983 #978] DEBUG -- : response-header: content-type => text/html;charset=ISO-8859-1
D, [2013-12-27T09:16:17.679003 #978] DEBUG -- : response-header: content-language => en
D, [2013-12-27T09:16:17.679023 #978] DEBUG -- : response-header: transfer-encoding => chunked
D, [2013-12-27T09:16:17.679118 #978] DEBUG -- : Read 10 bytes (10 total)
D, [2013-12-27T09:16:17.679974 #978] DEBUG -- : Read 1443 bytes (1453 total)
D, [2013-12-27T09:16:17.680403 #978] DEBUG -- : Read 1448 bytes (2901 total)
D, [2013-12-27T09:16:17.680469 #978] DEBUG -- : Read 1112 bytes (4013 total)
D, [2013-12-27T09:16:17.680566 #978] DEBUG -- : gzip response
D, [2013-12-27T09:16:17.683042 #978] DEBUG -- : saved cookie: sessiontype=mpb
D, [2013-12-27T09:16:17.683568 #978] DEBUG -- : saved cookie: aac=332016DB0A85C1245596211F7D403A78
D, [2013-12-27T09:16:17.683684 #978] DEBUG -- : saved cookie: internetbankierenmi=1575004352.20480.0000
D, [2013-12-27T09:16:17.683778 #978] DEBUG -- : saved cookie: TS765584=312694932a75764058a09928f7990ca23a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7
I, [2013-12-27T09:16:17.685622 #978] INFO -- : form encoding: ISO-8859-1
D, [2013-12-27T09:16:17.685989 #978] DEBUG -- : query: "a8yzXvCb7f5ZKsFaT=n.aramjan&a9jSD_u8G4nO23oXI=EYoss2303"
I, [2013-12-27T09:16:17.686528 #978] INFO -- : Net::HTTP::Post: /internetbankieren/SesamLoginServlet
D, [2013-12-27T09:16:17.686562 #978] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2013-12-27T09:16:17.686587 #978] DEBUG -- : request-header: accept => */*
D, [2013-12-27T09:16:17.686609 #978] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22
D, [2013-12-27T09:16:17.686631 #978] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2013-12-27T09:16:17.686653 #978] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2013-12-27T09:16:17.686677 #978] DEBUG -- : request-header: cookie => TS765584=312694932a75764058a09928f7990ca23a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7; aac=332016DB0A85C1245596211F7D403A78; internetbankierenmi=1575004352.20480.0000; sessiontype=mpb
D, [2013-12-27T09:16:17.686711 #978] DEBUG -- : request-header: host => mijn.ing.nl
D, [2013-12-27T09:16:17.686739 #978] DEBUG -- : request-header: referer => https://mijn.ing.nl/internetbankieren/SesamLoginServlet
D, [2013-12-27T09:16:17.686760 #978] DEBUG -- : request-header: content-type => application/x-www-form-urlencoded
D, [2013-12-27T09:16:17.686781 #978] DEBUG -- : request-header: content-length => 55
I, [2013-12-27T09:16:18.225851 #978] INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2013-12-27T09:16:18.225913 #978] DEBUG -- : response-header: date => Fri, 27 Dec 2013 08:16:17 GMT
D, [2013-12-27T09:16:18.225938 #978] DEBUG -- : response-header: x-ci => r=HPHBPNBMP;c=;a=VRRCSL;u=JXWMBKM
D, [2013-12-27T09:16:18.225962 #978] DEBUG -- : response-header: cache-control => no-cache, no-store, no-cache, no-store
D, [2013-12-27T09:16:18.225983 #978] DEBUG -- : response-header: pragma => no-cache, no-cache
D, [2013-12-27T09:16:18.226005 #978] DEBUG -- : response-header: expires => Thu, 01 Jan 1970 00:00:00 GMT
D, [2013-12-27T09:16:18.226030 #978] DEBUG -- : response-header: set-cookie => SESSESSIONID=0000e3Sar84Bz7XlOymbQkge4gX:17s9ivth6; Path=/; Domain=.ing.nl; Secure, iid=VQGIG0Xq3%2BzTjLSPo5YDiw%3D%3D%3Bmss1; Expires=Wed, 26 Feb 2014 08:16:17 GMT; Path=/; Domain=.ing.nl, gsc=m=s; HttpOnly; Path=/; Domain=.mijn.ing.nl; Secure, Session201=HKVCKUYMLURFZQIENENOANWSSNRUERNF051e7e43; HttpOnly; Path=/; Domain=.mijn.ing.nl; Secure, cookiepref=3; Expires=Sat, 27 Dec 2014 08:16:17 GMT; Path=/; Domain=.ing.nl, TS765584=9e3c30b00a5f1162344f71403a543e573a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7eccc06cabcea90646ac25a59abcf00af2532f3ca7d510a274552b96c4b433c8b5f6a631185c8cdc8; Path=/
D, [2013-12-27T09:16:18.226053 #978] DEBUG -- : response-header: vary => Accept-Encoding,User-Agent
D, [2013-12-27T09:16:18.226073 #978] DEBUG -- : response-header: content-encoding => gzip
D, [2013-12-27T09:16:18.226094 #978] DEBUG -- : response-header: keep-alive => timeout=90, max=1000
D, [2013-12-27T09:16:18.226115 #978] DEBUG -- : response-header: connection => Keep-Alive
D, [2013-12-27T09:16:18.226139 #978] DEBUG -- : response-header: content-type => text/html;charset=ISO-8859-1
D, [2013-12-27T09:16:18.226159 #978] DEBUG -- : response-header: content-language => en
D, [2013-12-27T09:16:18.226180 #978] DEBUG -- : response-header: transfer-encoding => chunked
D, [2013-12-27T09:16:18.226252 #978] DEBUG -- : Read 326 bytes (326 total)
D, [2013-12-27T09:16:18.226381 #978] DEBUG -- : gzip response
D, [2013-12-27T09:16:18.226730 #978] DEBUG -- : saved cookie: SESSESSIONID=0000e3Sar84Bz7XlOymbQkge4gX:17s9ivth6
D, [2013-12-27T09:16:18.226933 #978] DEBUG -- : saved cookie: iid=VQGIG0Xq3%2BzTjLSPo5YDiw%3D%3D%3Bmss1
D, [2013-12-27T09:16:18.227051 #978] DEBUG -- : saved cookie: gsc=m=s
D, [2013-12-27T09:16:18.227152 #978] DEBUG -- : saved cookie: Session201=HKVCKUYMLURFZQIENENOANWSSNRUERNF051e7e43
D, [2013-12-27T09:16:18.227311 #978] DEBUG -- : saved cookie: cookiepref=3
D, [2013-12-27T09:16:18.227423 #978] DEBUG -- : saved cookie: TS765584=9e3c30b00a5f1162344f71403a543e573a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7eccc06cabcea90646ac25a59abcf00af2532f3ca7d510a274552b96c4b433c8b5f6a631185c8cdc8
I, [2013-12-27T09:16:18.230318 #978] INFO -- : Net::HTTP::Get: /particulier/betalen/index
D, [2013-12-27T09:16:18.230355 #978] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2013-12-27T09:16:18.230379 #978] DEBUG -- : request-header: accept => */*
D, [2013-12-27T09:16:18.230401 #978] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22
D, [2013-12-27T09:16:18.230421 #978] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2013-12-27T09:16:18.230456 #978] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2013-12-27T09:16:18.230478 #978] DEBUG -- : request-header: cookie => SESSESSIONID=0000e3Sar84Bz7XlOymbQkge4gX:17s9ivth6; Session201=HKVCKUYMLURFZQIENENOANWSSNRUERNF051e7e43; cookiepref=3; gsc=m=s; iid=VQGIG0Xq3%2BzTjLSPo5YDiw%3D%3D%3Bmss1
D, [2013-12-27T09:16:18.230501 #978] DEBUG -- : request-header: host => bankieren.mijn.ing.nl
I, [2013-12-27T09:16:18.714637 #978] INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2013-12-27T09:16:18.714692 #978] DEBUG -- : response-header: content-length => 506
D, [2013-12-27T09:16:18.714716 #978] DEBUG -- : response-header: content-type => text/html
D, [2013-12-27T09:16:18.714738 #978] DEBUG -- : response-header: date => Fri, 27 Dec 2013 08:16:18 GMT
D, [2013-12-27T09:16:18.714760 #978] DEBUG -- : response-header: p3p => CP="NON CUR OTPi OUR NOR UNI"
D, [2013-12-27T09:16:18.714781 #978] DEBUG -- : response-header: cache-control => no-cache
D, [2013-12-27T09:16:18.714801 #978] DEBUG -- : response-header: pragma => no-cache
D, [2013-12-27T09:16:18.714825 #978] DEBUG -- : response-header: set-cookie => S-SESSION-ID=2_0_nG1XBMZWH2kJdOBjCloZA651nXttTm5wiYowRBYDQY7gxH1L; Path=/; Secure; HttpOnly, lb-4-25a-1=208176394.20480.0000; path=/, TS31e294=06caec21f242b201e4b52847e45a8a475bd58411f71b544852bd3752022ecc54fb717d9b1b4d09a87ea0bb21; Path=/
D, [2013-12-27T09:16:18.714889 #978] DEBUG -- : Read 506 bytes (506 total)
D, [2013-12-27T09:16:18.715236 #978] DEBUG -- : saved cookie: S-SESSION-ID=2_0_nG1XBMZWH2kJdOBjCloZA651nXttTm5wiYowRBYDQY7gxH1L
D, [2013-12-27T09:16:18.715340 #978] DEBUG -- : saved cookie: lb-4-25a-1=208176394.20480.0000
D, [2013-12-27T09:16:18.715437 #978] DEBUG -- : saved cookie: TS31e294=06caec21f242b201e4b52847e45a8a475bd58411f71b544852bd3752022ecc54fb717d9b1b4d09a87ea0bb21
Any ideas? It would be great to solve this on Christmas day!
Double check if cookies are persisted across clicks, and try selenium - it's possible some funny JS stuff is going on that page and selenium would handle it for you (http://rubygems.org/gems/selenium-webdriver)
UPDATE
See this answer for more info on cookie handling in Mechanize: Maintaining cookies between Mechanize requests
Also, try disabling JS in the browser and then log into your bank, make sure everything still works without JS (if it works then great, if not then you need to use a JS-capable driver)
CONCLUSION
Cause in this case was the bank's website, it depends on client-side JavaScript code to function properly so a JS-incapable driver like mechanize won't do. You will need to use one of the JavaScript-capable drivers - most popular of which are selenium, webkit and poltergeist - all decent, but neither really satisfying.
Considering all three require some host-side dependencies (poltergeist least of all though) the "what installs best" might be the prevailing factor in choosing between them.
For a Heroku deployment....
poltergeist does seem to have some issues (websocket related, seems heroku is blocking them) but it's the best bet for heroku deployment https://github.com/jonleighton/poltergeist/issues/194
and for selenium and webkit, quick google shows a number of issues - they seem to require X and/or xvfb, see Is it possible to run capybara-webkit (i.e. forked webkit_server) on Heroku Cedar? and Is there a working nodejs/phantomjs Heroku buildpack? and Running selenium browser on server (Flask/Python/Heroku) for further reading
Related
Oracle UTL_HTTP request returns a shorter response than when the same request is done in Python or curl
I'm accessing a REST service for data query and download. This is the very first call that performs the authentication. The response is a json structure that contains an authentication token.
When I do this call with curl ...
$ curl -v -X POST ${AUTH_URL} \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'apikey='${API_KEY}'&grant_type=api_key&client_id=IDP'
... I get the following response.
First the headers:
< server: IIS
< date: Thu, 25 Feb 2021 17:59:34 GMT
< content-type: application/json
< content-length: 1500
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< strict-transport-security: max-age=31536000; includeSubdomains;
< cache-control: no-store
< set-cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/IDP/; HttpOnly
< pragma: no-cache
< x-frame-options: SAMEORIGIN
< referrer-policy: no-referrer
< vary: Origin
< via: 1.1 google
< alt-svc: clear
And the content:
{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIxMThSRENzZTlqUWR4UVRnSkt2ZXlvSHBaaWE4R0pIVEU5RjJPSmE1M3N3In0.eyJleHAiOjE2MTQyOTAzNzQsImlhdCI6MTYxNDI3NTk3NCwianRpIjoiNzBkMmMwZGMtZWY3Yy00NDM5LWJiNmUtNmQ4MDEzZGU2YTU0IiwiaXNzIjoiaHR0cHM6Ly9hdXRoZW50aWNhdGUuZm91bmRhdGlvbi5hcGkub25lYXRsYXMuYWlyYnVzLmNvbS9hdXRoL3JlYWxtcy9JRFAiLCJhdWQiOiJJRFAiLCJzdWIiOiJmNmI0ZjVkNC0zMzZiLTRlMTctODc3Ni1jNjA1ZTczNTRjYmIiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJJRFAiLCJzZXNzaW9uX3N0YXRlIjoiODJkYWM4MjMtMTViOS00MmVlLWE2YzEtZjg2ZDQ2MzY1ZjAzIiwiYWNyIjoiMSIsInNjb3BlIjoiIiwibmJmIjowLCJyb2xlIjoie1wiZ2VvLmlkcC5ub3RpZnlcIjpbXCJ1c2VyXCJdLFwiZ2VvLmFwcC5vYWRcIjpbXCJ1c2VyXCJdLFwiZ2VvLmlkcC5kYXRhc3RvcmVcIjpbXCJ1c2VyXCJdLFwiZ2VvLmFwcC53b3JrYmVuY2hcIjpbXCJ1c2VyXCJdfSIsInJvbGVzIjp7Imdlby5pZHAubm90aWZ5IjpbInVzZXIiXSwiZ2VvLmFwcC5vYWQiOlsidXNlciJdLCJnZW8uaWRwLmRhdGFzdG9yZSI6WyJ1c2VyIl0sImdlby5hcHAud29ya2JlbmNoIjpbInVzZXIiXX0sInN1aWQiOiIxMTg2NTEzNTQ2IiwidXVpZCI6Ijc5Yjg0MTZlLTY0NDYtNGMwYy1hODg1LWY1MzE2ZGMzOWMyZSIsImxvYSI6MTAwfQ.5W_E4fkhirbJZNAJ_TwMbLhcKdmnHBXOjvLUr4vW-DBRvSFfQrpdlDHLMIVI4B7bZ-OU_FVnH__i_diKYJFRH4l3Zqy8maa1pyj_WhZJksqBB69ehv8xx_3qtuJCZ0z0hln0FzmyG_Ep_uaru3gK_h33SuFxjdKr4F5XocyrYpGE-ewm-mBLj4DOBnZSJ4HgV0BG02LJIPIU8BybTmvgV-4mW3LXOVKDUJMmP4TF_ZEUzNz4a1vhoW4VIOvaNkk_8v8m_R4zjNOGmd_4jWEywORBZ1ofqvn72usY7TWEVpGBxR-rKYgzWXrdeBE4_l61MT420rBID9dbI2zRgEyVIQ","expires_in":14400,"refresh_expires_in":0,"token_type":"bearer","not-before-policy":0,"session_state":"82dac823-15b9-42ee-a6c1-f86d46365f03","scope":""}
Notice that the length of the content is 1500. That is also what the content-length: 1500 header says. When I do the same test using Python, I get the same result: a 1500 characters result.
But when I do the same test using Oracle UTL_HTTP, the result is only 1453 characters. Here is debugging from my PL/SQL code:
% resp.status_code=200
% resp.reason_phrase=OK
% resp.http_version=HTTP/1.0
% resp.get_headers
% .. Server: IIS
% .. Date: Thu, 25 Feb 2021 17:50:51 GMT
% .. Content-Type: application/json
% .. Content-Length: 1453
% .. X-Content-Type-Options: nosniff
% .. X-XSS-Protection: 1; mode=block
% .. Strict-Transport-Security: max-age=31536000; includeSubdomains;
% .. Cache-Control: no-store
% .. Set-Cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT;
Max-Age=0; Path=/auth/realms/IDP/; HttpOnly
% .. Pragma: no-cache
% .. X-Frame-Options: SAMEORIGIN
% .. Referrer-Policy: no-referrer
% .. Vary: Origin
% .. Via: 1.1 google
% .. Alt-Svc: clear
% Response:
{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIxMThSRENzZT
lqUWR4UVRnSkt2ZXlvSHBaaWE4R0pIVEU5RjJPSmE1M3N3In0.eyJleHAiOjE2MTQyODk4NTEsImlhd
CI6MTYxNDI3NTQ1MSwianRpIjoiMDQ4ZjhlMjctZTgwZi00MjIyLWFmNDAtMjZlNDdmYTFhMDg0Iiwi
aXNzIjoiaHR0cHM6Ly8zNS4xOTAuNTkuNzkvYXV0aC9yZWFsbXMvSURQIiwiYXVkIjoiSURQIiwic3V
iIjoiZjZiNGY1ZDQtMzM2Yi00ZTE3LTg3NzYtYzYwNWU3MzU0Y2JiIiwidHlwIjoiQmVhcmVyIiwiYX
pwIjoiSURQIiwic2Vzc2lvbl9zdGF0ZSI6ImYzYmZlOGJiLTFiZGYtNDQ5OS04NDQwLWIxODk5OGYwY
jg5NiIsImFjciI6IjEiLCJzY29wZSI6IiIsIm5iZiI6MCwicm9sZSI6IntcImdlby5pZHAubm90aWZ5
XCI6W1widXNlclwiXSxcImdlby5hcHAub2FkXCI6W1widXNlclwiXSxcImdlby5pZHAuZGF0YXN0b3J
lXCI6W1widXNlclwiXSxcImdlby5hcHAud29ya2JlbmNoXCI6W1widXNlclwiXX0iLCJyb2xlcyI6ey
JnZW8uaWRwLm5vdGlmeSI6WyJ1c2VyIl0sImdlby5hcHAub2FkIjpbInVzZXIiXSwiZ2VvLmlkcC5kY
XRhc3RvcmUiOlsidXNlciJdLCJnZW8uYXBwLndvcmtiZW5jaCI6WyJ1c2VyIl19LCJzdWlkIjoiMTE4
NjUxMzU0NiIsInV1aWQiOiI3OWI4NDE2ZS02NDQ2LTRjMGMtYTg4NS1mNTMxNmRjMzljMmUiLCJsb2E
iOjEwMH0.XHwxx3TzNNwgzVMv18Jav4bqXW9Q4n2bP_HV1iy0K4VPH-w84tXsHjXfH_f05Ynn2CXqv-
rdHds6KtuZaI1aypNnIvNvmbUiNHd6M1geLY4w8Yy9rg9-WFjYiFXbLTP7vvUAMSHueJmeT6WvzAsUT
Z7IQdp0w5aLDQ6ElV8pX1khBMCC7uXedRRDK-UC1MlJBrWtbhIMu5MaqpdpPeBcBMCvmqUBFTFfW6dQ
Ko01jeDjxePz_gZ2wdyU8fkV8UNTzkS3i6PYUkcxi3pmEC5r93JSNGVRUsZ53y5IjcaJK4aRXvvZQzV
iOitsbu8Pfciii2E_NDlk3qYgSqlxVrmzNA","expires_in":14400,"refresh_expires_in":0,
"token_type":"bearer","not-before-policy":0,"session_state":"f3bfe8bb-1bdf-4499
-8440-b18998f0b896","scope":""}
Notice the content length is now 1453 characters. The difference is with the token information inside the JSON response. It should be 1330 characters, but is only 1283 characters. The rest of the JSON document is the same. And the returned token is not valid for further use.
I can't find any explanation as to why the response is shorter when requested from UTL_HTTP. I first thought it had something to do with character set encoding. I have everything set to UTF-8. The token is returned in a base64 encoding.
Here is the code I use (I did not include the debugging code):
-- Setup the http request type and add the content
http_req := utl_http.begin_request(url, 'POST', 'HTTP/1.0');
utl_http.set_header(http_req, 'Content-Type', content_type);
utl_http.set_body_charset(http_req,'UTF-8');
utl_http.set_header(http_req, 'Content-Length', length(post_content));
utl_http.write_text(http_req, post_content);
-- Call the REST endpoint and fetch the http response
http_resp := utl_http.get_response(http_req);
utl_http.set_body_charset(http_resp,'UTF-8');
-- Read the response content
begin
json_response := '';
i := 0;
loop
utl_http.read_line(http_resp, response_line, false);
json_response := json_response || response_line;
i := i + 1;
end loop;
exception
when utl_http.end_of_body then
utl_http.end_response(http_resp);
end;
I have been staring at this problem for hours. I tried various things, like setting or not setting explicit character set encoding, all to not effect. I can't see what I'm doing wrong and why Oracle would do anything with the response. I could imagine it truncating it for some reason - but why would characters be removed from inside the response ?
I'm slightly ashamed. The answer is simple: in the begin_request() call I was explicitly specifying the HTTP 1.0 protocol: http_req := utl_http.begin_request(url, 'POST', 'HTTP/1.0'); Once I got rid of that (which means Oracle uses the HTTP 1.1 protocol): http_req := utl_http.begin_request(url, 'POST'); Then everything started working correctly: I now get the full and entire response from the server. There is still an oddity, in that when I do curl --http1.0 to also force use of HTTP 1.0, I still get the right answer. So I assume that there is something inside the Oracle implementation that causes trouble when using HTTP 1.0 on some workloads.
Gmail API removes message part
When I send a message with attachment using the Gmail API, the recipient receives the message without the attachment.
What is strange though is that:
1: in the sent folder of the sender, I do see the attachment properly
2: if I send to myself, both message are fine (in sent folder and in inbox folder)
3: if I use with GMail SMTP with the same raw message, it works fine
4: if I use a 3rd party SMTP with the same raw message, it works fine.
Point number 1+2 super puzzling.
Here is the source of the original message in the sent folder:
Received: from 13936824666 named unknown by gmailapi.google.com with HTTPREST; Wed, 25 Jan 2017 18:44:30 -0500
Date: Wed, 25 Jan 2017 18:44:30 -0500
From: Jeremy Chatelaine <source#gmail.com>
To: Jeremy <target#domain.com>
Message-Id: <CABX8Avad0vTtu8=jotRD5HM1r0My-ZKeV7RFRo0TmTxf5PNd0g#mail.gmail.com>
Subject: Export
Mime-Version: 1.0
Content-Type: multipart/alternative; boundary="--==_mimepart_5889385be5066_a133fd0f785e20837629"; charset=UTF-8
Content-Transfer-Encoding: 7bit
----==_mimepart_5889385be5066_a133fd0f785e20837629
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Please find attached the message
----==_mimepart_5889385be5066_a133fd0f785e20837629
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 7bit
Please find attached the message
----==_mimepart_5889385be5066_a133fd0f785e20837629
Content-Type: text/csv; charset=UTF-8; filename=export.csv
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=export.csv
Content-ID: <5889385be6fd7_a133fd0f785e20837735#jeremy.mail>
Some text...
----==_mimepart_5889385be5066_a133fd0f785e20837629--
Here is the source of the original message in the recipient inbox folder:
Delivered-To: target#domain.com
Received: by 10.55.110.193 with SMTP id j184csp1999707qkc;
Wed, 25 Jan 2017 16:30:42 -0800 (PST)
X-Received: by 10.107.34.213 with SMTP id i204mr540101ioi.203.1485390642385;
Wed, 25 Jan 2017 16:30:42 -0800 (PST)
Return-Path: <source#domain.com>
Received: from mail-it0-x22b.google.com (mail-it0-x22b.google.com. [2607:f8b0:4001:c0b::22b])
by mx.google.com with ESMTPS id 88si336719ioq.54.2017.01.25.16.30.42
for <target#domain.com>
(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Wed, 25 Jan 2017 16:30:42 -0800 (PST)
Received-SPF: pass (google.com: domain of source#domain.com designates 2607:f8b0:4001:c0b::22b as permitted sender) client-ip=2607:f8b0:4001:c0b::22b;
Authentication-Results: mx.google.com;
dkim=pass header.i=#domain.com;
spf=pass (google.com: domain of source#domain.com designates 2607:f8b0:4001:c0b::22b as permitted sender) smtp.mailfrom=source#domain.com
Received: by mail-it0-x22b.google.com with SMTP id 203so119411500ith.0
for <target#domain.com>; Wed, 25 Jan 2017 16:30:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=domain.com; s=domain;
h=from:mime-version:date:message-id:subject:to;
bh=wx26/V0bJk9VItDp3TAvKl28UAn7IRQq4NITJZDM+Co=;
b=L3KTzPTCoIJUfAacuJy+PE8jHnY9iwGuXUWSpZzneRs5bvMysigSMyPGn1YicyIvQ6
d/LvbEJPlsu+S0zElhIVPITjAmXKDKNIKwLQDHpkcKnnI3btBUrENN923fMtS1fDdHyV
3At0QenKrb34uQqYYoHtX2WU4nyYrISbYKL62=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:mime-version:date:message-id:subject:to;
bh=wx26/V0bJk9VItDp3TAvKl28UAn7IRQq4NITJZDM+Co=;
b=tGPSHjI3iRzjrI0jGHVX76uTw7OXYC4B2bP0qQKgQotBWru+Pn5Ci9A1Qop9oGN1Ys
fnxCgLOLG8ZJU165ODBNX1DGjPa8ud9SWg18FTsxIjNw9qTr1yJqbWr0LToJi7HdQUr8
7Aaiqil7PbPUf5SdxLCqwBNf660Rn9Sd/ADZeT1Bc2+iYQcizjiK/rOPPX+X1ZndvqxP
Ok3Ac2yyIWxi+m4xaPEztcF4JXFZDlJWFdclUDv4s5Jdc0eb1HmB5d2r1qroGLo5MTjd
d7jO1nRsKTO5I9I69p9AgC+LpDiWBxgzZMBVsU6vVpeZ03/pCroyk9DHDUAjn3ijtlFh
O2vw==
X-Gm-Message-State: AIkVDXJo2tPJlkHkthgEjnxYp7vz5Rfpn91pWCS7zEurkSiDhJtyzLpUoSDORq37K/7ATCRSyLAypKIYrdYwEiL2
X-Received: by 10.36.84.148 with SMTP id t142mr20263701ita.90.1485390641934; Wed, 25 Jan 2017 16:30:41 -0800 (PST)
Received: from 13936824667 named unknown by gmailapi.google.com with HTTPREST; Wed, 25 Jan 2017 19:30:41 -0500
From: Jeremy Chatelaine <source#domain.com>
Mime-Version: 1.0
Date: Wed, 25 Jan 2017 19:30:41 -0500
Message-ID: <CABK5xHonVstaJHWHfvBJFF1BK5Y0B8NJsAJTPFokPNAUcawhGA#mail.gmail.com>
Subject: Export
To: Jeremy <target#domain.com>
Content-Type: multipart/alternative; boundary=001a1143a6cc90c9040546f4753f
--001a1143a6cc90c9040546f4753f
Content-Type: text/plain; charset=UTF-8
Please find attached your requested export
--001a1143a6cc90c9040546f4753f
Content-Type: text/html; charset=UTF-8
Please find attached your requested export
--001a1143a6cc90c9040546f4753f--
As you can see, the mime part where I had my text attachment vanished.
Here is how the message is produced (cut for clarity)
msg = Mail.new
html_part = Mail::Part.new do
content_type 'text/html; charset=UTF-8'
body html_body
end
msg.html_part = html_part
new_text = plan_text
text_part = Mail::Part.new do
body new_text
end
msg.text_part = text_part
file_paths.each do |file_path|
msg.add_file(file_path)
# I also tried like that, same result
#open(file_path) do |file|
# msg.attachments[file_path] = file.read
#end
end
raw_message = msg.to_s
Here is how I send with the gmail api
client = Google::APIClient.new(:application_name => "app", :application_version => "1")
client.authorization.client_id = "someverylongnumbers.apps.googleusercontent.com"
client.authorization.client_secret = "morerandomletters"
client.authorization.access_token = token
client.authorization.scope = [
"https://www.googleapis.com/auth/gmail.modify"
]
gmail_api = client.discovered_api('gmail', 'v1') # https://www.googleapis.com/auth/gmail.modify
result = client.execute(
:api_method => gmail_api.users.messages.to_h['gmail.users.messages.send'],
:parameters => {
'userId' => "me"
},
:body_object => {
'raw' => Base64.urlsafe_encode64(raw_message)
}
)
What is wrong with this?
Typhoeus gem is not fetching the primary IP
I am using Typhoeus gem to run HTTP request. But when I am running in another system its running fine and returning the correct primary IP. But in my system it's not giving the primary IP -- instead of that it returns nil.
My code is like
request = Typhoeus.post("www.google.com")
response in another system:
Typhoeus::Response:0xb45c1f70 #options={:httpauth_avail=>0, :total_time=>0.101638, :starttransfer_time=>0.10146, :appconnect_time=>0.0, :pretransfer_time=>0.035663, :connect_time=>0.035659, :namelookup_time=>0.00018, :effective_url=>"http://www.google.com", :primary_ip=>"74.125.236.48", :response_code=>405, :redirect_count=>0, :return_code=>:ok,
response in my system:
Typhoeus::Response:0xbc9704c #options={:httpauth_avail=>0, :total_time=>0.589821, :starttransfer_time=>0.419524, :appconnect_time=>nil, :pretransfer_time=>0.194883, :connect_time=>0.194829, :namelookup_time=>0.02522, :effective_url=>"http://www.google.com", :primary_ip=>nil, :response_code=>200, :redirect_count=>0, :return_code=>:ok,
I want to know why its happening like this. I am waiting for good answer.
UPDATED
FULL RESPONSE from another system:
Typhoeus::Response:0xb4cebdb4 #options={:httpauth_avail=>0, :total_time=>0.077503, :starttransfer_time=>0.077249, :appconnect_time=>0.0, :pretransfer_time=>0.013626, :connect_time=>0.01362, :namelookup_time=>0.000143, :effective_url=>"http://www.google.com", :primary_ip=>"74.125.236.50", :response_code=>405, :redirect_count=>0, :return_code=>:ok, :response_headers=>"HTTP/1.1 405 Method Not Allowed\r\nAllow: GET, HEAD\r\nDate: Thu, 10 Apr 2014 12:42:44 GMT\r\nContent-Type: text/html; charset=UTF-8\r\nServer: gws\r\nContent-Length: 1453\r\nX-XSS-Protection: 1; mode=block\r\nX-Frame-Options: SAMEORIGIN\r\nAlternate-Protocol: 80:quic\r\n\r\n", :response_body=>"\n\n \n \n Error 405 (Method Not Allowed)!!1\n \n {margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px} > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}#media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/errors/logo_sm_2.png) no-repeat}#media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/errors/logo_sm_2_hr.png) 0}}#media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:55px;width:150px}\n \n \n 405. That\xE2\x80\x99s an error.\n The request method POST is inappropriate for the URL /. That\xE2\x80\x99s all we know.\n", :debug_info=>#}, #request=#:post}, #options={:method=>:post, :headers=>{"User-Agent"=>"Typhoeus - https://github.com/typhoeus/typhoeus"}, :maxredirs=>50}, #on_headers=[], #response=#, #on_complete=[], #on_failure=[]>
FULL RESPONSE from MY system:
Typhoeus::Response:0xbf277e4 #options={:httpauth_avail=>0, :total_time=>0.542332, :starttransfer_time=>0.374829, :appconnect_time=>nil, :pretransfer_time=>0.192083, :connect_time=>0.192035, :namelookup_time=>0.025185, :effective_url=>"http://www.google.com", :primary_ip=>nil, :response_code=>200, :redirect_count=>0, :return_code=>:ok, :response_headers=>"HTTP/1.1 200 OK\r\nDate: Thu, 10 Apr 2014 12:44:38 GMT\r\nExpires: -1\r\nCache-Control: private, max-age=0\r\nContent-Type: text/html; charset=ISO-8859-1\r\nSet-Cookie: PREF=ID=075796115d23a806:FF=0:TM=1397133878:LM=1397133878:S=9K9PiB355V_pLs56; expires=Sat, 09-Apr-2016 12:44:38 GMT; path=/; domain=.google.com\r\nSet-Cookie: NID=67=lbUdPyBSFruSAjPvOJZJhr25WNN43JzP--0oYtYxHvMxYYJqeWZBj8wRZi6qMPxw7XPHHylholrcflVZ-SpIdci8GL_guJcfAnh1O8XJHKQb3Qu67MA62L-bhlCbvgST; expires=Fri, 10-Oct-2014 12:44:38 GMT; path=/; domain=.google.com; HttpOnly\r\nP3P: CP=\"This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info.\"\r\nServer: gws\r\nX-XSS-Protection: 1; mode=block\r\nX-Frame-Options: SAMEORIGIN\r\nAlternate-Protocol: 80:quic\r\nTransfer-Encoding: chunked\r\n\r\n", :response_body=>"http://schema.org/WebPage\" lang=\"en\">Google(function(){\nwindow.google={kEI:\"NpJGU4rKMeWCiQe46IGABA\",getEI:function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute(\"eid\")));)a=a.parentNode;return b||google.kEI},https:function(){return\"https:\"==window.location.protocol},kEXPI:\"17259,4000116,4007661,4007830,4008067,4008133,4008142,4009033,4009565,4009641,4010806,4010858,4010899,4011228,4011258,4011679,4012373,4012504,4012508,4013374,4013414,4013591,4013723,4013747,4013757,4013787,4013823,4013967,4013979,4014016,4014431,4014515,4014636,4014671,4014810,4014813,4014909,4014991,4015119,4015155,4015234,4015260,4015444,4015497,4015519,4015550,4015589,4015638,4015639,4015642,4015644,4015646,4015685,4015772,4015812,4015853,4015900,4016007,4016031,4016127,4016309,4016323,4016326,4016331,4016367,4016373,4016452,4016456,4016466,4016479,4016487,4016623,4016643,4016703,4016730,4016851,4016902,4016969,8300015,8300017,8500165,8500223,8500239,8500252,8500256,8500283,8500306,10200002,10200012,10200013,10200029,10200038,10200040,10200045,10200048,10200053,10200055,10200066,10200083,10200103,10200120,10200134,10200136,10200155,10200157,10200159,10200178\",kCSI:{e:\"17259,4000116,4007661,4007830,4008067,4008133,4008142,4009033,4009565,4009641,4010806,4010858,4010899,4011228,4011258,4011679,4012373,4012504,4012508,4013374,4013414,4013591,4013723,4013747,4013757,4013787,4013823,4013967,4013979,4014016,4014431,4014515,4014636,4014671,4014810,4014813,4014909,4014991,4015119,4015155,4015234,4015260,4015444,4015497,4015519,4015550,4015589,4015638,4015639,4015642,4015644,4015646,4015685,4015772,4015812,4015853,4015900,4016007,4016031,4016127,4016309,4016323,4016326,4016331,4016367,4016373,4016452,4016456,4016466,4016479,4016487,4016623,4016643,4016703,4016730,4016851,4016902,4016969,8300015,8300017,8500165,8500223,8500239,8500252,8500256,8500283,8500306,10200002,10200012,10200013,10200029,10200038,10200040,10200045,10200048,10200053,10200055,10200066,10200083,10200103,10200120,10200134,10200136,10200155,10200157,10200159,10200178\",ei:\"NpJGU4rKMeWCiQe46IGABA\"},authuser:0,ml:function(){},kHL:\"en\",time:function(){return(new Date).getTime()},log:function(a,b,c,h,k){var d=\nnew Image,f=google.lc,e=google.li,g=\"\";d.onerror=d.onload=d.onabort=function(){delete f[e]};f[e]=d;c||-1!=b.search(\"&ei=\")||(g=\"&ei=\"+google.getEI(h));c=c||\"/\"+(k||\"gen_204\")+\"?atyp=i&ct=\"+a+\"&cad=\"+b+g+\"&zx=\"+google.time();a=/^http:/i;a.test(c)&&google.https()?(google.ml(Error(\"GLMM\"),!1,{src:c}),delete f[e]):(d.src=c,google.li=e+1)},lc:[],li:0,y:{},x:function(a,b){google.y[a.id]=[a,b];return!1},load:function(a,b,c){google.x({id:a+l++},function(){google.load(a,b,c)})}};var l=0;})();\n(function(){google.sn=\"webhp\";google.timers={};google.startTick=function(a,b){google.timers[a]={t:{start:google.time()},bfr:!!b}};google.tick=function(a,b,g){google.timers[a]||google.startTick(a);google.timers[a].t[b]=g||google.time()};google.startTick(\"load\",!0);\ntry{}catch(d){}})();\nvar _gjwl=location;function _gjuc(){var a=_gjwl.href.indexOf(\"#\");if(0<=a&&(a=_gjwl.href.substring(a),0#gbar,#guser{font-size:13px;padding-top:1px !important;}#gbar{height:22px}#guser{padding-bottom:7px !important;text-align:right}.gbh,.gbd{border-top:1px solid #c9d7f1;font-size:1px}.gbh{height:0;position:absolute;top:24px;width:100%}#media all{.gb1{height:22px;margin-right:.5em;vertical-align:top}#gbar{float:left}}a.gb1,a.gb4{text-decoration:underline !important}a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27 !important}.gbf .gb4{color:#900 !important}body,td,a,p,.h{font-family:arial,sans-serif}body{margin:0;overflow-y:scroll}#gog{padding:3px 8px 0}td{line-height:.8em}.gac_m td{line-height:17px}form{margin-bottom:20px}.h{color:#36c}.q{color:#00c}.ts td{padding:0}.ts{border-collapse:collapse}em{font-weight:bold;font-style:normal}.lst{height:25px;width:496px}.gsfi,.lst{font:18px arial,sans-serif}.gsfs{font:17px arial,sans-serif}.ds{display:inline-box;display:inline-block;margin:3px 0 4px;margin-left:4px}input{font-family:inherit}a.gb1,a.gb2,a.gb3,a.gb4{color:#11c !important}body{background:#fff;color:black}a{color:#11c;text-decoration:none}a:hover,a:active{text-decoration:underline}.fl a{color:#36c}a:visited{color:#551a8b}a.gb1,a.gb4{text-decoration:underline}a.gb3:hover{text-decoration:none}#ghead a.gb2:hover{color:#fff !important}.sblc{padding-top:5px}.sblc a{display:block;margin:2px 0;margin-left:13px;font-size:11px}.lsbb{background:#eee;border:solid 1px;border-color:#ccc #999 #999 #ccc;height:30px}.lsbb{display:block}.ftl,#fll a{display:inline-block;margin:0 12px}.lsb{background:url(/images/srpr/nav_logo80.png) 0 -258px repeat-x;border:none;color:#000;cursor:pointer;height:30px;margin:0;outline:0;font:15px arial,sans-serif;vertical-align:top}.lsb:active{background:#ccc}.lst:focus{outline:none}#addlang a{padding:0 3px}(function(){var src='/images/nav_logo176.png';var iesg=false;document.body.onload = function(){window.n && window.n();if (document.images){new Image().src=src;}\nif (!iesg){document.f&&document.f.q.focus();document.gbqf&&document.gbqf.q.focus();}\n}\n})(); Search http://www.google.com/imghp?hl=en&tab=wi\">Images http://maps.google.com/maps?hl=en&tab=wl\">Maps https://play.google.com/?hl=en&tab=w8\">Play http://www.youtube.com/?tab=w1\">YouTube http://news.google.com/nwshp?hl=en&tab=wn\">News https://mail.google.com/mail/?tab=wm\">Gmail https://drive.google.com/?tab=wo\">Drive http://www.google.com/intl/en/options/\">More »http://www.google.com/history/optout?hl=en\" class=gb4>Web History | Settings | https://accounts.google.com/ServiceLogin?hl=en&continue=http://www.google.com/\" class=gb4>Sign in Advanced searchLanguage toolsAdvertising ProgramsBusiness Solutionshttps://plus.google.com/116899029375914044550\" rel=\"publisher\">+GoogleAbout Google© 2013 - Privacy & Termsif(google.y)google.y.first=[];(function(){function b(a){window.setTimeout(function(){var c=document.createElement(\"script\");c.src=a;document.getElementById(\"xjsd\").appendChild(c)},0)}google.dljp=function(a){google.xjsu=a;b(a)};google.dlj=b;})();\nif(!google.xjs){window._=window._||{};window._._DumpException=function(e){throw e};if(google.timers&&google.timers.load.t){google.timers.load.t.xjsls=new Date().getTime();}google.dljp('/xjs/_/js/k\\x3dxjs.hp.en_US.75bv2nh_qxI.O/m\\x3dsb_he,pcc/rt\\x3dj/d\\x3d1/sv\\x3d1/rs\\x3dAItRSTOX5WMsAVpkgEafYeVKZ7ZCJdNXcg');google.xjs=1;}google.pmc={\"sb_he\":{\"agen\":true,\"cgen\":true,\"client\":\"heirloom-hp\",\"dh\":true,\"ds\":\"\",\"eqch\":true,\"fl\":true,\"host\":\"google.com\",\"jam\":0,\"jsonp\":true,\"msgs\":{\"dym\":\"Did you mean:\",\"lcky\":\"I\\u0026#39;m Feeling Lucky\",\"lml\":\"Learn more\",\"oskt\":\"Input tools\",\"psrc\":\"This search was removed from your \\u003Ca href=\\\"/history\\\"\\u003EWeb History\\u003C/a\\u003E\",\"psrl\":\"Remove\",\"sbit\":\"Search by image\",\"srch\":\"Google Search\"},\"ovr\":{},\"pq\":\"\",\"qcpw\":false,\"scd\":10,\"sce\":5,\"stok\":\"nXO_Z2rNte7VdOqPtwZ_V-hNE4c\"},\"pcc\":{}};google.y.first.push(function(){if(google.med){google.med('init');google.initHistory();google.med('history');}});if(google.j&&google.j.en&&google.j.xi){window.setTimeout(google.j.xi,0);}(function(){if(google.timers&&google.timers.load.t){var b,c,d,e,g=function(a,f){a.removeEventListener?(a.removeEventListener(\"load\",f,!1),a.removeEventListener(\"error\",f,!1)):(a.detachEvent(\"onload\",f),a.detachEvent(\"onerror\",f))},h=function(a){e=(new Date).getTime();++c;a=a||window.event;a=a.target||a.srcElement;g(a,h)},k=document.getElementsByTagName(\"img\");b=k.length;for(var l=c=0,m;l", :debug_info=>#}, #request=#:post}, #options={:method=>:post, :headers=>{"User-Agent"=>"Typhoeus - https://github.com/typhoeus/typhoeus"}, :maxredirs=>50}, #on_headers=[], #response=#, #on_complete=[], #on_success=[]>
Universal Analytics Measurement Protocol 200 - Nothing shows up
Like the title says, nothing shows up in real time events nor in any other event
The request being sent is done with ruby's faraday gem
require 'faraday'
require 'json'
GOOGLE_ANALYTICS_SETTINGS = {}
class GoogleAnalyticsApi
def event(client_id = '555', category, action, label, value, user_agent)
return unless !GOOGLE_ANALYTICS_SETTINGS["tracking_code"].empty?
params = {
v: GOOGLE_ANALYTICS_SETTINGS["version"],
tid: GOOGLE_ANALYTICS_SETTINGS["tracking_code"],
cid: client_id,
t: "event",
ec: category,
ea: action,
el: label,
ev: value
}
begin
puts params
c = Faraday.new() do |f|
f.request :url_encoded
f.response :logger
f.adapter Faraday.default_adapter
end
response = c.post GOOGLE_ANALYTICS_SETTINGS["endpoint"], params, { "User-Agent" => user_agent || "Subscribing Worker theSkimm" }
puts response.headers
puts response.body
return true
rescue Exception => rex
return false
end
end
end
From the console:
{:v=>1, :tid=>"UA-XXXXXXXX-1", :cid=>"1999999999.1389999972", :t=>"event", :ec=>"test-event", :ea=>"test-action", :el=>nil, :ev=>nil}
I, [2014-01-15T18:04:04.555555 #7666] INFO -- : post http://www.google-analytics.com/collect
D, [2014-01-15T18:04:04.555667 #7666] DEBUG -- request: User-Agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36"
Content-Type: "application/x-www-form-urlencoded"
I, [2014-01-15T18:04:04.670273 #7666] INFO -- Status: 200
D, [2014-01-15T18:04:04.670512 #7666] DEBUG -- response: pragma: "no-cache"
expires: "Mon, 07 Aug 1995 23:30:00 GMT"
cache-control: "private, no-cache, no-cache=Set-Cookie, proxy-revalidate"
access-control-allow-origin: "*"
last-modified: "Sun, 17 May 1998 03:00:00 GMT"
x-content-type-options: "nosniff"
content-type: "image/gif"
date: "Wed, 15 Jan 2014 23:04:04 GMT"
server: "Golfe2"
content-length: "35"
alternate-protocol: "80:quic"
connection: "close"
{"pragma"=>"no-cache", "expires"=>"Mon, 07 Aug 1995 23:30:00 GMT", "cache-control"=>"private, no-cache, no-cache=Set-Cookie, proxy-revalidate", "access-control-allow-origin"=>"*", "last-modified"=>"Sun, 17 May 1998 03:00:00 GMT", "x-content-type-options"=>"nosniff", "content-type"=>"image/gif", "date"=>"Wed, 15 Jan 2014 23:04:04 GMT", "server"=>"Golfe2", "content-length"=>"35", "alternate-protocol"=>"80:quic", "connection"=>"close"}
GIF89a�����,D;
For whomever encounters this the problem was setting nil value for el and ev. When the params hash is moved through Faraday the nil valued items get converted to the key name without any value and google didn't like that. IE a query string with nil values going through Faraday would be: ...&el&ev Not good for google's Measurement Protocol.
Unable to create s3 bucket in EU region
body = "<CreateBucketConfiguration><LocationConstraint>EU</LocationConstraint></CreateBucketConfiguration>"
content_length = body.bytesize
content_type = "text/plain"
url = URI.parse("http://#{#name}.s3.amazonaws.com/")
req = Net::HTTP::Put.new(url.path)
req.body = body
req.add_field 'Date' , #time
req.add_field 'Host', "#{#name}.s3.amazonaws.com"
req.add_field 'Content-Type', "#{content_type}"
req.add_field 'Authorization', "#{signature}"
req.add_field 'Content-Length', "#{content_length}"
response = Net::HTTP.new(url.host, url.port).start do |http|
http.request(req)
end
puts response.read_body
returns 200 and creates bucket but in U.S Standard and not in EU. What am I missing here? Thanks for the help.
Here is the entire conversation
PUT / HTTP/1.1
Accept: */*
User-Agent: Ruby
Date: Wed, 19 Jan 2011 22:14:31 -0800
Host: mytest.s3.amazonaws.com
Content-Type: text/plain
Authorization: AWS AC8RVKAXAU8Q:41uTqvfncc2mE561YabgpGUouio=
Content-Length: 146
<CreateBucketConfiguration xmlns='http://s3.amazonaws.com/doc/2006-03-01/'>
<LocationConstraint>EU</LocationConstraint>
</CreateBucketConfiguration>
HTTP/1.1 200 OK
x-amz-id-2: lrlPt8Y19ZxFXPbZf9Gf6dYxTGLYkkMzo0tSNXCNk29o9xghcob502mcttQ/oo4W
x-amz-request-id: 3504CCA0E7AFFE95
Date: Thu, 20 Jan 2011 06:14:32 GMT
Location: /mytest
Content-Length: 0
Server: AmazonS3
HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Date: Thu, 20 Jan 2011 06:14:32 GMT
Connection: close
Server: AmazonS3
0
The only thing that I can see is that you haven't included the xmlns in the request body - not sure if that will make any difference though. xmlns="http://s3.amazonaws.com/doc/2006-03-01/" body = "<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>EU</LocationConstraint></CreateBucketConfiguration>"