I'm trying to decode a cookie and the interpreter is escaping backslashes that I need to remain intact.
This works
encrypted = %Q{"pgmQKWFl3ZqLWjMR6HISwjzbTSbF3qNj/xABLuR7sTRhG4hhVFDUBhrWLpHyXjOR0I9UB8zpzyOT\012J0RDv0s9QcJgm5kx0+pPZFmIt/dfv5LH0UIYvEmODnbfqpzjV7uIgpoi+fdVJQaHwk1GF5W1F3hg\0122pWmc/aDz8saLG4j5I4U+cctEo7uUOOKn2xRszlytNIOylFQramG1UKC6TLFrkXjz24/MEfEPbaO\012pHa2dzYbKN+2AOACVrspysnnAeG0W97EvaV9Q3OXdIY/9eElpLzrbgf+4weTG/qRTd7iSjautF0G\0129KMpzusxQxEI6hSrq+aYIBHuyUVMBkhGzobPjw==\012"}
=> "\"pgmQKWFl3ZqLWjMR6HISwjzbTSbF3qNj/xABLuR7sTRhG4hhVFDUBhrWLpHyXjOR0I9UB8zpzyOT\nJ0RDv0s9QcJgm5kx0+pPZFmIt/dfv5LH0UIYvEmODnbfqpzjV7uIgpoi+fdVJQaHwk1GF5W1F3hg\n2pWmc/aDz8saLG4j5I4U+cctEo7uUOOKn2xRszlytNIOylFQramG1UKC6TLFrkXjz24/MEfEPbaO\npHa2dzYbKN+2AOACVrspysnnAeG0W97EvaV9Q3OXdIY/9eElpLzrbgf+4weTG/qRTd7iSjautF0G\n9KMpzusxQxEI6hSrq+aYIBHuyUVMBkhGzobPjw==\n\""
But when I hit the server with that as a cookie
http localhost:9393 Cookie:cookie="\"pgmQKWFl3ZqLWjMR6HISwjzbTSbF3qNj/xABLuR7sTRhG4hhVFDUBhrWLpHyXjOR0I9UB8zpzyOT\nJ0RDv0s9QcJgm5kx0+pPZFmIt/dfv5LH0UIYvEmODnbfqpzjV7uIgpoi+fdVJQaHwk1GF5W1F3hg\n2pWmc/aDz8saLG4j5I4U+cctEo7uUOOKn2xRszlytNIOylFQramG1UKC6TLFrkXjz24/MEfEPbaO\npHa2dzYbKN+2AOACVrspysnnAeG0W97EvaV9Q3OXdIY/9eElpLzrbgf+4weTG/qRTd7iSjautF0G\n9KMpzusxQxEI6hSrq+aYIBHuyUVMBkhGzobPjw==\n\""
and try to retrieve the cookie from my app, it escapes the backslashes and turns + into spaces
encrypted_string = request.cookies['cookie']
"\"pgmQKWFl3ZqLWjMR6HISwjzbTSbF3qNj/xABLuR7sTRhG4hhVFDUBhrWLpHyXjOR0I9UB8zpzyOT\\nJ0RDv0s9QcJgm5kx0 pPZFmIt/dfv5LH0UIYvEmODnbfqpzjV7uIgpoi fdVJQaHwk1GF5W1F3hg\\n2pWmc/aDz8saLG4j5I4U cctEo7uUOOKn2xRszlytNIOylFQramG1UKC6TLFrkXjz24/MEfEPbaO\\npHa2dzYbKN 2AOACVrspysnnAeG0W97EvaV9Q3OXdIY/9eElpLzrbgf 4weTG/qRTd7iSjautF0G\\n9KMpzusxQxEI6hSrq aYIBHuyUVMBkhGzobPjw==\\n\""
string = key.private_decrypt(Base64.decode64(encrypted))
OpenSSL::PKey::RSAError: padding check failed
What am I missing? Thanks for your help.
For now, I'm just using gsub.
gsub("\\012","").gsub(" ","+")
I have tried below code. According to that it perform decoding on cookies you are passing and produce same result.
require ‘uri’
Say your cookie is:
c = “cookie=pgmQKWFl3ZqLWjMR6HISwjzbTSbF3qNj/xABLuR7sTRhG4hhVFDUBhrWLpHyXjOR0I9UB8zpzyOT\nJ0RDv0s9QcJgm5kx0+pPZFmIt/dfv5LH0UIYvEmODnbfqpzjV7uIgpoi+fdVJQaHwk1GF5W1F3hg\n2pWmc/aDz8saLG4j5I4U+cctEo7uUOOKn2xRszlytNIOylFQramG1UKC6TLFrkXjz24/MEfEPbaO\npHa2dzYbKN+2AOACVrspysnnAeG0W97EvaV9Q3OXdIY/9eElpLzrbgf+4weTG/qRTd7iSjautF0G\n9KMpzusxQxEI6hSrq+aYIBHuyUVMBkhGzobPjw==\n”
URI.decode_www_form(c).to_h
Output will be:
{"cookie"=>"pgmQKWFl3ZqLWjMR6HISwjzbTSbF3qNj/xABLuR7sTRhG4hhVFDUBhrWLpHyXjOR0I9UB8zpzyOT\nJ0RDv0s9QcJgm5kx0 pPZFmIt/dfv5LH0UIYvEmODnbfqpzjV7uIgpoi fdVJQaHwk1GF5W1F3hg\n2pWmc/aDz8saLG4j5I4U cctEo7uUOOKn2xRszlytNIOylFQramG1UKC6TLFrkXjz24/MEfEPbaO\npHa2dzYbKN 2AOACVrspysnnAeG0W97EvaV9Q3OXdIY/9eElpLzrbgf 4weTG/qRTd7iSjautF0G\n9KMpzusxQxEI6hSrq aYIBHuyUVMBkhGzobPjw==\n"}
See value of cookie key in hash.
pgmQKWFl3ZqLWjMR6HISwjzbTSbF3qNj/xABLuR7sTRhG4hhVFDUBhrWLpHyXjOR0I9UB8zpzyOT\nJ0RDv0s9QcJgm5kx0 pPZFmIt/dfv5LH0UIYvEmODnbfqpzjV7uIgpoi fdVJQaHwk1GF5W1F3hg\n2pWmc/aDz8saLG4j5I4U cctEo7uUOOKn2xRszlytNIOylFQramG1UKC6TLFrkXjz24/MEfEPbaO\npHa2dzYbKN 2AOACVrspysnnAeG0W97EvaV9Q3OXdIY/9eElpLzrbgf 4weTG/qRTd7iSjautF0G\n9KMpzusxQxEI6hSrq aYIBHuyUVMBkhGzobPjw==\n
You can see + is replaced with space.
Perform gsub(‘ ’, ‘+’) on cookie you received on server end.
Related
I have set up the rack-attack config per the advanced configuration instructions. I am using Heroku and have confirmed the env variable contains all of the urls and everything is properly formatted.
I have even gone into the console on Heroku and run the following:
req = Rack::Attack::Request.new({'HTTP_REFERER' => '4webmasters.org'})
and then tested with:
Rack::Attack.blacklisted?(req)
to which I get:
=> true
but in my analytics on google the referrals are filled with every url on my list. What am I missing?
My config includes this pretty standard block:
# Split on a comma with 0 or more spaces after it.
# E.g. ENV['HEROKU_VARIABLE'] = "foo.com, bar.com"
# spammers = ["foo.com", "bar.com"]
spammers = ENV['HEROKU_VARIABLE'].split(/,\s*/)
#
# Turn spammers array into a regexp
spammer_regexp = Regexp.union(spammers) # /foo\.com|bar\.com/
blacklist("block referer spam") do |request|
request.referer =~ spammer_regexp
end
#
HEROKU_VARIABLE =>
"ertelecom.ru, 16clouds.com, bee.lt, belgacom.be, virtua.com.br, nodecluster.net, telesp.net.br, belgacom.be, veloxzone.com.br, baidu.com, floating-share-buttons.com, 4webmasters.org, trafficmonetizer.org, webmonetizer.net, success-seo.com, buttons-for-website.com, videos-for-your-business.com, Get-Free-Traffic-Now.com, 100dollars-seo.com, e-buyeasy.com, free-social-buttons.com, traffic2money.com, erot.co, success-seo.com, semalt.com"
These types of referrers are Google Analytic spam referrers. They never actually hit your website so blocking them with rack-attack is pointless. The data you see from them in GA is all fake. To stop this in your GA, set up a filter to ignore visits from that referrer.
I'm using the google custom search api and I'm trying to access it through some ruby code:
Here is a snippet of the code
req = Typhoeus::Request.new("https://www.googleapis.com/customsearch/v1?key={my_key}&cx=017576662512468239146:omuauf_lfve&q=" + keyword, followlocation: true)
res = req.run
It appears that the body of the answer is this one:
<p>Your client has issued a malformed or illegal request. <ins>That’s all we know.</ins>
'
from /usr/local/lib/ruby/2.1.0/json/common.rb:155:in `parse'
from main.rb:20:in `initialize'
from main.rb:41:in `new'
from main.rb:41:in `<main>'
When I try to do the same thing from the browser it works like a charm. Even more confusing is that this same code worked 12 hours ago. I only changed the keyword that it should look for, however it started returning the error.
Any suggestions? I'm sure that I have enough credits for more requests
You probably have problems with special characters in your get parameter keyword. If you enter the URL in your browser, the brower adjusts these. However, for ruby you need to escape these characters, in such a way that a string like "sky line" becomes "sky+line" and so on. There is a utility function CGI::escape, which is used like this:
require 'cgi'
CGI::escape("sky line")
=> "sky+line"
Your fixed code would look something like this:
req = Typhoeus::Request.new("https://www.googleapis.com/customsearch/v1?key={my_key}&cx=017576662512468239146:omuauf_lfve&q=" + CGI::escape(keyword), followlocation: true)
res = req.run
However, since you're using Typhoeus anyway, you should be able to use its params parameter and let Typhoeus handle the escaping:
req = Typhoeus::Request.new(
"https://www.googleapis.com/customsearch/v1?&cx=017576662512468239146:omuauf_lfve",
followlocation: true,
params: {q: keyword, key: my_key}
)
res = req.run
There's more examples on Typhoeus' GitHub page.
I am trying to get a cookie value but keep getting an error of <Selenium::WebDriver::Driver:0x13a0e0e8 browser=:firefox>
I am calling
#browser.cookie_named("configsession").each do |cookie|
puts cookie[:name]
is there something I i'm doing wrong?
The methods for working with cookies are defined in the Selenium::WebDriver::Options - see the API docs.
To access these cookie methods, you need to call the manage method for the driver:
#browser.manage
To get a cookie based on its name, you need to do:
#browser.manage.cookie_named("configsession")
Note that cookie_named returns a single cookie that matches. The cookies values are a hash. Therefore, you can get values of the cookie by doing:
cookie = #browser.manage.cookie_named("configsession")
cookie[:name]
#=> "configsession"
If you want to get the name of all the cookies on the page, use the all_cookies method:
driver.manage.all_cookies.each do |cookie|
puts cookie[:name]
end
This worked for me:
Cookie cookie= driver.manage().getCookieNamed("sitename.session");
String cookieVal= cookie.getValue();
Set<Cookie> cook = driver.manage().getCookies();
for(Cookie cooks : cook)
{
System.out.println(cooks.getName());
}
Cookie t = driver.manage().getCookieNamed("_gid");
if(t!=null){
String s1 = t.getValue();
System.out.println("The Cookie value is : " + s1);
}
I was going through the Ruby tutorials provided at http://ruby.bastardsbook.com/ and I encountered the following code:
require "open-uri"
remote_base_url = "http://en.wikipedia.org/wiki"
r1 = "Steve_Wozniak"
r2 = "Steve_Jobs"
f1 = "my_copy_of-" + r1 + ".html"
f2 = "my_copy_of-" + r2 + ".html"
# read the first url
remote_full_url = remote_base_url + "/" + r1
rpage = open(remote_full_url).read
# write the first file to disk
file = open(f1, "w")
file.write(rpage)
file.close
# read the first url
remote_full_url = remote_base_url + "/" + r2
rpage = open(remote_full_url).read
# write the second file to disk
file = open(f2, "w")
file.write(rpage)
file.close
# open a new file:
compiled_file = open("apple-guys.html", "w")
# reopen the first and second files again
k1 = open(f1, "r")
k2 = open(f2, "r")
compiled_file.write(k1.read)
compiled_file.write(k2.read)
k1.close
k2.close
compiled_file.close
The code fails with the following trace:
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:277:in `open_http': 403 Forbidden (OpenURI::HTTPError)
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:518:in `open'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:30:in `open'
from /Users/arkidmitra/tweetfetch/samecode.rb:11
My problem is not that the code fails but that whenever I change r2 to anything other than Steve_Jobs, it works. What is happening here?
Your code runs fine for me (Ruby MRI 1.9.3) when I request a wiki page that exists.
When I request a wiki page that does NOT exist, I get a mediawiki 404 error code.
Steve_Jobs => success
Steve_Austin => success
Steve_Rogers => success
Steve_Foo => error
Wikipedia does a ton of caching, so if you see reponses for "Steve_Jobs" that are different than other people who do exist, then best-guess this is because wikipedia is caching the Steve Jobs article because he's famous, and potentially adding extra checks/verifications to protect the article from rapid changes, defacings, etc.
The solution for you: always open the url with a User Agent string.
rpage = open(remote_full_url, "User-Agent" => "Whatever you want here").read
Details from the Mediawiki docs: "When you make HTTP requests to the MediaWiki web service API, be sure to specify a User-Agent header that properly identifies your client. Don't use the default User-Agent provided by your client library, but make up a custom header that includes the name and the version number of your client: something like "MyCuteBot/0.1".
On Wikimedia wikis, if you don't supply a User-Agent header, or you supply an empty or generic one, your request will fail with an HTTP 403 error. See our User-Agent policy."
I think this happens for locked down entries like "Steve Jobs", "Al-Gore" etc. This is specified in the same book that you are referring to:
For some pages – such as Al Gore's locked-down entry – Wikipedia will
not respond to a web request if a User-Agent isn't specified. The
"User-Agent" typically refers to your browser, and you can see this by
inspecting the headers you send for any page request in your browser.
By providing a "User-Agent" key-value pair, (I basically use "Ruby"
and it seems to work), we can pass it as a hash (I use the constant
HEADERS_HASH in the example) as the second argument of the method
call.
It is specified later at http://ruby.bastardsbook.com/chapters/web-crawling/
I am doing a video crawler in ruby. In there I have to log in to a page by enabling cookies and download pages. For that I am using the CURL library in ruby. I can successfully log in, but I can't download the pages inside that with curl. How can I fix this or download the pages otherwise?
My code is
curl = Curl::Easy.new(1st url)
curl.follow_location = true
curl.enable_cookies = true
curl.cookiefile = "cookie.txt"
curl.cookiejar = "cookie.txt"
curl.http_post(1st url,field)
curl.perform
curl = Curl::Easy.perform(2nd url)
curl.follow_location = true
curl.enable_cookies = true
curl.cookiefile = "cookie.txt"
curl.cookiejar = "cookie.txt"
curl.http_get
code = curl.body_str
What I've seen in writing my own similar "post-then-get" script is that ruby/Curb (I'm using version 0.7.15 with ruby 1.8) seems to ignore the cookiejar/cookiefile fields of a Curl::Easy object. If I set either of those fields and the http_post completes successfully, no cookiejar or cookiefile file is created. Also, curl.cookies will still be nil after your curl.http_post, however, the cookies ARE set within the curl object. I promise :)
I think where you're going wrong is here:
curl = Curl::Easy.perform(2nd url)
The curb documentation states that this creates a new object. That new object doesn't have any of your existing cookies set. If you change your code to look like the following, I believe it should work. I've also removed the curl.perform for the first url since curl.http_post already implicitly does the "perform". You were basically http_post'ing twice before trying your http_get.
curl = Curl::Easy.new(1st url)
curl.follow_location = true
curl.enable_cookies = true
curl.http_post(1st url,field)
curl.url = 2nd url
curl.http_get
code = curl.body_str
If this still doesn't seem to be working for you, you can verify if the cookie is getting set by adding
curl.verbose = true
Before
curl.http_post
Your Curl::Easy object will dump all the headers that it gets in the response from the server to $stdout, and somewhere in there you should see a line stating that it added/set a cookie. I don't have any example output right now but I'll try to post a follow-up soon.
HTTPClient automatically enables cookies, as does Mechanize.
From the HTTPClient docs:
clnt = HTTPClient.new
clnt.get_content(url1) # receives Cookies.
clnt.get_content(url2) # sends Cookies if needed.
Posting a form is easy too:
body = { 'keyword' => 'ruby', 'lang' => 'en' }
res = clnt.post(uri, body)
Mechanize makes this sort of thing really simple (It will handle storing the cookies, among other things).