I am building a scraping tool based on ferrum (CDP tool) and I cannot set properly a new user-agent.
User-agent appears in the headers but is not effective. (I think the default_user_agent is not properly overridden).
browser = Ferrum::Browser.new
new_user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_0)"
headers = {
"User-Agent" => new_user_agent,
"default_user_agent" => new_user_agent
}
browser.headers.add(headers)
Does anyone has an example of a successful setup ?
It is working with:
browser.headers.set({"User-Agent" => new_user_agent})
It is possible to pass many headers :
browser.headers.set({"User-Agent" => new_user_agent, "Accept-Language" => "fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7"})
Related
I'm sending hits to GA Measurement Protocol, and some of them do not make it to the GA. I've noticed that all of them have one thing in common: the user-agent is Firefox, only varying version and device. Some examples:
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:103.0) Gecko/20100101 Firefox/103.0
Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0
Mozilla/5.0 (Android 10; Mobile; rv:103.0) Gecko/103.0 Firefox/103.0
GA validator is OK with those examples when checking them through the debug mode like this:
https://www.google-analytics.com/debug/collect?v=1&tid=UA-XXXXXXXX-1&t=event&ec=Ecommerce&ea=purchase&pa=purchase&cid=1234567890.1234567890&ni=1&ti=184242&tr=1060&uip=X.X.X.X&ua=Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%3B+rv%3A103.0%29+Gecko%2F20100101+Firefox%2F103.0&pr1id=test_1&pr1pr=530&pr1qt=1&pr1ps=1
I get this response:
{
"hitParsingResult": [ {
"valid": true,
"parserMessage": [ ],
"hit": "/debug/collect?v=1..."
} ],
"parserMessage": [ {
"messageType": "INFO",
"description": "Found 1 hit in the request."
} ]
}
BUT in the production settings GA responses with 400 bad request error to the same requests without providing any details: "Your client has issued a malformed or illegal request. That’s all we know.".
So what might be wrong with Firefox UA?
UPD: I've managed to make this work by unsetting the 'User-Agent' header in case it contains 'Firefox' - and the corresponding 'ua' parameter in the payload gets accepted then.
if (strpos($requestHeaders['User-Agent'], 'Firefox') !== false) {
unset($requestHeaders['User-Agent']);
}
But it's still unclear what was wrong with such headers in the first place.
I am trying hit an internal testing API server using RestClient and Ruby v. 2.2.1.
This is essentially the code:
url = "https://10.10.0.10/thing/i/want/to/get"
header = {
:content_type => "application/json",
:"x-auth-token" => "testingtoken"
}
response = RestClient.get url, header
This is the failure message I get:
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (RestClient::SSLCertificateNotVerified)
If I'm reading this right, it looks like Ruby couldn't accept the SSL security certificate. This call works in the Chrome app Postman, but in order for it to work, I have to hit the URL in Chrome itself and accept that the connection is not secure (but proceed anyway), and THEN it will work in postman.
Is there a way to ignore the certificate failures and proceed anyway in Ruby?
Try using #execute(&block) with verify_ssl set to false.
:verify_ssl enable ssl verification, possible values are constants
from OpenSSL::SSL::VERIFY_*, defaults to OpenSSL::SSL::VERIFY_PEER
url = "https://10.10.0.10/thing/i/want/to/get"
headers = {
:content_type => "application/json",
:"x-auth-token" => "testingtoken"
}
RestClient::Request.execute(
:url => url,
:method => :get,
:headers => headers,
:verify_ssl => false
)
see: http://www.rubydoc.info/github/rest-client/rest-client/RestClient/Request#execute-instance_method
RVM
Additional solution for RVM users from: https://toadle.me/2015/04/16/fixing-failing-ssl-verification-with-rvm.html
This discussion on Github finally gave the solution: Somehow RVM comes
with a precompiled version of ruby that is statically linked against
an openssl that looks into /etc/openssl for it's certificates.
What you wanna do is NOT TO USE any of the precompiled rubies and
rather have ruby compiled on your local machine, like so:
rvm install 2.2.0 --disable-binary
rest-client verify certificates using the system's CA store on all platforms by default. But is possible set to false the option :verify_ssl or specify :ssl_ca_file or :ssl_ca_path or :ssl_cert_store to customize the certificate authorities accepted.
See documentation
So you could simply set :verify_ssl to false:
url = "https://10.10.0.10/thing/i/want/to/get"
header = {
:content_type => "application/json",
:"x-auth-token" => "testingtoken"
}
resource = RestClient::Resource.new(
url,
headers: header,
verify_ssl: false
)
response = resource.get
You could try immediately with a host which use a self-signed certificated provided by https://badssl.com/. Simply copy the snippet below in your irb console.
response = RestClient::Resource.new(
'https://self-signed.badssl.com/',
:verify_ssl => false
).get
I am developing a react-native application and I am using Couchbase lite as a database locally. The way this works is that you spawn a local REST server when the app starts and you use the REST API to communicate with the CouchbaseLite server.
I have created a few design documents, but when I try to update those I do not get the new results when I run my REST client (seperate app I use for debugging). When I GET the design document it has a new _rev after the update, the new map function is as I updated it, but whenever I do a get on the view the result is the same as the first version of the map function.
Apparently the updated docs are not used by get.
The design doc:
var designDoc = {
name: 'expenses',
language: 'javascript',
views: {
contact_parts_for_group: {
'map': function(doc){
if(doc.type == 'expense'){
emit('some things I emit', doc.amount)
}
}.toString()
}
}
};
I send this to the server along with the proper _rev as the json body: JSON.stringify(designDoc)
.1. I am updating my design document with a PUT call:
PUT /kittydb/_design/expenses?rev=4-6f89f1e13d1fbb89c712d6bab53ee7d4 HTTP/1.1
Host: 127.0.0.1:5800
Connection: close
User-Agent: Paw/2.2.2 (Macintosh; OS X/10.11.2) GCDHTTPRequest
Content-Length: 356
{"name":"expenses","language":"javascript","views":{"contact_parts_for_group":{"map":"function (doc){ if(doc.type=='expense'){ var i,len,part,ref; ref=doc.parts; for(i=0,len=ref.length;i<len;i++){ part=ref[i]; var amount=part.contact==doc.expense_by?-1*part.amount:part.amount; emit([doc.group_id,part.contact,part.contact==doc.expense_by],amount);}}}"}}}
.2. I populate the database using the interface of the app prototype I developed so far
.3. I am not sure what you mean by this.
.4. This is the get:
GET /kittydb/_design/expenses/_view/contact_parts_for_group HTTP/1.1
Host: 127.0.0.1:5800
Connection: close
User-Agent: Paw/2.2.2 (Macintosh; OS X/10.11.2) GCDHTTPRequest
More information in reaction to some comments:
I am using the CouchbaseLite Community Edition, version 1.1.1 for iOS. I am running the simulator as an iPhone 6 with iOS 9.2.
I made some screenshots to illustrate what is going on a bit more:
I don't know how to retrieve the map function that goes with this but what it seems to do is:
emit([doc.group_id,part.contact],amount)
I used the get as above.
Now my update:
PUT /kittydb/_design/expenses?rev=7-6f979706f38acce9c7db380fba8565e4 HTTP/1.1
Host: 127.0.0.1:5800
Connection: close
User-Agent: Paw/2.2.2 (Macintosh; OS X/10.11.2) GCDHTTPRequest
Content-Length: 350
{
"name": "expenses",
"language": "javascript",
"views": {
"contact_parts_for_group": {
"map": "function (doc){ if(doc.type=='expense'){ var i,len,part,ref; ref=doc.parts; for(i=0,len=ref.length;i<len;i++){ part=ref[i]; var amount=part.contact==doc.expense_by?-1*part.amount:part.amount; emit('Hello SO', 'Overflow');}}}"
}
}
}
What it should do now is: emit('Hello SO', 'Overflow');
I get this response when I run the above request:
HTTP/1.1 201 Created
Location: http://127.0.0.1:5800/kittydb/_design/expenses
Content-Type: application/json
Server: CouchbaseLite 1.1 (unofficial)
Etag: "8-3ae4b6ff37b936657ca23acb8d836619"
Accept-Ranges: bytes
Date: Wed, 20 Jan 2016 21:57:03 GMT
Transfer-Encoding: chunked
{"id":"_design\/expenses","rev":"8-3ae4b6ff37b936657ca23acb8d836619","ok":true}
Now I run the get request again:
And nothing changed...
When I create a new document with 'type = expense' I get the same result, just more of them.
I don't know how to retrieve the map function that goes with this
Aha -- if you don't know where the original view definition is, and you can't get it from the design document, it's probably being defined in native code (at app launch time.) Such a definition will override one in a design document.
I don't know anything about React-Native. Is there (as the name implies) native code in the app? If so, look for a call to [CBLView setMapBlock: ...].
I'm trying to integrate Sinch into my ROR webapp, and am having some difficulty formatting the signedUserToken to start the sinchClient.
Here is my view, using haml :
#{#signedUserTicket}
%script{src: "//cdn.sinch.com/latest/sinch.min.js", type: "text/javascript"}
= javascript_tag do
$(function(){
$sinchClient = new SinchClient({
applicationKey: 'APP_KEY',
capabilities: {messaging: true, calling: true},
supportActiveConnection: true,
onLogMessage: function(message) {
console.log(message);
},
});
$sinchClient.start({
'userTicket' : "#{#signedUserTicket}",
});
});
And whatever formatting I try to do in the controller, the closest I get to succeeding is :
DOMException [InvalidCharacterError: "String contains an invalid character"
code: 5
nsresult: 0x80530005
location: http://cdn.sinch.com/latest/sinch.min.js:5]
I'd appreciate a little help and would even build a Rubygem for integrating Sinch in Rails if I get the right info and can spare some time.
Cheers,
James
Edit :
I have tried a few modifications and am getting closer (I think).
The problem of InvalidCharacter came from the trailing '='s which apparently don't decode well in Javascript.
My new controller is now :
class SinchController < ApplicationController
skip_before_filter :verify_authenticity_token
before_filter :authenticate_user!
def client
username = current_user.username
applicationKey = "APP_KEY"
applicationSecret = "APP_SECRET_B64"
userTicket = {
"identity" => {"type" => "username", "endpoint" => username},
"expiresIn" => 3600,
"applicationKey" => applicationKey,
"created" => Time.now.utc.iso8601
}
userTicketJson = userTicket.to_json
userTicketBase64 = Base64.strict_encode64(userTicketJson).chop
digest = Digest::HMAC.digest(Base64.decode64(applicationSecret), userTicketJson, Digest::SHA256)
signature = Base64.strict_encode64(digest).chop
#signedUserTicket = (userTicketBase64 + ':' + signature).remove('=')
end
end
But now I'm facing the following error:
POST https://api.sinch.com/v1/instance 500 (Internal Server Error)
client:1 XMLHttpRequest cannot load https://api.sinch.com/v1/instance. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http:// localhost:3000' is therefore not allowed access. The response had HTTP status code 500.
(the space before localhost is due to new user restrictions on SO)
I added Rack::Cors to my rails server to try and allow Cross-domain requests in case it came from my own requests, but whatever configuration I tried, it seems the request never contains the right headers.
Am I misunderstanding CORS requests? Does the problem come from the requests generated by sinch.min.js?
Regards,
James
Error message is due to Firefox base64 decoder can't decode the token, due to symbols (such as #) that are not in the base64 character set. This suggest that the ticket is actually not passed to start(), and this line may be incorrect;
'userTicket' : "#{#signedUserTicket}",
I dont know HAML but shouldnt
'userTicket' : "#{#signedUserTicket}",
be 'userTicket' : #signedUserTicket,
I want to run tests that change the user-agent in the http request sent from the browser (like the FF add-on, user agent switcher does). I saw you can do it by playing with the FF profile (http://seleniumhq.org/docs/09_webdriver.html).
Is there a way to do it within a test? Something like the function addCustomRequestHeader() that sets a header rather than adding it
You could insert a function like this to change the user agent on the fly before you make your http request:
function changeuserAgent() {
var altuserAgentGetter = function () {
return "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 <choose your string>";
};
if (Object.defineProperty) {
Object.defineProperty(navigator, "userAgent", {
get: altuserAgentGetter
});
}
else if (Object.prototype.__defineGetter__) {
navigator.__defineGetter__("userAgent", altuserAgentGetter);
}
}
If you're using the Selenium 2 Web Driver in Java, you can create a Firefox profile and set the agent string as a preference in the profile. Then use the profile to create the WebDriver object:
FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("general.useragent.override", "Mozilla/5.0 (iPad; U; CPU OS 4_3 like Mac OS X; de-de) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F191 Safari/6533.18.5");
WebDriver driver = new FirefoxDriver(profile);
For slightly more information and source code examples, see the Selenium Web Driver documentation for Firefox Driver at http://seleniumhq.org/docs/03_webdriver.html#firefox-driver.