I'm a noob to Cocoa programming, so please excuse the possible stupidity of this question.
I'm trying to build a web browser using Cocoa/MacRuby but am running into this strange rendering problem - a lot of sites render just fine, but all Google sites and a bunch of other sites (such as the CouchDB website) render with very small fonts.
I tried to mimic Firefox's HTTP request and set headers like so:
def initialize_request(url)
url_request = NSMutableURLRequest.requestWithURL(NSURL.URLWithString(url))
url_request.setValue("ISO-8859-1,utf-8;q=0.7,*;q=0.7", forHTTPHeaderField:"Accept-Charset")
url_request.setValue("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", forHTTPHeaderField:"Accept")
url_request.setValue("en-us,en;q=0.5", forHTTPHeaderField:"Accept-Language")
url_request.setValue("gzip,deflate", forHTTPHeaderField:"Accept-Encoding")
url_request.setValue("keep-alive", forHTTPHeaderField:"Connection")
url_request.setValue("300", forHTTPHeaderField:"Keep-Alive")
headers = url_request.allHTTPHeaderFields
NSLog("All headers: #{headers.inspect}")
url_request
end
Full source file is here
Any pointers will be helpful, thanks in advance!
Check the font size in your app's WebPreferences and the text-size multiplier on the WebView.
Related
How can I add .locahost to the list of TLDs that Safari will load — instead of searching — without supplying a protocol? Alternatively (or perhaps additionally), how do I get Safari to stop removing the http:// protocol from the URL?
I've been using project.localhost to handle local development for a while, now that .dev is no longer a viable development TLD. But I'm getting frustrated with Safari's default behavior. What I'd like to do is prevent Safari from submitting the domain name to the search engine.
Desired:
enter project.localhost into address bar
browser loads http://project.localhost
I would be satisfied with Safari not removing the http:// in the address bar, but I can't find a way to do that, either.
What actually happens:
enter project.localhost into address bar
browser sends project.localhost to Duck Duck Go
user is tempted to test the ballistic properties of keyboard
Or, after I've loaded http://project.localhost and then try to add something after the current URL:
click in address bar, URL has been shortened to project.localhost
add /test.html to end
browser sends project.localhost/test.html to Duck Duck Go
user considers software violence
If you just type the trailing '/', then the search will be bypassed.
Turns out that the Safari extension might be a long term solution. At this moment, I can't manage the developer membership cost. But here's the working code, for Future Me and anyone else looking into this.
In the extension's global page <script>:
(function() {
safari.application.addEventListener("beforeSearch", function(event) {
if (event.query.split("/")[0].split(".").pop() == "localhost") {
event.preventDefault();
safari.application.activeBrowserWindow.activeTab.url = "http://" + event.query;
}
}, true);
})();
I just found this question while looking for a solution to the same problem. Good news, there's an easy solution that I just stumbled upon after not finding an answer.
Open System Preferences
Open Network
Click Advanced button
Click Proxies tab
Then in the "Bypass proxy settings for these Hosts & Domains:" add in localhost to the list. Here's what I did with mine. I'm using .home for my local domains in my PiHole. Now if I type "router.home" into Safari my router's config page loads up.
I compiled my reactjs using webpack and got a bundle file bundles.js. My bundles.js contains a component that make API calls to get the data.
I put this file in my html and pass the url to phantom.js to pre-compile static html for SEO reasons.
I am witnessing something strange here, the ajax calls for APIS are not getting fired at all.
For example, I have a component called Home which is called when I request for url /home. My Home component makes an ajax request to backend (django-rest) to get some data. Now when I call home page in phantomjs this api call is not getting fired.
Am I missing something here?
I have been using React based app rendering in Phantomjs since 2014. Make sure you use the latest Phantomjs version v2.x. The problems with Phantomjs occur because it uses older webkit engine, so if you have some CSS3 features used make sure they are prefixed correctly example flexbox layout.
From the JS side the PhantomJS does not support many newer APIs (example fetch etc.), to fix this add the polyfills and your fine. The most complicated thing is to track down errors, use the console.log and evaluate code inside the Phantomjs. There is also debugging mode which is actually quite difficult to use, but this could help you track down complex errors. I used webkit engine based browser Aurora to track down some of the issues.
For debugging the network traffic, try logging the requested and received events:
var page = require('webpage').create();
page.onResourceRequested = function(request) {
console.log('Request ' + JSON.stringify(request, undefined, 4));
};
page.onResourceReceived = function(response) {
console.log('Receive ' + JSON.stringify(response, undefined, 4));
};
How do you change the user agent used by WKWebview?
With the older WebView, I could write the following to change the user agent:
[myWebView setCustomUserAgent:#"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4)
AppleWebKit/537.77.4 (KHTML,like Gecko) Version/7.0.5 Safari/537.77.4"];
Very simple in Swift. Just place the following into your App DelegatedidFinishLaunchingWithOptions.
NSUserDefaults.standardUserDefaults().registerDefaults(["UserAgent" : "Custom Agent"])
If you want to append to the existing agent string then:
let userAgent = UIWebView().stringByEvaluatingJavaScriptFromString("navigator.userAgent")! + " Custom Agent"
NSUserDefaults.standardUserDefaults().registerDefaults(["UserAgent" : userAgent])
Note: You will need to uninstall and reinstall the App to avoid appending to the existing agent string.
I don't have an answer for this. However, some pointers from my research so far:
In iOS, it's possible to set a custom user agent for a UIWebView like this:
NSDictionary *dictionary = [NSDictionary dictionaryWithObjectsAndKeys:agent, #"UserAgent", nil];
[[NSUserDefaults standardUserDefaults] registerDefaults:dictionary];
In OSX, there was a setCustomUserAgent method for WebView elements that did the trick.
However, this doesn't work for WKWebView (at least, in OSX). I couldn't find any documentation about it from Apple, either.
Hope somebody can help!
I ran into the same issue, but managed to work around it using a combination of loadHTMLString on the WKWebView and a NSMutableURLRequest to do the heavy lifting.
My search on how to call some method on the WKWebView itself lead me to http://trac.webkit.org/changeset/165594, which implies there is a private method _setCustomUserAgent to do this. I'm not proficient enough in cocoa/swift to figure this one out.
I ended up using the code below, as I really only need to fetch the contents of a single URL and display it, but it may be helpful in some way.
What it does is simply loading the contents of an URL into the WKWebView as string, I suspect you may lose back/forward navigation and such, and it will only work for the initial page display, as the WKWebView will take over clicks and asset loading.
(please note, this example is written in Swift and not Objective-C)
self.webView = WKWebView(frame: webViewRect, configuration: webViewConfig)
// create the request
let url = NSURL(string: "https://example.com/")
let request = NSMutableURLRequest(URL: url!)
request.setValue("YourUserAgent/1.0", forHTTPHeaderField: "User-Agent")
NSURLConnection.sendAsynchronousRequest(request, queue: NSOperationQueue.mainQueue()) {(response, data, error) in
let content = NSString(data: data, encoding: NSUTF8StringEncoding)
self.webView!.loadHTMLString(content!, baseURL: url)
}
It wouldn't be ideal but you could likely implement a custom NSURLProtocol handler to intercept HTTP requests and modify them with your custom user-agent header. I don't think this would work on iOS since WKWebView makes requests out-of-process and bypasses any registered NSURLProtocols. But it might work on OS X?
With a user entered URL in a text field, you can completely control the NSURLRequest using an NSMutableURLRequest object, and set the header field for it.
However, with things the user actually clicks on within the web view, you're kind of not in control from obvious and clearly documented Objective-C land.
I do not see any documented way beyond what WKWebView seems to push things toward, JavaScript. So, that means you can do things like posted here:
Mocking a useragent in javascript?
Using the script injection APIs.
This is why I do not like WKWebView. I want to like it, but I do not want to learn to do half of everything in JavaScript.
So, you can create a WKUserScript object to do this, and set its injection time to WKUserScriptInjectionTimeAtDocumentStart. That will enable you to handle things requested from page elements within the document, as long as the page itself is not loading other scripts that conflict.
First Quit Safari. Then open Terminal and paste this command, and press enter:
defaults write com.apple.Safari CustomUserAgent "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/538.46 (KHTML, like Gecko) Version/7.1 Safari/537.85.7\""
Open Safari and you're done.
To undo this change, close Safari and paste this command into terminal:
defaults delete com.apple.Safari CustomUserAgent
Sometimes a restart may be required to get these changes to stick, not sure why, could be a cache thing.
I'm trying to pull data from Google trends and got a "You have reached your daily limit" error after only 2 tries.
Is there any way to go around this? I know Google API projects have special quota limits but Google Trends doesn't have an API. I also read that we may need to pass it a cookie file so that it seems like I'm logged in. Has anyone faced this issue before?
I'm struggling with the same issue!
From your question I can't figure out what stage have you achieved...
But here is the solution that I've found:
You should emulate browser with cookies.
I think the best way to do it is to use Mechanize library.
At first your program should "login" using GET request to "https://accounts.google.com/Login?hl=en"
Immediately after that you can access some other personal resources, but not google trends!
After some significant time you can successfully get google trends data as CSV.
I still have not discovered the exact time period, but it is more than 10 minutes and less than several hours :). That is why saving your cookies for latter use is a good idea!
Few more tips:
If you are developing using python / ruby under Windows do not forget to set up CA ROOT certificates package for OpenSSL library. Otherwise HTTPS connection will fail and you won't login! See Getting the `certificate verify failed (OpenSSL::SSL::SSLError)` erro with Mechanize object
I recommend you to save cookies to external file at program shutdown. And restoring them at startup.
Do not forget to allow redirects, because Google is using redirects all the time.
Ruby code example:
require 'mechanize'
require 'logger'
begin
agent = Mechanize.new { |a|
a.user_agent = 'Opera/9.80 (Windows NT 5.1) Presto/2.12.388 Version/12.16'
cert_store = OpenSSL::X509::Store.new
cert_store.add_file 'cacert.pem'
a.cert_store = cert_store
a.log = Logger.new('mech.log')
if File.file?('mech.cookies')
cookies = Mechanize::CookieJar.new
cookies.load('mech.cookies')
a.cookie_jar = cookies
end
a.open_timeout = 5
a.read_timeout = 6
a.keep_alive = true
a.redirect_ok = true
}
LOGIN_URL = "https://accounts.google.com/Login?hl=en&continue=http://www.google.com/trends/"
login_page = agent.get(LOGIN_URL)
login_form = login_page.forms.first
login_form.Email = *
login_form.Passwd = *
login_response_page = agent.submit(login_form)
page = agent.get(url)
# DO SOME TRENDS REQUESTS AFTER SIGNIFICANT PERIOD OF TIME
ensure
if agent
agent.cookie_jar.save('mech.cookies')
end
end
You probably disabled your cookies, which makes Google Trends think you're a robot
I think I have found a way to solve the problem. Just make sure that you call the Google Trends API with the cookie PREF. That is you don't need to login the Google account. Of course, you don't need to emulate browser. The cookie PREF is just enough.
OK. Where the cookie PREF comes from? It is very easy. Just open the browser, and login in your Google account. Finally, look up the cookie PREF under the Google website, it is just under the domain www.google.com.Then copy the value of the cookie PREF to your program or script. That's all.
I have called the Google Trends API hundreds of times in several seconds by this way. Good Luck to you!
I found this paper about prevention or just a Zeta-Jones effect in google Trends, it was so useable:
G Fond, A Gamanb, E Haffenb, P Llorca. "Google Trends: ready for real-time suicide prevention or just a Zeta-Jones effect ?." International Journal of Computer Networks and Communications Security 3, no. 1 (2015): 1-5.
I know this is possible via a simple registry change to accomplish this as long as IE/firefox is being used. However, I am wondering if there is a reliable way to do so for other browsers,
I am specifically looking for a way to do this via an installer, so editing a preference inside a specific browser will not cut it.
Here is the best I can come up with:
IE: http://msdn.microsoft.com/en-us/library/aa767914(VS.85).aspx
FireFox: http://kb.mozillazine.org/Register_protocol
Chrome: Since every other browser in seems to support the same convention, I created a bug for chrome.
Opera: I can't find any documentation, but it appears to follow the same method as IE/Firefox (see above links)
Safari: Same thing as opera, it works, but I can't find any documentation on it
Yes. Here is how to do it with FireFox:
http://kb.mozillazine.org/Register_protocol
and Opera:
http://www.opera.com/support/kb/view/535/
If someone looks like a solution for an intranet web site (for all browsers, not only IE), that contains hyperlinks to a shared server folders (like in my case) this is a possible solution:
register protocol (URI scheme) via registry (this can be done for all corporative users i suppose). For example, "myfile:" scheme. (thanks to Greg Dean's answer)
The hyperlink href attribute will then should look like
<a href='myfile:\\mysharedserver\sharedfolder\' target='_self'>Shared server</a>
Write a console application that redirects argument to windows explorer (see step 1 for example of such application)
This is piece of mine test app:
const string prefix = "myfile:";
static string ProcessInput(string s)
{
// TODO Verify and validate the input
// string as appropriate for your application.
if (s.StartsWith(prefix))
s = s.Substring(prefix.Length);
s = System.Net.WebUtility.UrlDecode(s);
Process.Start("explorer", s);
return s;
}
I think this app can be easily installed by your admins for all intranet users :)
I couldn't set up scheme setting to open such links in explorer without this separate app.