Update your browser | Facebook; shown by HtmlAgilityPack - html-agility-pack

Question

I am using HtmlAgilityPack to get the meta and othe descriptions of the page. The code works find for the simple websites such as Tumblr., Twitter, Stack Overflow.
But when I try to load major sites, such as Google it shows me just a title as Google and no description tag. Similary for Facebook it shows me no description but for the title it shows me Update your browser | Facebook.
I am new to this package, I downloaded latest version of it from NuGet package in MS WebMatrix. The code I am using is as:
#using HtmlAgilityPack;
#{
Layout = "~/_SiteLayout.cshtml";
var Title = "";
var Description = "";
using(var client = new WebClient()){
var html = client.DownloadString("http://www.facebook.com");
var doc = new HtmlDocument();
doc.LoadHtml(html);
var title = doc.DocumentNode.Descendants("title").FirstOrDefault();
if(title != null){
Title = title.InnerText;
}
var description = doc.DocumentNode.Descendants("meta")
.Where(n => n.GetAttributeValue("name", String.Empty)
.Contains("description")).FirstOrDefault();
if(description != null){
Description = description.GetAttributeValue("content", string.Empty);
}
}
}
This issue, by name, looks like an old browser issue. How to fix this?

Answer 1

After searching for a long time for this, I got the solution from Mike Brind on ASP.NET Forums.
var Image = "";
using(var client = new WebClient()){
client.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 6.3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36");
client.Headers.Add("method", "GET");
client.Headers.Add("version", "HTML/1.1");
client.Headers.Add("accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
var html = client.DownloadString("http://www.google.com");
var doc = new HtmlDocument();
doc.LoadHtml(html);
var title = doc.DocumentNode.Descendants("title").FirstOrDefault();
if(title != null){
Title = title.InnerText;
}
var description = doc.DocumentNode.Descendants("meta")
.Where(n => n.GetAttributeValue("name", String.Empty)
.Contains("description")).FirstOrDefault();
if(description != null){
Description = description.GetAttributeValue("content", string.Empty);
}
var image = doc.DocumentNode.Descendants("link")
.Where(n => n.GetAttributeValue("rel", String.Empty)
.Contains("shortcut icon")).FirstOrDefault();
if(image != null) {
Image = image.GetAttributeValue("href", string.Empty);
}
}
This was the code required for this. Actually the main thing was, when the person creates a new request from his Computer, Browser sends some details to the server, whereas I wasn't sending anything. That's why, Facebook and Google were not returning anything to me. When I tried to include some fake headers, they gave me the details I needed from them.
After that, it was good to go!