Agnostic Screen scraper using HtmlAgilityPack - html-agility-pack

Question

Lets say I want a screen scraper that doesn't care if you pass it an HTML page, url that goes to an XML Document, or a Url that goes to a text file.
examples:
http://tonto.eia.doe.gov/oog/info/wohdp/dslpriwk.txt
http://google.com
This will work if the page is HTML or a text file:
public class ScreenScrapingService : IScreenScrapingService
{
public XDocument Scrape(string url)
{
var scraper = new HtmlWeb();
var stringWriter = new StringWriter();
var xml = new XmlTextWriter(stringWriter);
scraper.LoadHtmlAsXml(url, xml);
var text = stringWriter.ToString();
return XDocument.Parse(text);
}
}
However; if it is an XML file such as:
http://www.eia.gov/petroleum/gasdiesel/includes/gas_diesel_rss.xml
[Test]
public void Scrape_ShouldScrapeSomething()
{
//arrange
var sut = new ScreenScrapingService();
//act
var result = sut.Scrape("http://www.eia.gov/petroleum/gasdiesel/includes/gas_diesel_rss.xml");
//assert
}
Then I get the error:
An exception of type 'System.Xml.XmlException' occurred in System.Xml.dll but was not handled in user code
Is it possible to write this so that it doesn't care what the URL ultimately is?

Answer 1

to get the exact exception on visual studio CTR+ALT+E and enable CommonLanguageRunTimeExceptions, it seems like LoadHtmlAsXml expects html, so probably your best bet is to use a WebClient.DownloadString(url) and HtmlDocument with property OptionOutputAsXml set to true as the following, when that fails catch it
public XDocument Scrape(string url)
{
var wc = new WebClient();
var htmlorxml = wc.DownloadString(url);
var doc = new HtmlDocument() { OptionOutputAsXml = true};
var stringWriter = new StringWriter();
doc.Save(stringWriter);
try
{
return XDocument.Parse(stringWriter.ToString());
}
catch
{
//it only gets here when the string is xml already
try
{
return XDocument.Parse(htmlorxml);
}
catch
{
return null;
}
}
}