Need some clarification regarding getting started with HTML Agility Pack - html-agility-pack

Question

My background:
I am a newbie when it comes to HTML scrubbing. It has been about four years since I did my only work coding for with C# for html. My other coding with C# equally a while back was for forms to manipulate data in SQL Server databases.
What I have done to try to get started with HTML Agility Pack (HAP):
I have spent several days trying to make sense of instructions found from various online sources about how to get started with HTML Agility Pack. Some of what I have found so far is listed below:
www.4guysfromrolla.com/articles/011211-1.aspx
olussier.net/2010/03/30/easily-parse-html-documents-in-csharp/
stackoverflow.com/questions/846994/how-to-use-html-agility-pack
shatalov.su/en/articles/web/parser_1.php
still more referred to below...
My Results so far:
I have found the material to be quite confusing with each source seeming to tell me something different. All my attempts have come to dead ends.
So that you can efficiently sort out my confusion and reply to my specific situation I will describe in three sections below my project, my environment and my questions;
My Project
I am tasked with creating a process to scrub data from html files. I know the files well. The files will reside on the file system on local on the machine. The html file(s) will be created elsewhere by a process we do not own and will be placed in the local folder I just referred to above. (FYI - Though it is not a part of my question, I expect to create a project or app that will be run on a schedule to perform the scrubbing task and then input the collected data into a database table.)
My Environment
As stated above the html file(s) to be processed will reside on the local machine.
I have newly installed Visual Studio 2010 Professional on this machine to code for this project.
The HTML Agility Pack is now accessible to this machine on a file share.
Under REGEIT: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP are listed the following indicating the version of .NET framework installed on this machine;
CDF
V2.0.50727
V3.0
V3.5
V4
V4.0
My Questions
1.) I am told by some sites to download HTML Agility Pack and to use the file "HtmlAgilityPack.dll," however the zip file contains nine folders, each with a different copy of this file. Which one do I want?
Here are the names of the folders;
Net20
Net40
Net40-client
Net45
sl3-wp
sl4
sl4-windowsphone71
sl5
winrt45
2.) An answer to a forum question “How to I use the HTML Agility Pack” at stackoverflow.com/questions/846994/how-to-use-html-agility-pack instructs the questioner to “Download and build the HTML Agility Pack Solution”, and directs the questioner to the site htmlagilitypack.codeplex.com which then has a link to nuget.org/packages/HtmlAgilityPack which says to ‘install’ the HTMLAgilityPack by running the command “PM> Install-Package HtmlAgilityPack” in the “Package Manager Console”
What does all this mean? Other sites say to bout the dll in the bin folder. What is that telling me to do?
Please explain with more detailed to get me started.
3.) Assuming I am using C# what kind of project should I create?
4.) Please direct me to any other resources that you believe is applicable to my project.

Answer 1

Looks like you can create a .NET 4.0 project, given the .NET framework versions you have installed on your machine. What type of project depends on how you'd like your application to run. I'd personally opt for creating a C# Class Library project that contains the load html and scrub code and then host that in whatever mechanism you want to use to actually open the files.
To open a file from FileSystem, either use File.OpenRead or File.ReadAllText from System.IO.File. You can pass the stream or the file contents to the HtmlDocument.Load/LoadHtml methods.
HtmlDocument doc = new HtmlDocument();
// Use File.ReadAllText
string contents = File.ReadAllText("PathToFileName");
doc.LoadHtml(contents);
// Or use a stream
using (var contents = File.OpenRead("PathToFileName"))
{
doc.Load(contents);
}
Possibilities for hosting are plentiful. Console Application (can be invoked from the command line or through the Task Scheduler), Windows Service (can be loaded in Windows, run in the background even when nobody is logged on to the machine and can potentially use the FileSystemWatcher to automatically pic up the files, or a Windows Forms/WPF application which will let the user select the files to process and then show the results somehow.
As for how to use it, this is one of the primary issues with the Html Agility Pack. New ways of using it have been added over time and the actual library has therefore several ways you can use. You could take the old fashioned XPath query route (which was the original API) or you can use the Linq-to-HTML/XML route (which is the newer, way). Neither is better than the other, they both have their distinct advantages. The XPath solution allows you to store the queries in a text file easily, so it's great for a configurable system, while the Linq-To-HTML version is a little easier on the eyes from a developer perspective.
As for how to download it, there are a number of options here as well.
You can indeed download the sources from the CodePlex website. Regardless of how you proceed, you might want to do that any way, it allows you to look under the hood and figrue out why something works the way it does, even if you don't compile the library yourself.
You can download the binaries from CodePlex and store them with your project, before the creation of services such as NuGet, this was the only easy way for developers to distribute their libraries.
I'd personally choose to go the NuGet route. When you're using Visual Studio 2012, NuGet is already integrated with Visual Studio. When you're using Visual Studio 2010, you'll have to install the NuGet extension to get the same functionality. Once installed you can open the Nuget Package manager Console from within Visual Studio. With a Visual Studio Solution open and your freshly created Class Library selected in the Solution Explorer you then proceed to enter the Install-Package HtmlAgilityPack command to let Visual Studio download and install the proper version of the HTML Agility Pack for your project. No worries about which library to select, Visual Studio will do that for you.
How to use it now that you've installed the library completely depends on what type of HTML scrubbing you're after and whether you choose the XPath or the Linq-to-HTML route. But it generally comes down to loading the HTML Document:
HtmlDocument doc = new HtmlDocument();
doc.Load(/* path to file or stream */); or doc.LoadHtml(/*string*/);
And after loading the file and catching any parsing errors that might occur, proceed to query the HTML using XPath like the contents are actually XML (the XML/XPath documentation from MSDN actually applies here):
var nodes = doc.DocumentNode.SelectNodes("//table/tr/td");
Or the same query using Linq-to-HTML:
var nodes = doc.DocumentNode.Descendants("table")
.Select(table => table.Elements("tr").Select(tr => tr.Elements("td")));
Or use the Linq-to-Html with Linq query syntax:
var tds = from tables in doc.DocumentNode.Descendants("table")
from tr in tables.Elements("tr")
from td in tr.Elements("td")
select td;
You can make the queries as wild as you want. The syntax is either similar to the standard XPathnavigator syntax in the .NET Framework (using SelectNodes/SelectSingleNode/Children etc) or the Linq-to-XML syntax (using .Descendants/.Ancesters/.Element(s) and standard Linq).
See also:
Linq to XML documentation
XPathNavigator/IXPathNavigable documentation