LINQ Extract data from website using HTMLAgilityPack - linq

Question

I'm using C# HTMLAgilityPack to extract the item name, prices & currency symbol from a chinese website: https://meadjohnson.world.tmall.com/search.htm?search=y&orderType=defaultSort&s‌​cene=taobao_shop. Here's a gist of what the html looks like:
<div class="SaleItems">
<dl class="item ">
<dt class="photo"></dt>
<dd class="detail">
<a class="item-name">iPad</a>
<div class="price-area">
<span class="symbol">USD</span>
<span class="price">379</span>
</div>
</dd>
</dl>
<dl class="item ">
<dt class="photo"></dt>
<dd class="detail">
<a class="item-name">iPod</a>
<div class="price-area">
<span class="symbol">CAD</span>
<span class="price">139</span>
</div>
</dd>
</dl>
</div>
So far, this is what my program looks like.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls
| SecurityProtocolType.Tls11
| SecurityProtocolType.Tls12
| SecurityProtocolType.Ssl3;
var htmlDocument = htmlWeb.Load(html);
var sItems = doc.DocumentNode.Descendants("SaleItems");
foreach (var item in sItems)
{
var data = new {
Currency = item["symbol"].InnerText,
Price = item["price"].InnerText,
};
}
This doesn't work. How can I fix what I'm doing wrong?

Answer 1

You can extract your data this way:
var input = #"<div class='SaleItems'>
<dl class='item '>
<dt class='photo'></dt>
<dd class='detail'>
<a class='item-name'>iPad</a>
<div class='price-area'>
<span class='symbol'>USD</span>
<span class='price'>379</span>
</div>
</dd>
</dl>
<dl class='item '>
<dt class='photo'></dt>
<dd class='detail'>
<a class='item-name'>iPod</a>
<div class='price-area'>
<span class='symbol'>CAD</span>
<span class='price'>139</span>
</div>
</dd>
</dl>
</div>";
var html = new HtmlDocument();
html.LoadHtml(input);
var root = html.DocumentNode;
var list = new List<Data>();
foreach (var node in root.Descendants("dl"))
{
var currency = node.Descendants()
.Where(n => n.GetAttributeValue("class", "").Equals("symbol")).FirstOrDefault().InnerText;
var price = node.Descendants()
.Where(n => n.GetAttributeValue("class", "").Equals("price")).FirstOrDefault().InnerText;
list.Add(new Data { Currency = currency, Price = price});
}
public class Data
{
public string Currency { get; set; }
public string Price { get; set; }
}
Or you can use query expression instead the foreach part:
var list = (from node in root.Descendants("dl")
let currency = node.Descendants().Where(n => n.GetAttributeValue("class", "").Equals("symbol")).FirstOrDefault().InnerText
let price = node.Descendants().Where(n => n.GetAttributeValue("class", "").Equals("price")).FirstOrDefault().InnerText
select new Data {Currency = currency, Price = price}).ToList();

Answer 2

The exact error is that in the foreach() block "item" is a variable of HtmlNode type but you are trying to "indexing" it. Instead of this you should use
item.Descendants("symbol")
or
item.SelectSingleNode(".//span[#class='symbol']");
Or You can use this code:
var document = new HtmlWeb();
var root = document.Load(url);
var data = new List<Item>();
foreach (var item in root.DocumentNode.SelectNodes("//dl"){
var name = item.SelectSingleNode(".//a[#class='item-name']").InnerText;
var price = item.SelectSingleNode(".//span[#class='price']").InnerText;
var symbol = item.SelectSingleNode(".//span[#class='symbol']").InnerText;
data.Add(new Item(){ Name = name, Price = price, Symbol = symbol });
}
public class Item{
public string Name;
public int Price;
public string Symbol;
}