Switching from beautifulsoup to htmlelement - how to find elements - xpath

Question

I have an exsiting process that extracts elements from html documents that make use of the xbrli xml standard.
And example of a document can be found here:
The process works well (I'm using multiprocessing to work in parallel) but I have ~20m html and xml files to process and I'm finding beautifulsoup is the core bottleneck.
I am looking at htmlelement as a hopefully quicker alternative to extracting the data I need but I'm struggling to find elements. For example, in BS I can do the following:
for tag in soup.find_all('xbrli:unit'):
l_unitid = tag.attrs.get('id')
l_value = tag.text
l_unit_dict[l_unitid] = {'unitid':l_unitid,'value':l_value}
Which will find all xbrli:unit tags and I can extract their values easily.
However, when I try something similar in htmlelement I get the following exception:
import htmlement
source = htmlement.parse("Prod223_2542_00010416_20190331.html")
for tag in source.iterfind('.//xbrli:unit'):
l_unitid = tag.attrs.get('id')
l_value = tag.text
print(l_unitid)
print(l_value)
SyntaxError: prefix 'xbrli' not found in prefix map
A bit of googling led me to a few articles, but I can't seem to make progress
SyntaxError: prefix 'a' not found in prefix map
Parsing XML with namespace in Python via 'ElementTree'
I've tried adding in a namespace map but it's just not finding anything, no matter which way round I put things, or what tags I look for
source = htmlement.parse("Prod223_2542_00010416_20190331.html")
namespaces = {'xbrli': 'period'}
for tag in source.iterfind('.//xbrli:period',namespaces):
l_unitid = tag.attrs.get('id')
l_value = tag.text
namespaces = {'xbrli': 'period'}
for tag in source.iterfind('.//{xbrli}period',namespaces):
l_unitid = tag.attrs.get('id')
l_value = tag.text
print(l_unitid)
print(l_value)
namespaces = {'period':'xbrli'}
for tag in source.iterfind('.//{xbrli}period',namespaces):
l_unitid = tag.attrs.get('id')
l_value = tag.text
print(l_unitid)
print(l_value)
namespaces = {'period':'xbrli'}
for tag in source.iterfind('.//period',namespaces):
l_unitid = tag.attrs.get('id')
l_value = tag.text
print(l_unitid)
print(l_value)
All return nothing - they don't enter the loop. I've clearly got something very wrong in my understanding of how to use the elementree structure vs BS, but I don't quite know how to move from one to the other.
Any suggestions would be welcome.

Answer 1

Two general comments before I get to a proposed answer:
First, you are dealing with an xml document, so it's generally better to use an xml, not html, parser. So that's what I'm using below instead of beautifull soup or htmlelement.
Second, about xbrl generally: from bitter experience (and as many others pointed out), xbrl is terrible. It's shiny on the surface, but once you pop the hood, it's a mess. So I don't envy you...
And, with that said, I tried to approximate what you are likely looking for. I didn't bother to create dictionaries or lists, and just used print() statement. Obviously, if it helps you, you can modify it to your own requirements:
from lxml import etree
import requests
r = requests.get('https://beta.companieshouse.gov.uk/company/00010416/filing-history/MzI1MTU3MzQzMmFkaXF6a2N4/document?format=xhtml&download=1')
root = etree.fromstring(r.content)
units = root.xpath(".//*[local-name()='unit'][#id]/#id")
for unit in units:
unit_id = unit
print('unit: ', unit)
print('----------------------------')
context = root.xpath(".//*[local-name()='context']")
for tag in context:
id = tag.xpath('./#id')
print('ID: ',id)
info = tag.xpath('./*[local-name()="entity"]')
identifier = info[0].xpath('.//*[local-name()="identifier"]')[0].text
print('identifier: ',identifier)
member = info[0].xpath('.//*[local-name()="explicitMember"]')
if len(member)>0:
dimension = member[0].attrib['dimension']
explicitMember = member[0].text
print('dimension: ',dimension,' explicit member: ',explicitMember)
periods = tag.xpath('.//*[local-name()="period"]')
for period in periods:
for child in period.getchildren():
if 'instant' in child.tag:
instant = child.text
print('instant: ',instant)
else:
dates = period.xpath('.//*')
start_date = dates[0].text
end_date = dates[1].text
print('start date: ', start_date,' end date: ',end_date)
print('===================')
A random sample from the output:
ID: ['cfwd_31_03_2018']
identifier: 00010416
instant: 2018-03-31
start date: 2017-04-01 end date: 2018-03-31
===================
ID: ['CountriesHypercube_FY_31_03_2019_Set1']
identifier: 00010416
dimension: ns15:CountriesRegionsDimension explicit member: ns15:EnglandWales
instant: 2018-03-31
start date: 2018-04-01 end date: 2019-03-31