ElasticSearch - indexing URLs and aggregating on Domains - elasticsearch

Question

Similar to this question Elasticsearch aggregate on URL hostname
I'm wanting to store URLs and be able to aggregate on Domains
Using ES6 I have the following mapping defined
PUT /test
{
"settings": {
"analysis": {
"filter": {
"url_domains": {
"type": "pattern_capture",
"preserve_original" : false,
"patterns": [
"https?:\/\/([^/]+)"
]
}
},
"analyzer": {
"url": {
"tokenizer": "uax_url_email",
"filter": [
"url_domains",
"lowercase",
"unique"
]
},
"full_url": {
"tokenizer": "uax_url_email",
"filter": ["lowercase", "stop"]
}
}
}
},
"mappings": {
"urls": {
"properties": {
"url": {
"type": "text",
"fielddata" : true,
"fields": {
"domain": {
"type": "text",
"fielddata" : true,
"analyzer": "url"
},
"full": {
"type": "text",
"fielddata" : true,
"analyzer": "full_url"
}
}
}
}
}
}
}
I then load in the following data
POST /test/urls/_bulk
{"index":{"_id":"1"}}
{"url": "http://gmail1.com/hello/world"}
{"index":{"_id":"2"}}
{"url": "https://gmail2.com/hello"}
{"index":{"_id":"3"}}
{"url": "http://gmail3.com/"}
{"index":{"_id":"4"}}
{"url": "https://gmail4.com"}
{"index":{"_id":"5"}}
{"url": "https://www.gmail5.com"}
{"index":{"_id":"6"}}
{"url": "www.gmail6.coom"}
{"index":{"_id":"7"}}
{"url": "www.gmail7.co.uk/hello"}
When I do the following query
GET /test/urls/_search
{
"size": 0,
"query": {
"match_all": {}
},
"aggs": {
"by_domain": {
"terms": {
"field": "url.domain",
"size": 10
}
}
}
}
The following aggregations are returned:
"aggregations": {
"by_domain": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "coom",
"doc_count": 1
},
{
"key": "gmail1.com",
"doc_count": 1
},
{
"key": "gmail2.com",
"doc_count": 1
},
{
"key": "gmail3.com",
"doc_count": 1
},
{
"key": "gmail4.com",
"doc_count": 1
},
{
"key": "www.gmail5.com",
"doc_count": 1
},
{
"key": "www.gmail6",
"doc_count": 1
},
{
"key": "www.gmail7.co.uk/hello",
"doc_count": 1
}
]
}
}
From running the Analyzer I can see that documents 6 and 7 are not treated as URL's, but are of type ALPHANUM.
How can I get the domains of documents 6 and 7 correctly extracted ?

Answer 1

Bit of persistence paid off ! Changed the pattern to this:
"patterns": [
"https?:\/\/([^/]+)",
"(www[^/]+)"
]