How to query items in a nested list? - elasticsearch

Question

I use Elasticsearch to store system vulnerabilities. Right now my typical entry is
{
_id: somenadomid
_source: {
"ip": "10.10.10.10",
"vuln_name": "v1",
"vuln_type": 1
}
This approach has the advantage to simplify queries ("number of machines with a vuln of type 1" -> an aggregation, "number of vulnerabilities" - a query_all search and associated totalvalue, ...).
It aslo has drawbacks, in particular:
the information is heavily demultiplied: the information about one host is copied over all vulnerabilities
there are as many lines as vulnerabilities, and not hosts (50x more in average)
the natural container is "host" and not "vulnerability" - it can be updated, deleted, etc. more easily.
I am therefore considering changing the scheme to a "host" base one:
{
_id: machine1
_source: {
"ip": "10.10.10.10",
"vuln": [
{
"name": "v1",
"type": 1
},
{
"name": "v2",
"type": 1
}
]
}
The problem I am running into is that I still fundamentally query vulnerabilities and do not know how to "explode" them in a query.
Specifically (I believe my problem will gravitate around this family of queries), how can I query
the total number of vulnerabilities of type 1 (not the hosts - there can be several vulns of type 1 per host, the basic query retrieves the entries, which are hosts)
the same as above, but with some filtering on, say, the vulnerability name ("number of vulnerabilities of type 1 with "Microsoft" in the name) - the filtering is on a feature of the vulnerability and not the host)

Answer 1

Just to give you a simple overview,
In Elasticsearch you have two way to mange nested data, you can use Nested Object or Inner Object, behind the scene they are completely different.
The nested type is a specialized version of the object datatype that allows arrays of objects to be indexed and queried independently of each other.
Nested docs are stored in the same Lucene block as each other, which helps read/query performance.
Reading a nested doc is faster than the equivalent parent/child.
Updating a single field in a nested document (parent or nested children) forces ES to reindex the entire nested document. This can be very expensive for large nested docs
"Cross referencing" nested documents is impossible
Best suited for data that does not change frequently
Inner Object is an objects embedded inside the parent document.
Easy, fast, performant Only applicable when one-to-one relationships
are maintained No need for special queries Nested
Please have a look the following link for further information the difference between Inner Object and Nested Object.
https://www.elastic.co/blog/managing-relations-inside-elasticsearch
In order to query and aggregate(To Get the total Number) have look the following links:
Query : https://www.elastic.co/guide/en/elasticsearch/guide/master/nested-objects.html
Aggregations :
https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-aggregation.html