ElasticSearch Query for range OR missing in array - elasticsearch

Question

I'm trying to create a query that will query the below mapping for values that have a "period" that either matches a specific date, or is missing it's value (with a null value). Please note that I am working with a third-party database, so I cannot change the mappings. Bear with me if the example data and mappings are large.. I've tried to cut everything nonessential away.
{
"EXAMPLE":{
"mappings":{
"company":{
"properties":{
"CompanyData":{
"properties":{
"participantRelations":{
"type":"nested",
"include_in_parent":true,
"properties":{
"participant":{
"type":"nested",
"include_in_parent":true,
"properties":{
"unitNumber":{
"type":"long"
}
},
"organizations":{
"properties":{
"memberData":{
"type":"nested",
"include_in_parent":true,
"properties":{
"attributes":{
"properties":{
"values":{
"properties":{
"period":{
"properties":{
"validFrom":{
"type":"date",
"format":"dateOptionalTime"
},
"validTo":{
"type":"date",
"format":"dateOptionalTime"
}
}
}
"value":{
"type":"string"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Here is some example data. I've translated it from the (Danish) source it comes from, so any slight missspellings etc are just misstypes.
{
"company": {
"companyData":{
"participantRelations":[
{
"participant":{
"unitNumber":4003857309
},
"organizations":[
{
"memberData":[
{
"attributtes":[
{
"values":[
{
"value":"chairman",
"period":{
"validFrom":"2014-10-01",
"validTo":"2016-08-11"
}
}
]
},
{
"values":[
{
"value":"generalassembly",
"period":{
"validFrom":"2014-10-01",
"validTo":"2016-08-11"
}
}
]
}
]
}
]
},
{
"memberData":[
{
"attributes":[
{
"values":[
{
"value":"chairman",
"period":{
"validFrom":"2016-08-16",
"validTo":"2017-06-08"
}
},
{
"value":"boardmember",
"period":{
"validFrom":"2017-06-09",
"validTo":null
}
}
]
},
{
"values":[
{
"value":"generalassembly",
"period":{
"validFrom":"2016-08-16",
"validTo":"2017-06-08"
}
},
{
"value":"generalassembly",
"period":{
"validFrom":"2017-06-09",
"validTo":null
}
}
]
}
]
}
]
}
]
}
]
}
}
}
What I want to do is something like the query below, which doesn't quite work as it has cases it cannot handle for reasons I do not know. What it needs to do is look for any company.participantRelations.organizations.memberData.attributes.values.period.validTo over a certain date, OR if the date is null. Now I know nulls are funky in ES, but I know that the date properties will always be there, but the validTo will be set to null if there is no date yet.
Furthermore, it needs to be nested on organizations as well, as I need a specific unitNumber to be present.
{
"query":{
"nested":{
"filter":{
"bool":{
"must":[
{
"nested":{
"filter":{
"bool":{
"must":[
{
"bool":{
"should":[
{
"range":{
"company.companyData.participantRelations.organizations.memberData.attributtes.values.period.validTo":{
"gte":"2017-08-14T15:23:11.011"
}
}
},
{
"missing":{
"field":"company.companyData.participantRelations.organizations.memberData.attributtes.values.period.validTo"
}
}
]
}
}
]
}
},
"path":"company.companyData.participantRelations.organizations.memberData"
}
},
{
"term":{
"company..companyData.participantRelations.participant.unitNumber":4003857309
}
}
]
}
},
"path":"company.companyData.participantRelations"
}
}
}
This query works in two cases:
Where there is only one entry in the list of values, and it's validTo date is null
Where the validTo date is greater or equal to my date limit.
It does not seem to work if there are two entries, the first of which has a date that is earlier than my limit, and the second entry has a null value (as in the example).
I realize this is kind of convoluted, but with the database I'm querying that is just the way it is. I hope I've simplified it enough for you to get my issue.
Thanks in advance.