Find all ID where ID are not in my blacklist - elasticsearch

Question

After many lectures , I cannot say if this kind of query is possible with elasticsearch , I found the "getting started" really excellent but the rest of guide have a lack of examples (from my point of vue ).
See my structure below, I need to retrieve all id who are not in my blacklist. My blacklist is some reference id. For this example I am the id 1 with the firstname "me" . Here in the structure we see I blacklisted "bob" , so the bob id (2) is in my blacklist array because I don't want to find bob in my search result.. :)
Is it possible to only retrieve (dynamically for sure) all id who are not in my blacklist in one query?
If you come from SQL, the same logic could be :
SELECT id FROM index WHERE id NOT IN (SELECT * FROM blacklist WHERE id = 1)
I would like to avoid the 2 step query , if my schema is bad and should be reconsidered , please I'm totally open for advice or suggestions.
Here is the structure :
{
"id: 1,
"balance": 16623,
"firstname": "me",
"blacklist" : [2,1982,939,1982,98716,7611,983838, and thousands others ....],
}
{
"id: 2,
"balance": 16623,
"firstname": "bob,
"blacklist" : [18,1982,939,1982,98716,7611,983838, and thousands others ....],
}
{
"id: 3,
"balance": 16623,
"firstname": "jhon",
"blacklist" : [18,1982,939,1982,98716,7611,983838, and thousands others ....],
}

Answer 1

You can use use a terms filter lookup together with a not filter as follows.
I set up the index with the three docs you have listed:
DELETE /test_index
PUT /test_index
PUT /test_index/doc/1
{
"id": 1,
"balance": 16623,
"firstname": "me",
"blacklist" : [2,1982,939,1982,98716,7611,983838]
}
PUT /test_index/doc/2
{
"id": 2,
"balance": 16623,
"firstname": "bob",
"blacklist" : [18,1982,939,1982,98716,7611,983838]
}
PUT /test_index/doc/3
{
"id": 3,
"balance": 16623,
"firstname": "john",
"blacklist" : [18,1982,939,1982,98716,7611,983838]
}
Then set up a query that filters out docs that are in the blacklist for "me":
POST /test_index/doc/_search
{
"filter": {
"not": {
"filter": {
"terms": {
"id": {
"index": "test_index",
"type": "doc",
"id": "1",
"path": "blacklist"
}
}
}
}
}
}
...
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "test_index",
"_type": "doc",
"_id": "1",
"_score": 1,
"_source": {
"id": 1,
"balance": 16623,
"firstname": "me",
"blacklist": [2,1982,939,1982,98716,7611,983838]
}
},
{
"_index": "test_index",
"_type": "doc",
"_id": "3",
"_score": 1,
"_source": {
"id": 3,
"balance": 16623,
"firstname": "john",
"blacklist": [18,1982,939,1982,98716,7611,983838]
}
}
]
}
}
If you also want to filter out the user whose blacklist is being used, you can set up a slightly more complex filter using or:
POST /test_index/doc/_search
{
"filter": {
"not": {
"filter": {
"or": {
"filters": [
{
"terms": {
"id": {
"index": "test_index",
"type": "doc",
"id": "1",
"path": "blacklist"
}
}
},
{
"term": {
"id": "1"
}
}
]
}
}
}
}
}
...
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "test_index",
"_type": "doc",
"_id": "3",
"_score": 1,
"_source": {
"id": 3,
"balance": 16623,
"firstname": "john",
"blacklist": [18,1982,939,1982,98716,7611,983838]
}
}
]
}
}
Here is the code I used:
http://sense.qbox.io/gist/0b6808414f9447d4f7d23eb4c0d3e937ec2ea4e7