Completion Suggester in ElasticSearch On Existing Field - elasticsearch

Question

In my elasticsearch index, I have indexed a bunch of jobs. For simplicity, let's just say they are a bunch of Job Titles. When people are typing a job title into my search engine, I want to "Auto Complete" with possible matches.
I've investigated the Completion Suggester here : http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-suggesters-completion.html
However all the examples I've found involve creating a new field on your index, and manually populating this field while indexing/rivering.
Is there any way to have a completion suggester on an existing field? Even if it means reindexing data that's fine. For example, when I want to keep the original not_analysed text, I can do something like this in the mappings :
"JobTitle": {
"type": "string",
"fields": {
"Original": {
"type": "string",
"index": "not_analyzed"
}
}
}
Is this possible to do with the suggesters?
If not, is it possible to do a non whitespace tokenizing/N-Gram search instead to get these fields? While it would be slower, I assume that would work.

Answer 1

Okay, here is the easy way that (may or) may not scale, using prefix queries.
I'll create an index using the "fields" technique you mentioned, and some handy job description data I found here:
DELETE /test_index
PUT /test_index
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"doc": {
"properties": {
"title": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}
PUT /test_index/_bulk
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"experienced bra fitter", "desc":"I bet they had trouble finding candidates for this one."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"PlayStation Brand Ambassador", "desc":"please report to your residence in the United States of Nintendo."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Eyebrow Threading", "desc":"I REALLY hope this has something to do with dolls."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Administraive/ Secretary", "desc":"ok, ok, we get it. It’s clear where you need help."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Finish Carpenter", "desc":"for when the Start Carpenter gets tired."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Helpdesk Technician # Pentagon", "desc":"“Uh, hello? I’m having a problem with this missile…”"}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Nail Tech", "desc":"so nails can be pretty complicated…"}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Remedy Engineer", "desc":"aren’t those called “doctors”?"}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Saltlick Cashier", "desc":"new trend in the equestrian industry. Ok, enough horsing around."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Molecular Biologist II", "desc":"when Molecular Biologist I gets promoted."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Breakfast Sandwich Maker", "desc":"we also got one of these recently."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Hotel Housekeepers", "desc":"why can’t they just say ‘hotelkeepers’?"}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Preschool Teacher #4065", "desc":"either that’s a really big school or they’ve got robot teachers."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"glacéau drop team", "desc":"for a new sport at the Winter Olympics: ice-water spilling."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"PLUMMER/ELECTRICIAN", "desc":"get a dictionary/thesaurus first."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"DoodyCalls Technician", "desc":"they really shouldn’t put down janitors like that."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Golf Staff", "desc":"and here I thought they were called clubs."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Pressure Washers", "desc":"what’s next, heat cleaners?"}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Sandwich Artist", "desc":"another “Jesus in my food” wannabe."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Self Storage Manager", "desc":"this is for self storage?"}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Qualified Infant Caregiver", "desc":"too bad for all the unqualified caregivers on the list."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Ground Support", "desc":"but there’s just more dirt under there."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Gymboree Teacher", "desc":"the hardest part is not burning your hands sliding down the pole."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"COMMERCIAL space hunter", "desc":"so they did find animals further out in the cosmos? Who knew."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"JOB COACH", "desc":"if they’re unemployed when they get to you, what does that say about them?"}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"KIDS KAMP INSTRUCTOR!", "desc":"no spelling ability required."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"POOLS SUPERVISOR", "desc":"“yeah, they’re still wet…”"}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"HOUSE MANAGER/TEEN SUPERVISOR", "desc":"see the dictionary under P, for Parent."}
{"index":{"_index":"test_index","_type":"doc"}}
{"title":"Licensed Seamless Gutter Contractor", "desc":"just sounds bad."}
Then I can easily run a prefix query:
POST /test_index/_search
{
"query": {
"prefix": {
"title": {
"value": "san"
}
}
}
}
...
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "test_index",
"_type": "doc",
"_id": "mcRfqtwzTyWE7ZNsKFvwEg",
"_score": 1,
"_source": {
"title": "Breakfast Sandwich Maker",
"desc": "we also got one of these recently."
}
},
{
"_index": "test_index",
"_type": "doc",
"_id": "fIYV0WOWRe6gfpYy_u2jlg",
"_score": 1,
"_source": {
"title": "Sandwich Artist",
"desc": "another “Jesus in my food” wannabe."
}
}
]
}
}
Or if I want to be more careful about the matches I can use the un-analyzed field:
POST /test_index/_search
{
"query": {
"prefix": {
"title.raw": {
"value": "San"
}
}
}
}
...
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "test_index",
"_type": "doc",
"_id": "fIYV0WOWRe6gfpYy_u2jlg",
"_score": 1,
"_source": {
"title": "Sandwich Artist",
"desc": "another “Jesus in my food” wannabe."
}
}
]
}
}
This is the easy way. Ngrams are a bit more involved, but not difficult. I'll add that in another answer in a bit.
Here's the code I used:
http://sense.qbox.io/gist/4e066d051d7dab5fe819264b0f4b26d958d115a9
EDIT: Ngram version
Borrowing the analyzers from this blog post (shameless plug), I can set up the index as follows:
DELETE /test_index
PUT /test_index
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"analysis": {
"filter": {
"nGram_filter": {
"type": "nGram",
"min_gram": 2,
"max_gram": 20,
"token_chars": [
"letter",
"digit",
"punctuation",
"symbol"
]
}
},
"analyzer": {
"nGram_analyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
"lowercase",
"asciifolding",
"nGram_filter"
]
},
"whitespace_analyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
"lowercase",
"asciifolding"
]
}
}
}
},
"mappings": {
"doc": {
"properties": {
"title": {
"type": "string",
"index_analyzer": "nGram_analyzer",
"search_analyzer": "whitespace_analyzer",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}
Notice that I use different analyzers for indexing and for searching; that's important because if the search query is broken up into ngrams we will probably get a lot more hits than we want.
Populating with the same dataset used above, I can query with a simple match query to get the results I expect:
POST /test_index/_search
{
"query": {
"match": {
"title": "sup"
}
}
}
...
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 3,
"max_score": 1.8631258,
"hits": [
{
"_index": "test_index",
"_type": "doc",
"_id": "4pcAOmPNSYupjz7lSes8jw",
"_score": 1.8631258,
"_source": {
"title": "Ground Support",
"desc": "but there’s just more dirt under there."
}
},
{
"_index": "test_index",
"_type": "doc",
"_id": "DVFOC6DsTa6eH_a-RtbUUw",
"_score": 1.8631258,
"_source": {
"title": "POOLS SUPERVISOR",
"desc": "“yeah, they’re still wet…”"
}
},
{
"_index": "test_index",
"_type": "doc",
"_id": "klleY_bnQ4uFmCPF94sLOw",
"_score": 1.4905007,
"_source": {
"title": "HOUSE MANAGER/TEEN SUPERVISOR",
"desc": "see the dictionary under P, for Parent."
}
}
]
}
}
Here's the code:
http://sense.qbox.io/gist/b0e77bb7f05a4527de5ab4345749c793f923794c