How to execute a bulk ElasticSearch operation from an AppSync resolver? - elasticsearch

Question

I'm trying to execute a bulk operation over an ElasticSearch index from a VTL AppSync resolver.
Specifically this mutation:
input CreateTopicsInput {
language: String!
texts: [String!]!
}
type Mutation {
createTopics(input: CreateTopicsInput!): [Topic!]
}
And I created the following resolver:
Mutation.createTopics.req.vtl
#set( $body = "" )
#set( $q = '"' )
#foreach( $text in $ctx.args.input.texts )
#set( $body = $body.concat("{ ${q}index${q}: { ${q}_id${q}: ${q}${text}${q} } }
") )
#set( $body = $body.concat("{ ${q}text${q}: ${q}$text${q}, ${q}suggest${q}: { ${q}input${q}: ${q}$text${q} } }
") )
#end
{
"version": "2017-02-28",
"operation": "POST",
"path": "/topic-${ctx.args.input.language}/doc/_bulk",
"params": {
"headers" : [ { "Content-Type" : "application/x-ndjson" } ],
"body": "$body"
}
}
And when executed for example with this data:
mutation CreateTopic {
createTopics(input: {
language: "en",
texts: [ "COVID", "Election" ]}) {
text
}
}
Seems to output a correct invocation:
{
"version": "2017-02-28",
"operation": "POST",
"path": "/topic-en/doc/_bulk",
"params": {
"headers" : [ { "Content-Type" : "application/x-ndjson" } ],
"body": "{ "index": { "_id": "COVID" }
{ "text": "COVID" }
{ "index": { "_id": "Election" }
{ "text": "Election" }
"
}
}
but it doesn't work. Specifically it throws:
{
"data": {
"createTopics": null
},
"errors": [
{
"path": [
"createTopics"
],
"data": null,
"errorType": "MappingTemplate",
"errorInfo": null,
"locations": [
{
"line": 2,
"column": 3,
"sourceName": null
}
],
"message": "Unexpected character ('i' (code 105)): was expecting comma to separate Object entries\n at [Source: (String)\"{\n \"version\": \"2017-02-28\",\n \"operation\": \"POST\",\n \"path\": \"/topic-en/doc/_bulk\",\n \"params\": {\n \"headers\" : { \"Content-Type\" : \"application/x-ndjson\" },\n \"body\": \"{ \"index\": { \"_id\": \"CODID\" } }\n{ \"text\": \"CODID\", \"suggest\": { \"input\": \"CODID\" } }\n{ \"index\": { \"_id\": \"Election\" } }\n{ \"text\": \"Election\", \"suggest\": { \"input\": \"Election\" } }\n\"\n }\n}\n\"; line: 7, column: 18]"
}
]
}
Some idea?

Answer 1

i played a little bit around and figure out that you would need to
quote your double quotes
write all in one line, divided by a "\n"
the last part in that line needs to be an "\n" again
As example:
{
"version": "2018-05-29",
"method": "POST",
"resourcePath": "/_msearch/template",
"params": {
"headers": {
"Content-Type": "application/x-ndjson"
},
"body": "{ \"index\": { \"_id": \"COVID\" }\n{ \"text\": \"COVID\" }\n"
}
}
Of course, this is not very convinced and for your use case this could work better:
#set($ndjson = [])
$util.qr($ndjson.add({ "index": "COVID" }))
$util.qr($ndjson.add({ "text": "COVID", "Param2": "vaccine" } }))
{
"version": "2018-05-29",
"method": "POST",
"resourcePath": "/_msearch/template",
"params": {
"headers": {
"Content-Type": "application/x-ndjson"
},
"body": "#foreach ($line in $ndjson)$util.escapeJavaScript($util.toJson($line))\n#end"
}
}
The (maybe) problematic part could be the JavaScript Escaping. Not sure how your data looks like but it could result into some more quoted data as needed.
P.s: i switched my implementation to the last mentioned example and it works fine for the multi search.