how to upload bulk json data on elasticsearch sever using curl? - elasticsearch

I am using ES version 6.1.3.
I am trying to upload bulk of json data using curl cmd on windows 7
here's my cmd:
curl -H "Content-Type:application/json" -XPOST "http://localhost:9200/fooditems/item/_bulk?pretty" --data-binary #food_items.json
I tried an alternate cmd for it which are
curl -H "Content-Type:application/x-ndjson" -XPOST "http://localhost:9200/fooditems/item/_bulk?pretty" --data-binary #food_items.json
curl -XPOST "http://localhost:9200/fooditems/item/_bulk?pretty" --data-binary #food_items.json
and here is my sample data of food_items.json
> {"index":{"_index":"fooditems","_type":"item","_id":0}}
> {"id":"0","name":"Jowar Puffed"}
> {"index":{"_index":"fooditems","_type":"item","_id":1}}
> {"id":"1","name":"Uamp and tea"}
> {"index":{"_index":"fooditems","_type":"item","_id":2}}
> {"id":"2","name":"Banana Chips"}
I have provided mapping for data through Kibana is as-
POST /fooditems/item
{
"mappings": {
"_default_":{
"properties": {
"id":{
"type" :"text"
},
"name":{
"type": "integer"
}
}
}
}
}
I have been encountered errors
"The bulk request must be terminated by a newline [\n]"
illegal_argument_exception
i am not sure what it's trying to say.
pls help me out here.

Related

Elasticsearch 6 create new field requires data type but "Indices created in 6.x only allow a single-type per index"

Create a new field in Elasticsearch 6.6.2 gives the following error:
{
"error": {
"root_cause": [
{
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: mapping type is missing;"
}
],
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: mapping type is missing;"
},
"status": 400
}
The request:
curl --request PUT http://10.1.3.81:9200/netswitch_message/_mapping -H "Content-Type: application/json" -d \
'{
"properties": {
"amount": {"type": "integer"}
}
}'
gives error no matter what data type I use. The index already has types integer, text/keyword, text and date.
curl --request PUT http://10.1.3.81:9200/netswitch_message/_mapping -H "Content-Type: application/json" -d "{\"properties\": {\"amount\": {\"type\": \"integer\"}}}"
curl --request PUT http://10.1.3.81:9200/netswitch_message/_mapping -H "Content-Type: application/json" -d "{\"properties\": {\"amount\": {\"type\": \"text\"}}}"
curl --request PUT http://10.1.3.81:9200/netswitch_message/_mapping/data -H "Content-Type: application/json" -d "{\"properties\": {\"amount\": {}}}"
curl --request PUT http://10.1.3.81:9200/netswitch_message/_mapping -H "Content-Type: application/json" -d "{\"properties\": {\"amount\": {}}}
Expected to create a new field
Actually got syntax error:
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: mapping type is missing;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: mapping type is missing;"},"status":400}
You are right that 6.x restricts you to a single _type, but you do still need to supply the name of that type (in 7.x, it defaults to _doc).
Change your mapping to specify the _type, like the following which sets it to "my-type":
curl --request PUT http://10.1.3.81:9200/netswitch_message/_mapping/my-type -H "Content-Type: application/json" -d \
'{
"properties": {
"amount": {"type": "integer"}
}
}'
See: https://www.elastic.co/guide/en/elasticsearch/reference/6.6/indices-put-mapping.html#indices-put-mapping

Elasticsearch 7 unable to create index

I am trying to create index using following syntax
curl -H "Content-Type: application/json" -XPUT 127.0.0.1:9200/movies -d '
{
"mappings": {
"movie": {
"properties": {
"year": {"type":"date"}
}
}
}
}'
I guess "movie" cannot be child of the "mappings", can someone please help me transform this into Elasticsearch 7 compatible syntax.
I tried using "movie.year" : {"type":"date"} but then it fails on following insert statement
curl -H "Content-Type: application/json" -XPUT 127.0.0.1:9200/movies/movie/109487 -d '
{
"genre":["IMAX", "Sci-Fi"],
"title":"Intersteller",
"year":2014
}'
I copied from tutorial of Elasticsearch 6
"Rejecting mapping update to [movies] as the final mapping would have
more than 1 type: [_doc, movie]"
In ES 7, there are no more types. You need to do it like this.
First, create the index:
curl -H "Content-Type: application/json" -XPUT 127.0.0.1:9200/movies -d '
{
"mappings": {
"properties": {
"year": {"type":"date"}
}
}
}'
Then, index your document:
curl -H "Content-Type: application/json" -XPUT 127.0.0.1:9200/movies/_doc/109487 -d '
{
"genre":["IMAX", "Sci-Fi"],
"title":"Intersteller",
"year":2014
}'

Updating Elasticsearch 5.6 index type with new mapping

I'd like to add a new mapping to an index I already have, I'm trying with
curl -H 'Content-Type: application/json' -H 'Accept: application/json' -X PUT http://localhost:9200/videos_development/_mapping/video -d '
"video":{
"properties":{
"id_lookup":"text"
}
}
'
but it's returning
{"error":{"root_cause":[{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}],"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"},"status":500}%
and I really have no idea what it means...
Can anybody help?
Your JSON is malformed, you need opening and closing curly braces + you're also missing the type in the field definition:
curl -H 'Content-Type: application/json' -H 'Accept: application/json' -X PUT http://localhost:9200/videos_development/_mapping/video -d '{
"video":{
"properties":{
"id_lookup": {
"type": "text"
}
}
}
}'

Unable to create visualization using curl command in elaticearch

I am trying to create visualization using curl command. I am using elasticsearch 6.2.3. I am able to create the same in elasticsearch 5.6.8.
I am using this command
curl -XPUT http://localhost:9200/.kibana/visualization/vis1 -H 'Content-Type: application/json' -d #vis1.json
It is showing this error :
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [.kibana] as the final mapping would have more than 1 type: [visualization, doc]"}],"type":"illegal_argument_exception","reason":"Rejecting mapping update to [.kibana] as the final mapping would have more than 1 type: [visualization, doc]"},"status":400}
Contents of vis1.json:
{
"title": "vis1",
"visState": "{\"title\":\"vis1\",\"type\":\"table\",\"params\":{\"perPage\":10,\"showMeticsAtAllLevels\":false,\"showPartialRows\":false,\"showTotal\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"split\",\"params\":{\"field\":\"UsageEndDate\",\"interval\":\"M\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{},\"row\":false}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"ProductName.keyword\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}]}",
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"4eb9f840-3969-11e8-ae19-552e148747c3\",\"filter\":[],\"query\":{\"language\":\"lucene\",\"query\":\"\"}}"
}
}
This is working fine in elasticearch 5.6.8 but not in 6.2.3.
Thanks in Advance.
In Kibana 6, the mapping of the .kibanaindex has changed in order to satisfy the upcoming "one mapping per index" breaking change.
You can try this way instead:
curl -XPUT http://localhost:9200/.kibana/doc/visualization:vis1 -H 'Content-Type: application/json' -d #vis1.json
Also the vis1.json file needs to be changed a little bit (the content needs to be moved to the visualization sub-section), like this:
{
"type": "visualization",
"updated_at": "2018-04-10T10:00:00.000Z",
"visualization": {
"title": "vis1",
"visState": "{\"title\":\"vis1\",\"type\":\"table\",\"params\":{\"perPage\":10,\"showMeticsAtAllLevels\":false,\"showPartialRows\":false,\"showTotal\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"split\",\"params\":{\"field\":\"UsageEndDate\",\"interval\":\"M\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{},\"row\":false}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"ProductName.keyword\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}]}",
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"4eb9f840-3969-11e8-ae19-552e148747c3\",\"filter\":[],\"query\":{\"language\":\"lucene\",\"query\":\"\"}}"
}
}
}

Graph-Aided Search Result filtering example

I've duplicated the Movie database of Neo4j on Elasticsearch and it's indexed with the index nodes. It has two types Movie and Person. I am trying to make a simple Result Filtering with Graph-Aided Search using this curl command line:
curl -X GET localhost:9200/nodes/_search?pretty -d '{
"query": {
"match_all" : {}
},
"gas-filter": {
"name": "SearchResultCypherfilter",
"query": "MATCH (p:Person)-[ACTED_IN]->(m:Movie) WHERE p.name= 'Parker Posey' RETURN m.uuid as id",
"ShouldExclude": true,
"protocol": "bolt"
}
}'
But I get as results all the 171 nodes of both types Movie and Person in my index nodes. However, as my query says I want to return only the type Movie by its title. So basically it doesn't look to the gas-filter part.
Also when I put false as the value of shouldExclude I am getting the same results.
[UPDATE]
I tried the suggestion of #Tezra, I am now returning only the id uuid and I put shouldExclude instead of exclude but still getting the same results.
I am working with:
Elasticsearch 2.3.2
graph-aided-search-2.3.2.0
Neo4j-community 2.3.2.10
graphaware-uuid-2.3.2.37.7
graphaware-server-community-all-2.3.2.37
graphaware-neo4j-to-elasticsearch-2.3.2.37.1
Result that should be returning:
The uuid of the movie titled You've Got Mail.
I tried to follow this tutorial for the configuration, and I found out that index.gas.enable had the value false so I changed it and finished the configuration just like in the tutorial:
mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.neo4j.hostname=http://localhost:7474
{"acknowledged":true}
mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.enable=true
{"acknowledged":true}
mac$ curl -XPUT http://localhost:9200/indexname/_settings?index.gas.neo4j.user=neo4j
{"acknowledged":true}
mac$ curl -XPUT http://localhost:9200/indexname/_settings?index.gas.neo4j.password=mypassword
{"acknowledged":true}
After that I tried to add the settings of boltHostname and bolt.secure but it didn't work and I had this error:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't update non dynamic settings[[index.gas.neo4j.boltHostname]] for open indices [[nodes]]"}],"type":"illegal_argument_exception","reason":"Can't update non dynamic settings[[index.gas.neo4j.boltHostname]] for open indices [[nodes]]"},"status":400}
So I closed my index to configure it and then opened it again:
mac$ curl -XPOST http://localhost:9200/nodes/_close
{"acknowledged":true}
mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.neo4j.boltHostname=bolt://localhost:7687
{"acknowledged":true}
mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.neo4j.bolt.secure=false
{"acknowledged":true}
mac$ curl -XPOST http://localhost:9200/nodes/_open
{"acknowledged":true}
After finishing the configuration I tried again on Postman the same gas-filter query that I was executing with curl and now I am getting this error:
{
"error": {
"root_cause": [
{
"type": "runtime_exception",
"reason": "Failed to parse a search response."
}
],
"type": "runtime_exception",
"reason": "Failed to parse a search response.",
"caused_by": {
"type": "client_handler_exception",
"reason": "java.net.ConnectException: Connection refused (Connection refused)",
"caused_by": {
"type": "connect_exception",
"reason": "Connection refused (Connection refused)"
}
}
},
"status": 500
}
I don't know which connection the error is talking about. I am sure I passed the correct password of Neo4j in the configuration. I've even stopped and restarted again the servers of Elasticsearch and Neo4j but still the same errors.
The settings of my index nodes looks like this:
{
"nodes" : {
"settings" : {
"index" : {
"gas" : {
"enable" : "true",
"neo4j" : {
"hostname" : "http://localhost:7474",
"password" : "neo4j.",
"bolt" : {
"secure" : "false"
},
"boltHostname" : "bolt://localhost:7687",
"user" : "neo4j"
}
},
"creation_date" : "1495531307760",
"number_of_shards" : "5",
"number_of_replicas" : "1",
"uuid" : "SdrmQKhXQmyGKHmOh_xhhA",
"version" : {
"created" : "2030299"
}
}
}
}
}
Any ideas?
I figured out that the Connection refused exception that I was getting was because of the Wifi. So I had to disconnect from the internet to make it work. I know it is not the perfect solution. So if anyone found a better way of doing it please share it here.

Resources