ElasticSearch newbie - creating the first index - elasticsearch

Have a client that thinks this will be awesome to use to crawl volumes of 10MB log files. I'd agree having read up on it, but seem stuck on creating my first index. So far:
♦ Installed ElasticSearch 1.2.1 on Windows 8, and Curl 7.3.7 (HTTP for SSL, here: paehl.com/open_source/?CURL_7.37.0)
♦ Successful in a response from ES (status 200, hurrah, on localhost, 127.0.0.1 or my machine's IP all good with a 200 status)
♦ When I try to create an index, however, no love. Been trying to follow steps from both the Joel Abrahamsson's ElasticSearch 101 (joelabrahamsson.com/elasticsearch-101/) and Park's "ES Cookbook". Errors range, but most recently and shortest are on par with these:
C:\>curl -XPUT 'http://localhost:9200/blog/user/dilbert' -d '{ "name" : "Dilbert Brown" }'
curl: (1) Protocol 'http not supported or disabled in libcurl
curl: (6) Could not resolve host: name
curl: (7) Failed to connect to port 80: Connection refused
curl: (6) Could not resolve host: Dilbert Brown
curl: (3) [globbing] unmatched close brace/bracket in column 1
Earlier messages were far more wordy:
C:\>curl -XPUT "http://localhost:9200/test1/test/1" -d' { "title" : "Godfather", "director" : "Coppola", "year" : 1972 }'
{"error":"MapperParsingException[failed to parse]; nested: ElasticsearchParseException[Failed to derive xcontent from (offset=0, length=1): [39]]; ","status":400}curl: (3) [globbing] unmatched brace in column 1
curl: (6) Could not resolve host: title
curl: (7) Failed to connect to port 80: Connection refused
curl: (6) Could not resolve host: Godfather,
curl: (6) Could not resolve host: director
curl: (7) Failed to connect to port 80: Connection refused
curl: (6) Could not resolve host: Coppola,
curl: (6) Could not resolve host: year
curl: (7) Failed to connect to port 80: Connection refused
curl: (6) Could not resolve host: 1972
curl: (3) [globbing] unmatched close brace/bracket in column 1
It feels like I'm overlooking something basic, but as a newbie to ES, curl and JSON, I'm puzzled -- and at least from my seat, the brace/brackets appear balanced.Suggestions?

It's a problem with the cmd of Windows.
You can fix it by changing:
curl -XPUT 'http://localhost:9200/blog/user/dilbert' -d '{ "name" : "Dilbert Brown" }'
to:
curl -XPUT "http://localhost:9200/blog/user/dilbert" -d "{ """name""" : """Dilbert Brown""" }"
Update: (for windows 64 bit)
curl -XPUT "http://localhost:9200/blog/user/dilbert" -d "{ \"name\" : \"Dilbert Brown\" }"

It seems windows does not like the single quote thing around the url. I cannot test it for you but try it with double quotes around the url.
I got this info from the following blogpost:
http://bartwullems.blogspot.nl/2013/08/curl-1-protocol-not-supported-or.html

Create a local file with the json request message as indicated in the elasticsearch tutorial, go to this directory and execute the curl command, referencing this file:
curl -XPUT http://localhost:9200/shakespeare --data-binary "#_01_specify_schema.json"
where _01_specify_schema.json is a file containig:
{
"mappings" : {
"_default_" : {
"properties" : {
"speaker" : {"type": "string", "index" : "not_analyzed" },
"play_name" : {"type": "string", "index" : "not_analyzed" },
"line_id" : { "type" : "integer" },
"speech_number" : { "type" : "integer" }
}}}}
Take care to add the "#" symbol before the name of the referenced file

Related

{"error":"bad_request","reason":"invalid UTF-8 JSON"}

I'm trying using CouchDB and when I try to add a document it give me some error.
This is what I do:
curl -X PUT 'http://user:pass#127.0.0.1:5984/test/Movies -d {"_id" : "1", "Title" : "Toy Story (1995)", "Genres" : "Adventure|Animation|Children|Comedy|Fantasy"}'
This is what prompt give me:
{"error":"bad_request","reason":"invalid UTF-8 JSON"}
curl: (3) Bad URL, colon is first character
curl: (6) Could not resolve host: 1,
curl: (6) Could not resolve host: Title
curl: (3) Bad URL, colon is first character
curl: (3) [globbing] unmatched close brace/bracket in column 17
I tried to use escape like \"Title\" etc but it's the same.
I tried to insert "_id" : 1 and it doesn't work
And in the end I tried to insert only a genre. I don't know more what to do.
I've just tried and it works:
C:\>curl -X PUT "http://root:root#127.0.0.1:5984/test/Movies" -d "#prova.json"
It looks like There are missing single quotes (/Movies' -d '{) in your cURL request. Added missing single quotes in your cURL request and modified as follows.
root#312-nb-gqfcjm2# curl -X PUT 'http://user:pass#127.0.0.1:5984/test/Movies' -d '{
> "_id":"1",
> "Title":"Toy Story (1995)",
> "Genres":"Adventure|Animation|Children|Comedy|Fantasy"
> }'

adding snapshot to elasticsearch Windows

I have two snapshots that I want to insert into elasticsearch in path:
C:\Users\name\Downloads\book_backup\agg_example
C:\Users\name\Downloads\book_backup\search_example
which I properly listed in elasticsearch.yml
path.repo: ["C:\\Users\\olulo\\Downloads\\book_backup\\agg_example", "C:\\Users\\olulo\\Downloads\\book_backup\\search_example"]
my elasticsearch starts fine and creating a new index works too.
Now when I try to insert snapshot into my elasticsearch so I can work on it:
curl -XPUT "http://localhost:9200/movies/" -d '{"type":"fs", "settings":{"location":"C:\\Users\\name\\Downloads\\book_backup\\search_example", "compress":true}}'
It gives me:
curl: (1) Protocol "'http" not supported or disabled in libcurl
curl: (3) [globbing] unmatched brace in column 10
curl: (3) [globbing] unmatched close brace/bracket in column 14
In https://www.elastic.co/guide/en/elasticsearch/reference/5.2/modules-snapshots.html it seems like they've added a name at the end of path in location so I did
curl -XPUT "http://localhost:9200/movies/" -d '{"type":"fs", "settings":{"location":"C:\\Users\\name\\Downloads\\book_backup\\search_example\\test", "compress":true}}'
but still gives me same error.
following https://superuser.com/questions/1322567/http-not-supported-or-disabled-in-libcurl I've change everything to double quotes:
curl -XPUT "http://localhost:9200/_snapshot/javacafe" -d "{"type":"fs", "settings":{"location":"C:\\Users\\olulo\\Downloads\\book_backup\\search_example", "compress":true}}"
giving me :
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
Tried adding curl option as suggested Content-Type header [application/x-www-form-urlencoded] is not supported on Elasticsearch by
curl -XPUT "localhost:9200/_snapshot/javacafe" -H 'Content-Type: application/json' -d "{
"type":"fs",
"settings"{"location":"C:\\Users\\olulo\\Downloads\\book_backup\\search_example\\test", "compress":true}
}"
which outputs similar error with added statement at the end:
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}curl: (6) Could not resolve host: application
SOLVED : for windows you need to use backslash infront of double quotation mark inside {}:
curl -XPUT "localhost:9200/_snapshot/javacafe" -H "Content-Type: application/json" -d "{
\"type\":\"fs\",
\"settings\"{\"location\":"C:\\Users\\olulo\\Downloads\\book_backup\\search_example\\test", \"compress\":true}
}"
From my understanding windows use \ to consider " as it is. If so why not add backslash to all curl commands as well?
Try this:
curl -XPUT "localhost:9200/_snapshot/javacafe" -H 'Content-Type: application/json' -d '{
"type":"fs",
"settings" : {
"location":"C:\\Users\\olulo\\Downloads\\book_backup\\search_example\\test",
"compress":true
}
}'

how to pass arguments in curl command

i am getting the below error while executing the curl command in unix. Not sure if this is the right way to put the arguments in curl. Can you please advise on this?
curl -X POST -H Authorization:'Bearer AAEAAJ6ZNfGbzJkSuJ1o5rXLBec5Q' -H 'Content-Type: application/json' -d {"Filter": {"ClientName": "ABC","WorkflowName": "sk_lask"},"SortingName": "StartDate","SortingOrder": "Desc"} http://someaddress.com/api/status/search HTTP/1.1
error
curl: (3) [globbing] unmatched brace at pos 13
curl: (6) Could not resolve host: ABC,WorkflowName; Unknown error
curl: (3) [globbing] unmatched close brace/bracket at pos 9
curl: (6) Could not resolve host: StartDate,SortingOrder; Unknown error
curl: (3) [globbing] unmatched close brace/bracket at pos 5
You need to quote your header, payload and also remove HTTP/1.1 from end as this command:
curl -X POST -H 'Authorization: Bearer AAEAAJ6ZNfGbzJkSuJ1o5rXLBec5Q' \
-H 'Content-Type: application/json' \
-d '{"Filter": {"ClientName": "ABC","WorkflowName": "sk_lask"},"SortingName": "StartDate","SortingOrder": "Desc"}' \
'http://someaddress.com/api/status/search'

How to execute CURL post command in bash?

I am trying to execute curl post command through a bash script and I am facing lot of issues.
Here is the bash code I have:
#!/bin/bash
url='http://slc760.us.oracle.com:10601/crmCommonApi/resources/latest'
Content='Content-Type:application/vnd.oracle.adf.batch+json'
user="sales_representative"
payload='{ "parts": [{"id": "part1","path": "/latest/__ORACO__Inventory_c","operation": "create","payload": {"__ORACO__Product_Id2_c":204,"__ORACO__Facing_c":"20","__ORACO__Product_Id1_c":204,"__ORACO__Product_c": "Ghirardelli White Choco","__ORACO__ShelfStock_c":"10","__ORACO__Location_c" : "Aisle 2","Organization_Id___ORACO__Account_Inventory":"300100051199355"}},{"id": "part1","path": "/latest/__ORACO__Inventory_c","operation": "create","payload": {"__ORACO__Product_Id2_c":204,"__ORACO__Facing_c":"3","__ORACO__Product_Id1_c":204,"__ORACO__Product_c": "Ghirardelli White Chocolcate","__ORACO__ShelfStock_c":"4","__ORACO__Location_c" : "Aisle 2","Organization_Id___ORACO__Account_Inventory":"300100051199355"}}]}'
curl -X POST -H "Content-Type:application/vnd.oracle.adf.batch+json" \
--data $payload \
$url \
--user $user
When run throws bunch of errors:
curl: (6) Could not resolve host: "parts"
curl: (3) [globbing] bad range specification in column 2
curl: (6) Could not resolve host: "part1","path"
curl: (6) Could not resolve host: "
curl: (6) Could not resolve host: "create","payload"
curl: (3) [globbing] unmatched brace in column 1
curl: (6) Could not resolve host: "Ghirardelli
curl: (6) Could not resolve host: White
curl: (6) Could not resolve host: Choco","__ORACO__ShelfStock_c"
curl: (7) Failed to connect to port 80: Connection refused
curl: (6) Could not resolve host: "Aisle
curl: (3) [globbing] unmatched close brace/bracket in column 66
curl: (6) Could not resolve host: "part1","path"
curl: (6) Could not resolve host: "
curl: (6) Could not resolve host: "create","payload"
curl: (3) [globbing] unmatched brace in column 1
curl: (6) Could not resolve host: "Ghirardelli
curl: (6) Could not resolve host: White
curl: (6) Could not resolve host: Chocolcate","__ORACO__ShelfStock_c"
curl: (7) Failed to connect to port 80: Connection refused
curl: (6) Could not resolve host: "Aisle
curl: (3) [globbing] unmatched close brace/bracket in column 66
org.codehaus.jackson.JsonParseException: Unexpected end-of-input: expected close marker for OBJECT (from [Source: weblogic.servlet.internal.ServletInputStreamImpl#10514b51; line: 1, column: 0])
When executed seperated through command-line the command seems to work but same when I try to execute it in CURL throws the above error.I know I am missing with relative to the quotes/syntax.
Please try to change following line:
--data $payload
to
--data '$payload'
Try putting double quotes on your payload:
curl -X POST -H "Content-Type:application/vnd.oracle.adf.batch+json" \
--data "$payload" \
$url \
--user $user
That should work.

curl from list of URLs: curl splits out the line as arguments

I have this bash file
#!/bin/bash
while read LINE; do
curl $LINE
echo " $LINE"
done < $1
and the URLs are like this:
-v -X POST -H "Content-Type: application/json" -d '{"id": 1, "method": "something", "params": ["en", "1", "bob", "password"]}' https://HTTPAUTHUSER:HTTPAUTHPASS#someurl.com
But it seems like curl is getting
[11:35]:sh loop.sh tmp.txt
* Hostname was NOT found in DNS cache
* Could not resolve host: application
* Closing connection 0
curl: (6) Could not resolve host: application
* Rebuilt URL to: 1,/
* Hostname was NOT found in DNS cache
* Could not resolve host: 1,
* Closing connection 1
curl: (6) Could not resolve host: 1,
* Rebuilt URL to: "method":/
* Hostname was NOT found in DNS cache
* Could not resolve host: "method"
* Closing connection 2
curl: (6) Could not resolve host: "method"
* Rebuilt URL to: "something",/
* Hostname was NOT found in DNS cache
* Could not resolve host: "something",
* Closing connection 3
curl: (6) Could not resolve host: "something",
* Rebuilt URL to: "params":/
* Hostname was NOT found in DNS cache
I feel like I'm missing something really obvious. Any thoughts?
EDIT:
from comments below I added set -x and I see this
curl 'MYARGS'
when i have
curl "$LINE"
remove the quotes -> curl $LINE
change the curl command like this:
curl -v -X POST -H '"Content-Type:' 'application/json"' -d ''\''{"id":' 1, '"method":' '"something",' '"params":' '["en",' '"1",' '"bob",' '"password"]}'\''' https://HTTPAUTHUSER:HTTPAUTHPASS#someurl.com

Resources