ElasticSearch mongo always 0 hits - elasticsearch

I'm trying to make ElasticSearch run over my Mongodb server, everything looks fine, but every query I do returns me 0 hits. Always.
My installation and configuration log:
Installed Mongodb 2.6.4
Up and running. No problems here. I have like 7000 products inside "products" collection.
.2 Created replica set.
Confirmed with rs.status() on Mongo shell that it's created and it's
the primary replica Changed mongod.conf with resplSet = rs0
oplogSize=100
.3. Restarted MongoDB
.4. Initiated the replica set
On mongo shell rs.initiate(). Everything fine.
.5. Installed ElasticSearch 1.3.2
{
"status": 200,
"name": "Franz Kafka",
"version": {
"number": "1.3.2",
"build_hash": "dee175dbe2f254f3f26992f5d7591939aaefd12f",
"build_timestamp": "2014-08-13T14:29:30Z",
"build_snapshot": false,
"lucene_version": "4.9"
},
"tagline": "You Know, for Search"
}
.6. Installed Mapper plugin
.7. Installed River plugin
.8. Create index
curl -XPUT 'http://localhost:9200/indexprueba/products/_meta?pretty=true' -d '{
"type": "mongodb",
"mongodb": {
"db": "test",
"collection": "products"
},
"index": {
"name": "probando1",
"type": "products"
}
}'
it returns:
{
"_index": "indexprueba",
"_type": "products",
"_id": "_meta",
"_version": 1,
"created": true
}
--------EDIT---------
8.5 Restore database
I didn't do this. Once I've created the index, I restore my database with mongorestore and this is what I get:
connected to: 127.0.0.1:27017
2014-09-08T08:17:17.773+0000 /var/backup/bikebud/products.bson
2014-09-08T08:17:17.773+0000 going into namespace [test.products]
Restoring to test.products without dropping. Restored data will be inserted without raising errors; check your server log
6947 objects found
2014-09-08T08:17:18.043+0000 Creating index: { key: { _id: 1 }, name: "_id_", ns: "test.products" }
2014-09-08T08:17:18.456+0000 /var/backup/bikebud/retailers.bson
2014-09-08T08:17:18.457+0000 going into namespace [test.retailers]
Restoring to test.retailers without dropping. Restored data will be inserted without raising errors; check your server log
20 objects found
2014-09-08T08:17:18.457+0000 Creating index: { key: { _id: 1 }, name: "_id_", ns: "test.retailers" }
So I understand from here that my indexes are created and linked to the database
--------EDIT---------
.9. Create simple query
curl -XGET `'http://127.0.0.1:9200/indexprueba/_search?pretty=true&q=*:*'`
Always returns:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
----------------EDIT-------------------
After the edit, this is what I get:
{
"took": 14,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1.0,
"hits": [
{
"_index": "testindex1",
"_type": "products",
"_id": "1",
"_score": 1.0,
"_source": {
"type": "mongodb",
"mongodb": {
"servers": [
{
"host": "127.0.0.1",
"port": 27017
}
],
"options": {
"secondary_read_preference": true
},
"db": "test",
"collection": "products"
}
}
}
]
}
}
So now I get hits, but is the index itself. I was expecting to get all products from my database. I start to think I don't understand at all what elasticsearch does. Any clue??
----------------EDIT-------------------
I don't know what I'm missing here. Please, any advice?
----------------EDIT-------------------
It looks like it was a version problem. I have to downgrade ES to 1.2.2 (I'm using 1.3.2).
"Resolved"

Related

Error while remote indexing with elasticsearch

I'am trying to move from an ES cluster to another, in order to plan an update.
The both are same version (6.4). To achieve this, i'am using this command :
curl -XPOST -H "Content-Type: application/json" http://new_cluster/_reindex -d#reindex.json
And the reindex.json, is looking like this :
{
"source": {
"remote": {
"host": "http://old_cluster:9199"
},
"index": "megabase.33.2",
"query": {
"match_all": {}
}
},
"dest": {
"index": "megabase.33.2"
}
}
I whitelisted one the new cluster the old cluster, and its works but i can't go to the end of the migration of data, because i have this error, and i don't understand what it means here :
{
"took":1762,
"timed_out":false,
"total":8263428,
"updated":5998,
"created":5001,
"deleted":0,
"batches":11,
"version_conflicts":0,
"noops":0,
"retries":{
"bulk":0,
"search":0
},
"throttled_millis":0,
"requests_per_second":-1.0,
"throttled_until_millis":0,
"failures":[
{
"index":"megabase.33.2",
"type":"persona",
"id":"noYOA3IBTWbNbLJUqk6T",
"cause":{
"type":"mapper_parsing_exception",
"reason":"failed to parse [adr_inse]",
"caused_by":{
"type":"illegal_argument_exception",
"reason":"For input string: \"2A004\""
}
},
"status":400
}
]
}
The record in the original cluster look like this :
{
"took": 993,
"timed_out": false,
"_shards": {
"total": 4,
"successful": 4,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0,
"hits": [
{
"_index": "megabase.33.2",
"_type": "persona",
"_id": "noYOA3IBTWbNbLJUqk6T",
"_score": 0,
"_source": {
"address": "Obfucated",
"adr_inse": "2A004",
"age": 10,
"base": "Obfucated",
"city": "Obfucated",
"cp": 20167,
"email_md5": "Obfucated",
"fraicheur": "2020-01-12T19:39:04+01:00",
"group": 1,
"latlon": "Obfucated",
"partner": "Obfucated",
"partnerbase": 2,
"sex": 2,
"sms_md5": "Obfucated"
}
}
]
}
}
Any clue on what i'am doing wrong ?
Thanks a lot
Found out, the mapping is not well created when using only the reindex method.
So i drop the new indice, recreate mapping using elasticdump :
elasticdump --input=http://oldcluster/megabase.33.2 --output=http://newcluster/megabase.33.2 --type=mapping
Then run the previous script, everything works flawless (and was rather quick)

No results from search when passing more than one parameter in user metadata

I want to apply document level security in elastic, but once I provide more than one value in user metadata I get no matches.
I am creating a role and a user in elastic and passing values inside user metadata to the role on whose basis the search should happen. It works fine if I give one value.
For creating role:
PUT _xpack/security/role/my_policy
{
"indices": [{
"names": ["my_index"],
"privileges": ["read"],
"query": {
"template": {
"source": "{\"bool\": {\"filter\": [{\"terms_set\": {\"country_name\": {\"terms\": {{#toJson}}_user.metadata.country_name{{/toJson}},\"minimum_should_match_script\":{\"source\":\"params.num_terms\"}}}}]}}"
}
}
}]
}
And for user:
PUT _xpack/security/user/jack_black
{
"username": "jack_black",
"password":"testtest",
"roles": ["my_policy"],
"full_name": "Jack Black"
"email": "jb#tenaciousd.com",
"metadata": {
"country_name": ["india" , "japan"]
}
}
I expect the output to be results for india and japan only. If the user searches for anything else they should get no results.
However, I do not see any results at all:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}

How to insert data to elastic search from search query

I try to copy some data from one elastic db to another elasticsearch db, is there any way to insert data from query results?
Example of results:
{
"took": 29,
"timed_out": false,
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"hits": {
"total": 572,
"max_score": 1,
"hits": [
{
"_index": "ref",
"_type": "dic",
"_id": "12345",
"_score": 1,
"_source": {
"name": "Test name"
}
},
...
]
}
In each db mapping is equals.
I fork some project witch make a bulk data for elasticsearch, json-to-es-bulk
I it's compability with 5.6 es version, you can use it in 2 variants:
`node index.js -f inputdata.json --index newIndexName --type newIndexType --rewrite true`
or
`node index.js -f inputdata.json --index --type --rewrite false`
after run, you'll see a file request-data.txt, just use it to
POST /_bulk
[request-data.txt content]
Input data json file must contains an array of search hits, like this:
[
{
"_index": "oldIndexName",
"_type": "oldIndexName",
"_id": "SOME_ID-HiTfm",
"_score": 1,
"_source": {
"orderNumber": "2984",
"refId": "SOME_VALUE"
}
},
...
]

Elasticsearch 2.3 - delete documents by query

I'm using elasticsearch 2.3 & Sense and trying to delete documents by query.
I refer to these docs:
https://www.elastic.co/guide/en/elasticsearch/plugins/current/delete-by-query-usage.html
Request
DELETE /monitors/monitor/_query
{
"term": { "ProcessName" : "myProcName" }
}
Response
{
"found": false,
"_index": "monitors",
"_type": "monitor",
"_id": "_query",
"_version": 11,
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
}
}
As you can see, i'm not getting any results even though I have ProcessName named "myProcName".
Response also tells that the engine looks for _id equals to _query.
EDIT 1:
Even when sending request:
DELETE /monitors/monitor/_query
{
"query": {
"term": { "ProcessName" : "tibapp_qflowfile" }
}
}
I'm getting response:
{
"found": false,
"_index": "monitors",
"_type": "monitor",
"_id": "_query",
"_version": 1,
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
}
}
The output you're getting means that you haven't installed the delete-by-query plugin, which isn't installed by default.
Do that first, restart your node and it will work afterwards
bin/plugin install delete-by-query
FYI - The Plugin [delete-by-query] is incompatible with Elasticsearch [2.3.5]. Was designed for version [2.3.4]

Elasticsearch get by id doesn't work but document exists

I'm seeing weird behaviours with ids on elasticsearch 1.2.0 (recently upgraded from 1.0.1).
A search retrieves my document, showing the correct value for _id:
[terminal]
curl 'myServer:9200/global/_search?q=someField:something
result is
{
"took": 79,
"timed_out": false,
"_shards": {
"total": 12,
"successful": 12,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 17.715034,
"hits": [
{
"_index": "global",
"_type": "user",
"_id": "7a113e4f-44de-3b2b-a3f1-fb881da1b00a",
...
}
]
}
}
But a direct lookup on id doesn't:
[terminal]
curl 'myServer:9200/global/user/7a113e4f-44de-3b2b-a3f1-fb881da1b00a'
result is
{
"_index": "global",
"_type": "user",
"_id": "7a113e4f-44de-3b2b-a3f1-fb881da1b00a",
"found": false
}
It seems that this is on documents that have previously been updated using custom scripting.
Any ideas?
I think you should upgrade to 1.2.1
Due to release notes (http://www.elasticsearch.org/blog/elasticsearch-1-2-1-released/) there are some problems, especially with get:
`There was a routing bug in Elasticsearch 1.2.0 that could have a number of bad side effects on the cluster. Possible side effects include:
documents that were indexed prior to the upgrade to 1.2.0 may not be accessible via get. A search would find these documents, but not a direct get of the document by ID.`

Resources