I'am trying to move from an ES cluster to another, in order to plan an update.
The both are same version (6.4). To achieve this, i'am using this command :
curl -XPOST -H "Content-Type: application/json" http://new_cluster/_reindex -d#reindex.json
And the reindex.json, is looking like this :
{
"source": {
"remote": {
"host": "http://old_cluster:9199"
},
"index": "megabase.33.2",
"query": {
"match_all": {}
}
},
"dest": {
"index": "megabase.33.2"
}
}
I whitelisted one the new cluster the old cluster, and its works but i can't go to the end of the migration of data, because i have this error, and i don't understand what it means here :
{
"took":1762,
"timed_out":false,
"total":8263428,
"updated":5998,
"created":5001,
"deleted":0,
"batches":11,
"version_conflicts":0,
"noops":0,
"retries":{
"bulk":0,
"search":0
},
"throttled_millis":0,
"requests_per_second":-1.0,
"throttled_until_millis":0,
"failures":[
{
"index":"megabase.33.2",
"type":"persona",
"id":"noYOA3IBTWbNbLJUqk6T",
"cause":{
"type":"mapper_parsing_exception",
"reason":"failed to parse [adr_inse]",
"caused_by":{
"type":"illegal_argument_exception",
"reason":"For input string: \"2A004\""
}
},
"status":400
}
]
}
The record in the original cluster look like this :
{
"took": 993,
"timed_out": false,
"_shards": {
"total": 4,
"successful": 4,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0,
"hits": [
{
"_index": "megabase.33.2",
"_type": "persona",
"_id": "noYOA3IBTWbNbLJUqk6T",
"_score": 0,
"_source": {
"address": "Obfucated",
"adr_inse": "2A004",
"age": 10,
"base": "Obfucated",
"city": "Obfucated",
"cp": 20167,
"email_md5": "Obfucated",
"fraicheur": "2020-01-12T19:39:04+01:00",
"group": 1,
"latlon": "Obfucated",
"partner": "Obfucated",
"partnerbase": 2,
"sex": 2,
"sms_md5": "Obfucated"
}
}
]
}
}
Any clue on what i'am doing wrong ?
Thanks a lot
Found out, the mapping is not well created when using only the reindex method.
So i drop the new indice, recreate mapping using elasticdump :
elasticdump --input=http://oldcluster/megabase.33.2 --output=http://newcluster/megabase.33.2 --type=mapping
Then run the previous script, everything works flawless (and was rather quick)
Related
I want to apply document level security in elastic, but once I provide more than one value in user metadata I get no matches.
I am creating a role and a user in elastic and passing values inside user metadata to the role on whose basis the search should happen. It works fine if I give one value.
For creating role:
PUT _xpack/security/role/my_policy
{
"indices": [{
"names": ["my_index"],
"privileges": ["read"],
"query": {
"template": {
"source": "{\"bool\": {\"filter\": [{\"terms_set\": {\"country_name\": {\"terms\": {{#toJson}}_user.metadata.country_name{{/toJson}},\"minimum_should_match_script\":{\"source\":\"params.num_terms\"}}}}]}}"
}
}
}]
}
And for user:
PUT _xpack/security/user/jack_black
{
"username": "jack_black",
"password":"testtest",
"roles": ["my_policy"],
"full_name": "Jack Black"
"email": "jb#tenaciousd.com",
"metadata": {
"country_name": ["india" , "japan"]
}
}
I expect the output to be results for india and japan only. If the user searches for anything else they should get no results.
However, I do not see any results at all:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
I am working on a project where I would like to query the wikipedia database so I could bring some results ... I am using a Javascript application and Jquery with a self suggestion field as the user types a list of words should appear ....
For this to happen I need to create a query in elasticsearch by bringing all categories and only them into the results.
Type a SQL = SELECT Field FROM Table Group By Field ..
How would I do this?
My Test is basic and return all objects....
GET _search
{
"query": {
"query_string": {
"fields": ["category"],
"query": "*"
}
}
}
Part of result is:
{
"took": 34,
"timed_out": false,
"_shards": {
"total": 13,
"successful": 13,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2117924,
"max_score": 1,
"hits": [
{
"_index": "ptwikionary_content",
"_type": "page",
"_id": "41115",
"_score": 1,
"_source": {
"redirect": [],
"template": [
"Predefinição:-pt-",
"Predefinição:cabeçalho-idioma",
"Predefinição:nome categoria",
"Predefinição:pt",
"Predefinição:nome língua",
"Predefinição:flex.pt",
"Predefinição:link preto",
"Predefinição:paroxítona",
"Predefinição:gramática",
"Predefinição:gramática/core",
"Predefinição:etimologia",
"Predefinição:escopo",
"MediaWiki:Log"
],
"content_model": "wikitext",
"heading": [
"Português",
"Adjetivo",
"Antônimo",
"Etimologia"
],
"source_text": """
={{-pt-}}=
==Adjetivo==
{{flex.pt|s=desumilde|p=desumildes}}
{{paroxítona|de|su|mil|de}}, {{gramática|c2g}}
# que não é [[humilde]]
===Antônimo===
* [[humilde]]
=={{etimologia|pt}}==
: {{escopo|Morfologia}} [[des-]] + [[humilde]].
[[Categoria:Adjetivo (Português)]]
""",
"version_type": "external",
"wiki": "ptwiktionary",
"auxiliary_text": [
" Singular Plural Masculino desumilde desumildes Feminino"
],
"language": "pt",
"title": "desumilde",
"version": 2491983,
"external_link": [],
"namespace_text": "",
"namespace": 0,
"text_bytes": 274,
"incoming_links": 2,
"text": "de.su.mil.de, comum aos dois géneros que não é humilde humilde (Morfologia) des- + humilde.",
"category": [
"!Entrada (Português)",
"Polissílabo (Português)",
"Paroxítona (Português)",
"Entrada com etimologia (Português)",
"Adjetivo (Português)"
],
"defaultsort": false,
"outgoing_link": [
"desumilde",
"desumildes",
"des-",
"feminino",
"humilde",
"masculino",
"plural",
"português",
"singular",
"Categoria:Português"
],
"timestamp": "2018-03-29T21:53:29Z",
"popularity_score": 0.000065252908710355
}
},
{
How i Get category list from elasticsearch??? Type Sql :
Use aggregation for this. Here's an example:
GET _search
{
"aggs" : {
"categories" : {
"terms" : { "field" : "category" }
}
}
}
I try to copy some data from one elastic db to another elasticsearch db, is there any way to insert data from query results?
Example of results:
{
"took": 29,
"timed_out": false,
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"hits": {
"total": 572,
"max_score": 1,
"hits": [
{
"_index": "ref",
"_type": "dic",
"_id": "12345",
"_score": 1,
"_source": {
"name": "Test name"
}
},
...
]
}
In each db mapping is equals.
I fork some project witch make a bulk data for elasticsearch, json-to-es-bulk
I it's compability with 5.6 es version, you can use it in 2 variants:
`node index.js -f inputdata.json --index newIndexName --type newIndexType --rewrite true`
or
`node index.js -f inputdata.json --index --type --rewrite false`
after run, you'll see a file request-data.txt, just use it to
POST /_bulk
[request-data.txt content]
Input data json file must contains an array of search hits, like this:
[
{
"_index": "oldIndexName",
"_type": "oldIndexName",
"_id": "SOME_ID-HiTfm",
"_score": 1,
"_source": {
"orderNumber": "2984",
"refId": "SOME_VALUE"
}
},
...
]
I'm using elasticsearch 2.3 & Sense and trying to delete documents by query.
I refer to these docs:
https://www.elastic.co/guide/en/elasticsearch/plugins/current/delete-by-query-usage.html
Request
DELETE /monitors/monitor/_query
{
"term": { "ProcessName" : "myProcName" }
}
Response
{
"found": false,
"_index": "monitors",
"_type": "monitor",
"_id": "_query",
"_version": 11,
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
}
}
As you can see, i'm not getting any results even though I have ProcessName named "myProcName".
Response also tells that the engine looks for _id equals to _query.
EDIT 1:
Even when sending request:
DELETE /monitors/monitor/_query
{
"query": {
"term": { "ProcessName" : "tibapp_qflowfile" }
}
}
I'm getting response:
{
"found": false,
"_index": "monitors",
"_type": "monitor",
"_id": "_query",
"_version": 1,
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
}
}
The output you're getting means that you haven't installed the delete-by-query plugin, which isn't installed by default.
Do that first, restart your node and it will work afterwards
bin/plugin install delete-by-query
FYI - The Plugin [delete-by-query] is incompatible with Elasticsearch [2.3.5]. Was designed for version [2.3.4]
I'm trying to make ElasticSearch run over my Mongodb server, everything looks fine, but every query I do returns me 0 hits. Always.
My installation and configuration log:
Installed Mongodb 2.6.4
Up and running. No problems here. I have like 7000 products inside "products" collection.
.2 Created replica set.
Confirmed with rs.status() on Mongo shell that it's created and it's
the primary replica Changed mongod.conf with resplSet = rs0
oplogSize=100
.3. Restarted MongoDB
.4. Initiated the replica set
On mongo shell rs.initiate(). Everything fine.
.5. Installed ElasticSearch 1.3.2
{
"status": 200,
"name": "Franz Kafka",
"version": {
"number": "1.3.2",
"build_hash": "dee175dbe2f254f3f26992f5d7591939aaefd12f",
"build_timestamp": "2014-08-13T14:29:30Z",
"build_snapshot": false,
"lucene_version": "4.9"
},
"tagline": "You Know, for Search"
}
.6. Installed Mapper plugin
.7. Installed River plugin
.8. Create index
curl -XPUT 'http://localhost:9200/indexprueba/products/_meta?pretty=true' -d '{
"type": "mongodb",
"mongodb": {
"db": "test",
"collection": "products"
},
"index": {
"name": "probando1",
"type": "products"
}
}'
it returns:
{
"_index": "indexprueba",
"_type": "products",
"_id": "_meta",
"_version": 1,
"created": true
}
--------EDIT---------
8.5 Restore database
I didn't do this. Once I've created the index, I restore my database with mongorestore and this is what I get:
connected to: 127.0.0.1:27017
2014-09-08T08:17:17.773+0000 /var/backup/bikebud/products.bson
2014-09-08T08:17:17.773+0000 going into namespace [test.products]
Restoring to test.products without dropping. Restored data will be inserted without raising errors; check your server log
6947 objects found
2014-09-08T08:17:18.043+0000 Creating index: { key: { _id: 1 }, name: "_id_", ns: "test.products" }
2014-09-08T08:17:18.456+0000 /var/backup/bikebud/retailers.bson
2014-09-08T08:17:18.457+0000 going into namespace [test.retailers]
Restoring to test.retailers without dropping. Restored data will be inserted without raising errors; check your server log
20 objects found
2014-09-08T08:17:18.457+0000 Creating index: { key: { _id: 1 }, name: "_id_", ns: "test.retailers" }
So I understand from here that my indexes are created and linked to the database
--------EDIT---------
.9. Create simple query
curl -XGET `'http://127.0.0.1:9200/indexprueba/_search?pretty=true&q=*:*'`
Always returns:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
----------------EDIT-------------------
After the edit, this is what I get:
{
"took": 14,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1.0,
"hits": [
{
"_index": "testindex1",
"_type": "products",
"_id": "1",
"_score": 1.0,
"_source": {
"type": "mongodb",
"mongodb": {
"servers": [
{
"host": "127.0.0.1",
"port": 27017
}
],
"options": {
"secondary_read_preference": true
},
"db": "test",
"collection": "products"
}
}
}
]
}
}
So now I get hits, but is the index itself. I was expecting to get all products from my database. I start to think I don't understand at all what elasticsearch does. Any clue??
----------------EDIT-------------------
I don't know what I'm missing here. Please, any advice?
----------------EDIT-------------------
It looks like it was a version problem. I have to downgrade ES to 1.2.2 (I'm using 1.3.2).
"Resolved"