I'm using elasticdump and got weird error
Mon, 14 Nov 2022 14:42:21 GMT | starting dump
Mon, 14 Nov 2022 14:42:22 GMT | got 10 objects from source elasticsearch (offset: 0)
Mon, 14 Nov 2022 14:42:22 GMT | sent 10 objects to destination file, wrote 10
Mon, 14 Nov 2022 14:42:22 GMT | Error Emitted => This and all future requests should be directed to the given URI.
Mon, 14 Nov 2022 14:42:22 GMT | Error Emitted => This and all future requests should be directed to the given URI.
Mon, 14 Nov 2022 14:42:22 GMT | Total Writes: 0
Mon, 14 Nov 2022 14:42:22 GMT | dump ended with error (get phase) => MOVED_PERMANENTLY: This and all future requests should be directed to the given URI.
It successfully moved 10 objects and stopped
--input-index is for a different use case.
Try with just --input like this
elasticdump --input=http://localhost/dev_index --output=test2.json
Related
I'm currently building a versioning pipeline that would automatically promote tags based on date. I am able to return all my tags with this command: git for-each-ref --format="%(refname:short) | %(creatordate)" "refs/tags/*"
Which outputs something like this:
v1.0.0 | Mon Jan 24 14:36:19 2022 -0600
v1.1.0 | Mon Jan 24 14:37:06 2022 -0600
v1.1.1 | Mon Jan 24 14:40:39 2022 -0600
v1.1.2 | Tue Jan 25 17:47:32 2022 +0000
v1.1.3 | Tue Jan 25 21:20:03 2022 +0000
v1.1.4 | Tue Jan 25 21:22:40 2022 +0000
v1.2.0 | Tue Jan 25 21:24:08 2022 +0000
v1.3.0 | Tue Jan 25 21:52:28 2022 +0000
I'm looking to take this output and filter it based on current date. So for example if I want all tags created after Jan 24, I'd need to filter this output to only return the tags created from the 25th and beyond.
I'm thinking awk could help with this but I'm uncertain.. Any tips would be great!
I am trying to import index from json file to elasticsearch server but it is failing.
Specifications:
elasticsearch : 4.10.3
elasticdump : 2.4.2
command I am using
elasticdump --input=/home/ubuntu/Files/stocks.json --output=http://localhost:9200/ --type=data`
My stocks.json file looks like
{"_index":"stocks","_type":"stock","_id":"AVhKm5L8FPDye23IuJqe","_score":1,"_source":{"name":"Sun Pharmaceutical Industries Ltd.","industry":"PHARMA","isin":"INE044A01036","symbol":"SUNPHARMA","tweet":"sun pharma' OR 'SUNPHARMA'"}}
{"_index":"stocks","_type":"stock","_id":"AVhKm5L8FPDye23IuJqV","_score":1,"_source":{"name":"Tata Steel Ltd.","industry":"METALS","isin":"INE081A01012","symbol":"TATASTEEL","tweet":"tata steel' OR 'TATASTEEL'"}}
{"_index":"stocks","_type":"stock","_id":"AVhKm5L7FPDye23IuJp2","_score":1,"_source":{"name":"ICICI Bank Ltd.","industry":"FINANCIAL SERVICES","isin":"INE090A01021","symbol":"ICICIBANK","tweet":"icici bank' OR 'ICICIBANK'"}}
I am getting following message
Sat, 07 Oct 2017 05:46:52 GMT | starting dump
Sat, 07 Oct 2017 05:46:52 GMT | got 100 objects from source file
(offset: 0)
Sat, 07 Oct 2017 05:46:52 GMT | sent 100 objects to destination
elasticsearch, wrote 0
Sat, 07 Oct 2017 05:46:52 GMT | got 0 objects from source file
(offset: 100)
Sat, 07 Oct 2017 05:46:52 GMT | Total Writes: 0
Sat, 07 Oct 2017 05:46:52 GMT | dump complete
I had used same json file before but somehow this is not working in this new server. I have installed elasticsearch, node recently in this server.
Thanks for help
J
I use an apache storm topology on a cluster of 8+1 machines. The date on these machines is not the same and we may have more than 5 minutes of difference.
preprod-storm-nimbus-01:
Thu Feb 25 16:20:30 GMT 2016
preprod-storm-supervisor-01:
Thu Feb 25 16:20:32 GMT 2016
preprod-storm-supervisor-02:
Thu Feb 25 16:20:32 GMT 2016
preprod-storm-supervisor-03:
Thu Feb 25 16:14:54 UTC 2016 <<-- this machine is very late :(
preprod-storm-supervisor-04:
Thu Feb 25 16:20:31 GMT 2016
preprod-storm-supervisor-05:
Thu Feb 25 16:20:17 GMT 2016
preprod-storm-supervisor-06:
Thu Feb 25 16:20:00 GMT 2016
preprod-storm-supervisor-07:
Thu Feb 25 16:20:31 GMT 2016
preprod-storm-supervisor-08:
Thu Feb 25 16:19:55 GMT 2016
preprod-storm-supervisor-09:
Thu Feb 25 16:20:30 GMT 2016
Question:
Is the storm topology affected by this non-synchronization?
Note: I know that synchronizing is better, but the sysadmins won't do it without proving them proofs/reasons that they have to do it. Do they really have to do it, "for the topology's sake" :) ?
Thanks
It depends on the computation you are doing... It might have an effect on your result if you do time based window operations. Otherwise, it doesn't matter.
For Storm as an execution engine it has no effect at all.
I am using elasticdump to dump data from local machine to the server. But my dumps always ended with this error:
...
Tue, 20 Oct 2015 22:56:35 GMT | sent 100 objects to destination elasticsearch, wrote 100
Tue, 20 Oct 2015 22:56:35 GMT | got 100 objects from source elasticsearch (offset: 21200)
Tue, 20 Oct 2015 22:56:36 GMT | Error Emitted => read ECONNRESET
Tue, 20 Oct 2015 22:56:36 GMT | Total Writes: 21200
Tue, 20 Oct 2015 22:56:36 GMT | dump ended with error (set phase) => Error: read ECONNRESET
...
How should I solve this problem?
Is there a better way to dump data from local machine to the server? Thanks in advance!
It sounds like your issue is being caused by the elasticdump opening too many sockets to your elasticsearch cluster. You can use the --maxSockets option to limit the number of sockets opened.
For example:
$ elasticdump --input http://192.168.2.222:9200/index1 --output http://192.168.2.222:9200/index2 --type=data --maxSockets=5
You can find a detailed explanation of the issue here:
https://github.com/taskrabbit/elasticsearch-dump/issues/98
there are 200 records,for example:
[
{time:"Thu Nov 07 2013 13:09:08",value:"10"},
{time:"Thu Nov 07 2013 11:09:08",value:"30"},
{time:"Thu Nov 07 2013 11:09:08",value:"25"},
....more
{time:"Thu Nov 06 2013 10:09:08",value:"65"},
{time:"Tue Aug 06 2013 16:54:31",value:"25"},
{time:"Tue Aug 06 2013 16:54:31",value:"45"},
]
there are one or two unique recodes that the time is too early.
when i draw a line use time as xAxis, beacuse of the unique recodes ( {time:"Tue Aug 06 2013 16:54:31",value:"25"},{time:"Tue Aug 06 2013 16:54:31",value:"45"}) ,the line has a blank at between Nov to Aug .
how can i deal with the records what is unique and several .
Any help is appreciated.
Just add an unique id field to your data if you want to draw both or remove one (and display an average value or whatever is suitable)