ElasticSearch: Curator concurrent snapshots - elasticsearch

Question

We're getting this message:
[2017-08-11T04:00:02,908][WARN ][r.suppressed ] path: /_snapshot/s3_currently/curator-20170811040002, params: {repository=s3_currently, wait_for_completion=true, snapshot=curator-20170811040002}
org.elasticsearch.snapshots.ConcurrentSnapshotExecutionException: [s3_currently:curator-20170811040002]a snapshot is already running
We've configured x-pack curator with two actions:
/home/curator/actions/currently.yml
---
actions:
1:
action: snapshot
description: Create snapshot every 30 minutes.
options:
repository: s3_currently
wait_for_completion: true
filters:
- filtertype: alias
aliases: living
2:
action: delete_snapshots
description: Remove recently snapshots
options:
repository: s3_currently
retry_interval: 120
retry_count: 3
filters:
- filtertype: count
count: 48
And /home/curator/actions/currently-dev.yml:
---
actions:
1:
action: snapshot
description: Create snapshot every hour for development.
options:
repository: s3_currently_dev
wait_for_completion: true
filters:
- filtertype: alias
aliases: living
2:
action: delete_snapshots
description: Remove recently snapshots
options:
repository: s3_currently_dev
retry_interval: 120
retry_count: 3
filters:
- filtertype: count
count: 24
We've added two cron jobs:
0 * * * * -> currently_dev
0,30 * * * * -> currently
Any ideas? It seems that elasticsearch doesn't allow to execute two concurrent snapshots, does it?

Answer 1

Elasticsearch does not allow for more than one snapshot to run at a time. The reason for this is that it is compelled to freeze the Lucene segments for the selected indices for the duration of the snapshot. It would be extremely taxing to the cluster to do this for multiple concurrent snapshots, not in terms of processing, but in terms of how it has to track all segments at all times. It must allow for new data to be indexed into new segments while others are locked/frozen for snapshotting. This could create a situation where there are too many open segments, which could deprive one or more nodes of needed memory resources. As a result, it's safer for Elasticsearch to only permit a single snapshot at a time.