The Snapshot and Restore feature is a really nice set of tools for backing up and restoring your data. Because Bonsai is a managed service that offers a multitenant architecture, there are some limits on how it can be used.
Bonsai takes regular, automatic backups of all paid clusters, and stores them in an encrypted S3 bucket in the same region as the cluster. These snapshots are taken at the start of every hour and are retained for two weeks.
In some rare cases, you may see an error like this when attempting to alter an index:
<div class="code-snippet w-richtext">
<pre><code fs-codehighlight-element="code" class="hljs language-javascript">Cannot delete indices that are being snapshotted: [[my_index/dPzLciT9RlmS8OtGGm61IQ]]. Try again after snapshot finishes or cancel the currently running snapshot.
</code></pre>
</div>
This is happening because the action is taking place during the regular snapshot. Snapshots happen at the start of every hour (00:00, 01:00, 02:00, etc), and can take anywhere from a few seconds to a few minutes. If you attempt to delete your index during this time, the action will be blocked.
The solution is to wait a minute or two and try again.
In the unlikely event that your cluster suffers unrecoverable data loss (for example: a node hosting primary data is lost, and there’s no replica), then we will use the most recent successful snapshot to restore the data. Any data updates from the time of the snapshot to the time of its restoration will need to be reindexed.
Not at this time. The technical explanation is that snapshot and restore operations can be extremely demanding on IO, and Elasticsearch will only allow one snapshot/restore operation to occur at a time, with subsequent calls sitting in a queue.
We’re working on ways of making snapshot and restore features safer and easier to use on the Bonsai platform. If you have an idea or use case you’d like to see supported, shoot us an email and we’ll evaluate adding it to our development pipeline.
If you’ve read the article on Architecture Classes, specifically the section on Multi Tenant Class clusters, you’ll understand why this is problematic. Users with unrestrained access to that API could inadvertently take down a group of nodes with some ill-timed calls. Or, an impatient user wondering why his/her snapshot isn’t processing right away may repeat the call multiple times, populating the queue. It’s also plausible that a less experienced user may attempt taking a snapshot a minute.
Something to consider is the purpose of the snapshot. If your desire is simply to have an up to date backup, then Bonsai is already handling that with hourly snapshots. If the desire is to restore the snapshot locally, or to another cluster for testing/dev purposes, we may be able to accommodate through our support channels.
We can provide custom SLA’s – including more frequent snapshots and longer retention times for users on Enterprise plans. Send an email to support with your requirements and we’ll put together a quote.
The encrypted buckets we use are set up and managed automatically. It’s possible to have snapshots added to a different bucket, but only for Enterprise subscriptions. If you’re on an Enterprise tier cluster, please send us a request.
