The term “metering” in Bonsai refers to the limits imposed on the resources allocated to a cluster. A cluster’s limits are determined by its subscription level; higher-priced plans yield higher limits. There are several metered resources that can result in an overage. These documents explains what those resources are and how to resolve related overages:
How Bonsai Handles Overages
Overages are indicated on the Cluster dashboard. If your cluster is over its subscription limits, the overage will be indicated in red like so:
Bonsai uses “soft limits” for metering. This approach does not immediately penalize or disable clusters exceeding the limits of the subscription. This is a much more gentle way of treating users who are probably not even aware that they’re over their limits, or why that may be an issue.
When an overages is detected, it triggers a state machine that follows a specially-designed process that takes increasingly firm actions. This process has several steps:
1. Initial notification (immediately). The owner and members of their team is notified that there is an overage, and provided information about how to address it.
2. Second notification (5 days). A reminder is sent and warns that the cluster is about to be put into read-only mode. Clusters in read-only mode will receive an HTTP 403: Cluster Read Only error message.
3. Read-only mode (10 days). The cluster is put into read-only mode. Updates will fail with a 403 error.
4. Disabled (15 days). All access to the cluster is disabled. Both searches and updates will fail with a HTTP 403: Cluster Disabled error.
Extreme Overages Skip the Process
Overages that are particularly aggressive are subject to being disabled immediately. Bonsai uses a fairly generous algorithm to determine whether an overage is severe and subject to immediate cut-off. This step is uncommon, but is a definite possibility.
Stale Data May Be Lost
Free clusters that have been disabled for a period of time may be purged to free up resources for other users.
The fastest way to deal with an overage is to simply upgrade your cluster’s plan. Upgrades take effect instantly and will unlock any cluster set to read-only or disabled.
If upgrading is not possible for some reason, then the next best option is to address the issue directly. This document contains information about how to address an overage in each metered resource.
Give it a minute!
The resource usage indicators are not real time displays. Cluster stats are calculated every 10 minutes or so. If you address an overage and the change isn’t immediately reflected on the display, don’t worry. The changes will be detected, the dashboard will be updated, and any sanction in place on the cluster will be lifted automatically. Please wait 5-10 minutes before emailing support.
Concurrent Connections
Concurrent connections are effective number of connections a cluster can handle before it returns an HTTP 429 response. Bonsai meters on concurrency as one level of defense against Denial of Service (DoS) and noisy neighbor situations, which helps ensure users are not taking more than their fair share of resources.
More on concurrency and how to solve concurrency overages in our Reducing Concurrent Connections documentation.
Shards
A shard is the basic unit of work in Elasticsearch. If you haven’t read the Core Concept on Shards, that would be a good place to start. Bonsai meters on the total number of shards in a cluster. That means both primary and replica shards count towards the limit.
The relationship between shard scheme and your cluster’s usage can sometimes not be readily apparent. For example, if you have an index with a 3x2 sharding scheme (3 primaries, 2 replicas), that’s not 5 or 6 shards, it’s 9. If this is confusing, read our Shards and Replicas documentation for some nice illustrations.
If you have a shard overage you can look at out Reducing Shard Usage documentation to resolve it.
Disk
Bonsai meters on the total amount of disk space a cluster can consume. This is for capacity planning purposes, and to ensure multitenant customers have their fair share of resources. Bonsai calculates a cluster’s disk usage by looking at the total data store size in bytes. This information can be found in the Index Stats API.
Disk overages can be resolved in a couple different ways that are explained in our Reducing Data Usage documentation.
Documents
Bonsai meters on the number of documents in an index. There are several types of “documents” in Elasticsearch, but Bonsai only counts the live Lucene documents in primary shards towards the document limit. Documents which have been marked as deleted, but have not yet been merged out of the segment files do not count towards the limit.
Document overages can be resolved in a couple different ways that are explained in our Reducing Document Usage documentation.