{"_id":"5aeb6512ebd0390003928785","project":"5633ebff7e9e880d00af1a53","version":{"_id":"5a8fae0268264c001f20cc00","project":"5633ebff7e9e880d00af1a53","__v":4,"createdAt":"2018-02-23T06:00:34.961Z","releaseDate":"2018-02-23T06:00:34.961Z","categories":["5a8fae0268264c001f20cc01","5a8fae0268264c001f20cc02","5a8fae0368264c001f20cc03","5a8fae0368264c001f20cc04","5a8fae0368264c001f20cc05","5a8fae0368264c001f20cc06","5a8fae0368264c001f20cc07","5a8fae0368264c001f20cc08","5a8fae0368264c001f20cc09","5abaa7eb72d6dc0028a07bf3","5b8ee7842790f8000333f9ba","5b8ee8f244a21a00034b5cd9"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"","version_clean":"2.0.0","version":"2.0"},"category":{"_id":"5a8fae0368264c001f20cc04","version":"5a8fae0268264c001f20cc00","project":"5633ebff7e9e880d00af1a53","__v":0,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-05-31T04:26:39.925Z","from_sync":false,"order":5,"slug":"versions","title":"Bonsai.io Platform"},"user":"5637d336aa96490d00a64f81","githubsync":"","__v":0,"parentDoc":null,"updates":[],"next":{"pages":[],"description":""},"createdAt":"2018-05-03T19:37:54.074Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":4,"body":"The term \"metering\" in Bonsai refers to the limits imposed on the resources allocated to a cluster. A cluster's limits are determined by its subscription level; higher-priced plans yield higher limits. There are several metered resources that can result in an overage. This document explains what those resources are and how to resolve related overages:\n\n* [Shards](doc:metering-on-bonsai#shards) \n* [Documents](doc:metering-on-bonsai#documents) \n* [Disk](doc:metering-on-bonsai#disk) \n* [Memory](doc:metering-on-bonsai#memory) \n\nAdditionally, Bonsai meters on [Concurrent Connections](doc:metering-on-bonsai#concurrent-connections). Exceeding the connection limit will not result in an overage per se, but rather [HTTP 429's](doc:error-codes#http-429-too-many-requests).\n[block:api-header]\n{\n  \"title\": \"Checking on Cluster Status\"\n}\n[/block]\nAll Bonsai clusters have a dashboard that indicates the cluster's resource usage. The relevant section looks like this:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/982865c-usage.png\",\n        \"usage.png\",\n        1200,\n        234,\n        \"#f3f3f3\"\n      ],\n      \"caption\": \"This cluster has no indices or data.\"\n    }\n  ]\n}\n[/block]\nYou can find more information about checking on your cluster's status in the documentation:\n\n* [Direct Signups](doc:managing-your-cluster#overview) \n* [Heroku Users](doc:bonsai-elasticsearch-dashboard#overview) \n* [Manifold Users](doc:managing-your-cluster-1#overview) \n[block:api-header]\n{\n  \"title\": \"Overages\"\n}\n[/block]\nOverages are indicated on the Cluster dashboard (see [previous section](doc:metering-on-bonsai#checking-on-cluster-status). If your cluster is over its subscription limits, the overage will be indicated in red like so:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/f74af66-overage.png\",\n        \"overage.png\",\n        2514,\n        328,\n        \"#7e2c38\"\n      ],\n      \"caption\": \"This cluster has some data, and 14 shards out of 6 allowed. This is an overage.\"\n    }\n  ]\n}\n[/block]\nBonsai uses \"soft limits\" for metering. This approach does not immediately penalize or disable clusters exceeding the limits of the subscription. This is a much more gentle way of treating users who are probably not even aware that they're over their limits, or why that may be an issue.\n\nWhen an overages is detected, it triggers a state machine that follows a specially-designed process that takes increasingly firm actions. This process has several steps:\n\n**1. Initial notification (immediately).** The owner and members of their team is notified that there is an overage, and provided information about how to address it.\n\n**2. Second notification (5 days).** A reminder is sent and warns that the cluster is about to be put into read-only mode. Clusters in read-only mode will receive an [HTTP 403: Cluster Read Only](doc:error-codes#http-403-cluster-read-only) error message.\n\n**3. Read-only mode (10 days).** The cluster is put into read-only mode. Updates will fail with a [403](doc:error-codes#http-403-cluster-read-only) error.\n\n**4. Disabled (15 days).** All access to the cluster is disabled. Both searches and updates will fail with a [HTTP 403: Cluster Disabled](doc:error-codes#http-403-cluster-disabled) error.\n[block:callout]\n{\n  \"type\": \"warning\",\n  \"body\": \"Overages that are particularly aggressive are subject to being disabled immediately. Bonsai uses a fairly generous algorithm to determine whether an overage is severe and subject to immediate cut-off. This step is uncommon, but is a definite possibility.\",\n  \"title\": \"Extreme Overages Skip the Process\"\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"danger\",\n  \"title\": \"Stale Data May Be Lost\",\n  \"body\": \"Free clusters that have been disabled for a period of time may be purged to free up resources for other users.\"\n}\n[/block]\nThe fastest way to deal with an overage is to simply upgrade your cluster's plan. Upgrades take effect instantly and will unlock any cluster set to read-only or disabled. For more information on upgrading your plan, see the documentation for your account type:\n\n* [Direct Users](doc:managing-your-cluster#manage)\n* [Heroku Users](doc:changing-your-plan) \n* [Manifold Users](doc:changing-your-manifold-plan)\n\nIf upgrading is not possible for some reason, then the next best option is to address the issue directly. This document contains information about how to address an overage in each metered resource.\n[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"Give it a minute!\",\n  \"body\": \"The resource usage indicators are not real time displays. Cluster stats are calculated every 10 minutes or so. If you address an overage and the change isn't immediately reflected on the display, don't worry. The changes will be detected, the dashboard will be updated, and any sanction in place on the cluster will be lifted automatically. Please wait 5-10 minutes before [emailing support](mailto:support:::at:::bonsai.io).\"\n}\n[/block]\nHere are some quick links to questions you may have: \n\n* [How do resolve shard overages](doc:metering-on-bonsai#shards)\n* [How to reduce document counts](doc:metering-on-bonsai#documents)\n* [How to reduce disk usage](doc:metering-on-bonsai#disk)\n* [Dealing with concurrency issues](doc:metering-on-bonsai#concurrent-connections)\n* [Reducing memory footprint](doc:metering-on-bonsai#memory)\n[block:api-header]\n{\n  \"title\": \"Concurrent Connections\"\n}\n[/block]\nConcurrent connections are simultaneous _active_ connections to a cluster. Bonsai meters on concurrency as one level of defense against [Denial of Service (DoS)](https://en.wikipedia.org/wiki/Denial-of-service_attack) and [noisy neighbor](https://en.wikipedia.org/wiki/Cloud_computing_issues#Performance_interference_and_noisy_neighbors) situations, and to help ensure users are not taking more than their fair share of resources. \n\nBonsai distinguishes between types of connections for concurrency metering purposes:\n\n* **Searches.** Search requests. The concurrency allowance for search traffic is generally much higher than for updates and bulk.\n* **Updates.** Adding, modifying or deleting a given document. Generally has the lowest concurrency allowance due to the high overhead of single document updates.\n* **Bulk.** Adding, modifying or deleting a batch of documents using the [Bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html). Bulk requests are the preferred way to change data for efficiency reasons. \n\nBonsai also uses connection pools to account for short-lived bursts of traffic. If all available connections are in use, subsequent connections are put into a FIFO queue and served when a connection becomes available. If all connections are in use *and* the request pool is exhausted, then Bonsai will return an immediate [HTTP 429](doc:error-codes#http-429-too-many-requests).\n[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"Concurrency is NOT Throughput!\",\n  \"body\": \"Concurrency allowances are a limit on total active connections, not throughput (requests per second). A cluster with a search concurrency allowance of 10 connections can easily serve hundreds of requests per second. A typical p50 query time on Bonsai is ~10ms. At this latency, a cluster could serve a sustained 100 requests per second, _per connection_. That would give the hypothetical cluster a maximum throughput of around 1,000 requests per second before the queue would even be needed. In practice, throughput may be closer to 500 rps (to account for bursts, network effects and longer-running requests).\\n\\nTo put these numbers in perspective, a sustained rate of 500 rps is over 43M requests per day. StackExchange -- the parent of sites like [StackOverflow](https://stackoverflow.com/), [ServerFault](https://serverfault.com/) and [SuperUser](https://superuser.com/) (and [many more](https://stackexchange.com/sites#)) is performing [34M daily Elasticsearch queries](https://stackexchange.com/performance). By the time your application demands exceed StackExchange by 25%, you will likely already be on a [single tenant configuration](doc:architecture-classes#single-tenant-class) with no concurrency limits.\"\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"warning\",\n  \"title\": \"Queued Connections Have a 60s TTL\",\n  \"body\": \"If the connection queue begins to fill up with connection requests, those requests will only be held for up to 60 seconds. After that time, Bonsai will return a [HTTP 504](doc:error-codes#http-504-gateway-timeout) response. If you are seeing HTTP 429 and 504 errors in your logs, that is an indicator that your cluster has a high volume of long-running queries.\"\n}\n[/block]\n## Fixing Concurrency Issues\n\nAn HTTP 429 error indicates that traffic to the cluster is not being cleared fast enough given the concurrency allowance. This leaves three possibilities for a resolution:\n\n1. Upgrade to a plan with a higher concurrency allowance\n2. Reduce the request overhead\n3. Reduce the traffic rate\n\nIf reading over these suggestions doesn't provide enough ideas for resolving the issue, you can always email our support team at [support@bonsai.io](mailto:support@bonsai.io) to discuss options.\n\n### 1. Upgrading\n\nConcurrency allowances on Bonsai scale with the plan level. Often the fastest way to resolve concurrency issues is to simply upgrade the plan. Upgrades take effect instantly. You can find more information about updating your plan in the documentation below:\n\n* [Bonsai.io users](doc:managing-your-cluster#manage)\n* [Heroku users](doc:changing-your-plan)\n* [Manifold users](doc:changing-your-manifold-plan)\n\n\n### 2. Reduce Overhead\n\nThe more time a request takes to process, the longer the connection remains open and active. Too many of these requests in a given amount of time will exhaust the connection pool and request queue, resulting in HTTP 429 responses. Reducing the amount of time a request takes to process adds overhead for more traffic.\n\nSome examples of requests which tend to have a lot of overhead and take longer to process:\n\n* Aggregations on large numbers of results\n* Geospatial sorting on large numbers of results\n* Highlighting\n* Custom scripting\n* Wildcard matching\n\nIf you have requests that perform a lot of processing, then finding ways to optimize them with filter caches and better scoping can improve throughput quite a bit.\n\n\n### 3. Reduce Traffic Volume\n\nSometimes applications are written without considering scalability or impact on Elasticsearch. These often result in far more queries being made to the cluster than is really necessary. Some minor changes are all that is needed to reduce the volume and avoid HTTP 429 responses in a way that still makes the application usable. \n\nA common example is autocomplete / typeahead scripts. Often the frontend is sending a query to the Elasticsearch cluster each time a user presses a key. The average computer user types around 3-4 characters per second, and (depending on the query and typist speed) a search launched by one keypress may not even be returned before the next keypress is made. This results in a piling up of requests. More users searching the app exacerbate the problem. Initiating a search every ~500 milliseconds instead of every keypress will be much more scalable without impacting the user experience.\n\nAnother example might be a site that boosts relevancy scores by page view. Whenever a page is requested, the application updates the corresponding document in Elasticsearch by incrementing a counter for the number of times the page has been viewed. \n\nThis strategy will boost a document's position in the results in real time, but it also means the cluster is updated every time a user visits any page, potentially resulting in a high volume of expensive single document updates. It would be better to write the page counts to the local database and use a queue and worker system (like [Resque](https://github.com/resque/resque)) to push out updated page counts using the Bulk API every minute or so. This would be a much cheaper and more scalable approach, and would be just as effective.\n[block:api-header]\n{\n  \"title\": \"Shards\"\n}\n[/block]\nA shard is the basic unit of work in Elasticsearch. If you haven't read the [Core Concept on Shards](doc:basic-glossary#shards), that would be a good place to start. Bonsai meters on the total number of shards in a cluster. That means both primary and replica shards count towards the limit. \n\nThe relationship between shard scheme and your cluster's usage can sometimes not be readily apparent. For example, if you have an index with a 3x2 sharding scheme (3 primaries, 2 replicas), that's not 5 or 6 shards, it's 9. If this is confusing, read our [Shards and Replicas](doc:what-are-shards-and-replicas) documentation for some nice illustrations.\n\nIf you have a shard overage, that can mean one of three things:\n \n1. Extraneous indices.\n2. Sharding scheme is not optimal.\n3. Replication too high\n\nIt is also possible that the cluster and its data are already configured according to best practices. In that case, you may need to get creative with aliases and data collocation in order to remain on your current subscription. The [Reducing Shard Usage](doc:reducing-shard-usage) document has information about all of these possibilities.\n\nIf you find that you're unable to reduce shards through the options discussed on the [Reducing Shard Usage](doc:reducing-shard-usage) page, then you basically will need to upgrade to the next plan. You can find more information about updating your plan in the documentation below:\n\n* [Bonsai.io users](doc:managing-your-cluster#manage)\n* [Heroku users](doc:changing-your-plan)\n* [Manifold users](doc:changing-your-manifold-plan)\n[block:api-header]\n{\n  \"title\": \"Documents\"\n}\n[/block]\nBonsai meters on the number of documents in an index. There are several types of \"documents\" in Elasticsearch, but Bonsai only counts the live Lucene documents in primary shards towards the document limit. Documents which have been marked as deleted, but have not yet been merged out of the segment files do not count towards the limit.\n\nThe phrase \"live Lucene documents\" may be a little confusing, but this is due to how Elasticsearch counts [nested documents](https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html). \n\nThe [Index Stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html) is used to determine how many documents are in a cluster. The Index Stats API counts  [nested documents](https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html) by including all associated documents. In other words, if you have a document with 2 nested documents, this is reported as 3 total documents.\n\nElasticsearch has several different articles on how nested documents work, but the simplest answer is that it is creating the illusion of complex object by quietly creating multiple hidden documents.\n[block:callout]\n{\n  \"type\": \"info\",\n  \"body\": \"A common point of confusion is that the `/_cat/indices` endpoint will show one set of documents, while the `/_stats` endpoint shows a much larger count. This is because the Cat API is counting the \\\"visible\\\" documents, while the Index Stats API is counting _all_ documents. The `_stats` endpoint is a more true representation of a cluster's document usage, and is the most fair to all users for metering purposes.\",\n  \"title\": \"How Do I Have a Document Overage? I'm Way Under the Limit!\"\n}\n[/block]\nThere are several strategies for resolving a document overage:\n\n### Remove Old Data\n\nIf your index has a lot of old data, or \"stale\" data (documents which rarely show up in searches), then you could simply delete those documents. Deleted documents do not count against your limits.\n\n### Remove an Index\n\nOccasionally users are indexing time series data, or database tables that are not actually being searched by the application. Audit your usage by using the Interactive Console to check the `/_cat/indices` endpoint. If you find that there are old or unecessary indices with data, then delete those.\n\n### Compact Your Mappings\nChanging your mappings to nest less information can greatly reduce your document usage. Consider this sample document:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n  \\\"title\\\": \\\"Spiderman saves child from well\\\",\\n  \\\"body\\\":  \\\"Move over, Lassie! New York has a new hero. But is he also a menace?\\\",\\n  \\\"authors\\\": [ \\n    {\\n      \\\"name\\\":  \\\"Jonah Jameson\\\",\\n      \\\"title\\\": \\\"Sr. Editor\\\",\\n    },\\n    {\\n      \\\"name\\\":  \\\"Peter Parker\\\",\\n      \\\"title\\\": \\\"Photos\\\",\\n    }\\n  ],\\n  \\\"comments\\\": [ \\n    {\\n      \\\"username\\\": \\\"captain_usa\\\",\\n      \\\"comment\\\":  \\\"I understood that reference!\\\",\\n    },\\n    {\\n      \\\"username\\\": \\\"man_of_iron\\\",\\n      \\\"comment\\\":  \\\"Congrats on being slightly more useful than a ladder.\\\",\\n    }\\n  ],\\n  \\\"photos\\\": [ \\n    {\\n      \\\"url\\\":      \\\"https://assets.dailybugle.com/12345\\\",\\n      \\\"caption\\\":  \\\"Spiderman delivering Timmy back to his mother\\\",\\n    }\\n  ]\\n}\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\nNote that it's nesting data for authors, comments and photos. The mapping above would actually result in creating 6 documents. Removing the comments and photos (which usually don't need to be indexed anyway) would reduce the footprint by 50%.\n\nIf you're using nested objects, review whether any of the nested information could stand to be left out, and then reindex with a smaller mapping.\n\n### Upgrade the Subscription\n\nIf you find that you're unable to remove any documents or indices, or change your mappings, then you will simply need to upgrade to the next subscription level. You can find more information about updating your plan in the documentation below:\n\n* [Bonsai.io users](doc:managing-your-cluster#manage)\n* [Heroku users](doc:changing-your-plan)\n* [Manifold users](doc:changing-your-manifold-plan)\n[block:api-header]\n{\n  \"title\": \"Disk\"\n}\n[/block]\nBonsai meters on the total amount of disk space a cluster can consume. This is for capacity planning purposes, and to ensure [multitenant](doc:architecture-classes#multi-tenant-class) customers have their fair share of resources. Bonsai calculates a cluster's disk usage by looking at the total data store size in bytes. This information can be found in the [Index Stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html).\n\nResolving disk overages can be resolved in a couple different ways:\n\n### Remove Stale Data / Indices\n\nThere are some cases where one or more indices are created on a cluster for testing purposes, and are not actually being used for anything. These will count towards the data limits; if you're getting overage notifications, then you should delete these indices.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"GET /_cat/indices\\ngreen open prod20180101    1 1 1015123 0  32M  64M \\ngreen open prod20180201    1 1 1016456 0  35M  70M \\ngreen open prod20180301    1 1 1017123 0  39M  78M\\ngreen open prod20180401    1 1 1018456 0  45M  90M \\ngreen open prod20180501    1 1 1019123 0  47M  94M \\ngreen open prod20180601    1 1 1020456 0  51M  102M \",\n      \"language\": \"text\"\n    }\n  ]\n}\n[/block]\nRemoving the old and unneeded indices in the example above would free up 356MB. A single command could do it:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"# Delete a group of indices:\\nDELETE /prod20180101,prod20180201,prod20180301,prod20180401,prod20180501\",\n      \"language\": \"text\"\n    }\n  ]\n}\n[/block]\n### Purge Deleted Documents\n\nData in Elasticsearch is spread across lots of files called segments. Segments each contain some number of documents. An index could have dozens, hundreds or even thousands of segment files, and Elasticsearch will periodically merge some segment files into others.\n\nWhen a document is deleted in Elasticsearch, its segment file is simply updated to mark the document as deleted. The data is not actually removed until that segment file is merged with another. Elasticsearch normally handles segment merging automatically, but forcing a segment merging will reduce the overall disk footprint of the cluster by eliminating deleted documents.\n\nNormally this is done through the Optimize / Forcemerge API, but this is unfortunately one of Bonsai's [Unsupported API Endpoints](doc:bonsai-unsupported-actions). The same effect can be accomplished however, by simply reindexing. Reindexing will cause the data to be refreshed, and no deleted documents will be tracked by Elasticsearch. This will reduce disk usage.\n\nTo check whether this will work for you, look at the `/_cat/indices` data. There is a column called `docs.deleted`, which shows how many documents are sitting on the disk and are marked as deleted. This should give a sense of how much data could be freed up by reindexing. For example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"health status index     pri rep docs.count docs.deleted store.size pri.store.size\\ngreen  open   my_index  3   2   15678948   6895795      47.1G      15.7G\",\n      \"language\": \"text\"\n    }\n  ]\n}\n[/block]\nIn this case, the `docs.deleted` is around 30% of the primary store, or around 4.8G of primary data. With replication, this works out to something like 14.4GB of total disk marked for deletion. Reindexing would reduce the cluster's disk footprint by this much. The result would look like this:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"health status index     pri rep docs.count docs.deleted store.size pri.store.size\\ngreen  open   my_index  3   2   15678948   0            32.7G      10.9G\",\n      \"language\": \"text\"\n    }\n  ]\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"Protip: Queue Writes and Reindex in the Background for Minimal Impact\",\n  \"body\": \"Your app's search could be down or degraded during a reindex. If reindexing will take a long time, that may make this option unfeasible. However, you could minimize the impact by using something like Kafka to queue writes while reindexing to a new index.\\n\\nSearch traffic can continue to be served from the old index until its replacement is ready. Flush the queued updates from Kafka into the new index, then destroy the old index and use an alias to promote the new index.\\n\\nThe tradeoff of this solution is that you'll minimize the impact to your traffic/users, but you'll need to set up and manage the queue software. You'll also have a lot of duplicated data for a short period of time, so your footprint could be way above the subscription limit for a short time. \\n\\nTo prevent the state machine from disabling your cluster, you might want to consider temporarily upgrading to perform the operation, then downgrading when you're done. Billing is prorated, so this would not add much to your invoice. You can always [email us](mailto:support@bonsai.io) to discuss options before settling on a decision.\"\n}\n[/block]\n### Reindex with Smaller Mappings\n\n[Mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) define how data is stored and indexed in an Elasticsearch cluster. There are some settings which can cause the disk footprint to grow exponentially. \n\nFor example, [synonym expansion](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-synonym-tokenfilter.html) can lead to lots of extra tokens to be generated per input token (if you're using WordNet, see our documentation article on it, specifically [Why Wouldn't Everyone Want WordNet?](doc:using-wordnet-with-bonsai#why-wouldnt-everyone-want-wordnet)). If you're using lots of index-time synonym expansion, then you're essentially inflating the document sizes with lots of data, with the tradeoff (hopefully) being improved relevancy.\n\nAnother example would be [Ngrams](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-ngram-tokenizer.html). Ngrams are tokens generated from the parts of other tokens. A token like \"hello\" could be broken into 2-grams like \"he\", \"el\", \"ll\", and \"lo\". In 3-grams, it would be \"hel\", \"ell\" and \"llo\". And so on. The [Elasticsearch Guide](https://www.elastic.co/guide/en/elasticsearch/guide/current/_ngrams_for_partial_matching.html) has more examples. \n\nIt's possible to generate multiple gram sizes for a single, starting with values as low as 1. Some developers use this to maximize substring matching. But there is an exponential growth in the number of grams generated for a single token:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/043ff10-gram-vs-token.png\",\n        \"gram-vs-token.png\",\n        605,\n        340,\n        \"#163553\"\n      ]\n    }\n  ]\n}\n[/block]\nThis relationship is expressed mathematically as:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/3a7a58a-ngram-pattern.png\",\n        \"ngram-pattern.png\",\n        1500,\n        554,\n        \"#0795fc\"\n      ]\n    }\n  ]\n}\n[/block]\nIn other words, a token with a length of 5 and a minimum gram size of 1 would result in `(1/2)*5*(5+1)=15` grams. A token with a length of 10 would result in 55 grams. The grams are generated _per token_, which leads to an explostion in terms for a document.\n\nAs a sample calculation: if a typical document in your corpus has a field with ~1,000 tokens and a [Rayleigh distribution](https://en.wikipedia.org/wiki/Rayleigh_distribution) of length with an average of ~5, you could plausibly see something like a 1,100-1,200% inflation in disk footprint using Ngrams of minimum size 1. In other words, if the non-grammed document would need 100KB on disk, the Ngrammed version would need over 1MB. Virtually none of this overhead would improve relevancy, and would probably even hurt it.\n\n[Nested documents](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html) are another example of a feature can also increase your data footprint without necessarily improving relevancy.\n\nThe point is that there are plenty of features available that lead to higher disk usage than one might think at first glance. Check on your mappings carefully: look for large synonym expansions, make sure you're using Ngrams with a minimum gram size of 3 or more (also look into [EdgeNGrams](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenizer.html) if you're attempting autocomplete), and see if you can get away with fewer nested objects. Reindex your data with the updated mappings, and you should see a definite improvement.\n\n### Upgrade the Subscription\n\nIf you find that you're unable to remove data, reindex, or update your mappings -- or that these changes don't yeild a stable resolution -- then you will simply need to upgrade to the next subscription level. You can find more information about updating your plan in the documentation below:\n\n* [Bonsai.io users](doc:managing-your-cluster#manage)\n* [Heroku users](doc:changing-your-plan)\n* [Manifold users](doc:changing-your-manifold-plan)\n[block:api-header]\n{\n  \"title\": \"Memory\"\n}\n[/block]\nBonsai meters on the amount of memory that a cluster can use. There are several categories involving memory usage. For metering purposes, Bonsai uses the sum of the following catagories to determine memory footprint:\n\n* segments\n* fielddata\n* request_cache / query_cache\n\nThe exact amount of memory used by each category can be found in the [Index Stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html). Reducing memory footprint can be tricky, although fortunately overages of this class are rare. There are a couple of possible resolutions:\n\n### Reduce Segment Files\n\nData in Elasticsearch is spread across lots of files called segments. Segments each contain some number of documents. An index could have dozens, hundreds or even thousands of segment files, and Elasticsearch will periodically merge some segment files into others.\n\nAn interesting subject is how documents are deleted in Elasticsearch; when a document is deleted, the segment file that contains it is simply updated to mark the document as deleted. The data is not actually removed until that segment file is merged with another. So segment merging not only reduces the number of files Elasticsearch is tracking, but it will also reduce the overall footprint of the cluster, both in memory and on disk.\n\nMerging can be forced through the Optimize / Forcemerge API, but this is unfortunately one of Bonsai's [Unsupported API Endpoints](doc:bonsai-unsupported-actions). The same effect can be accomplished however, by simply reindexing. Reindexing will cause the data to be refreshed, and no deleted documents will be tracked by Elasticsearch. This will reduce memory usage.\n\n### Disable fielddata\n\nThe Elasticsearch documentation has an [awesome article](https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html) explaining fielddata:\n\n> Most fields are indexed by default, which makes them searchable. Sorting, aggregations, and accessing field values in scripts, however, requires a different access pattern from search.\n> \n> Search needs to answer the question \"Which documents contain this term?\", while sorting and aggregations need to answer a different question: \"What is the value of this field for this document?\".\n> \n> Most fields can use index-time, on-disk doc_values for this data access pattern, but text fields do not support doc_values.\n> \n> Instead, text fields use a query-time in-memory data structure called fielddata. This data structure is built on demand the first time that a field is used for aggregations, sorting, or in a script. It is built by reading the entire inverted index for each segment from disk, inverting the term ↔︎ document relationship, and storing the result in memory, in the JVM heap.\n> ...\n> Fielddata can consume a lot of heap space, especially when loading high cardinality text fields. Once fielddata has been loaded into the heap, it remains there for the lifetime of the segment. Also, loading fielddata is an expensive process which can cause users to experience latency hits. This is why fielddata is disabled by default.\n\nIf you're using fielddata for your text fields, then this can consume an inordinate amount of memory. Disabling it and reindexing should reduce your memory footprint considerably.\n\n### Clear the Caches\n\nWhen Elasticsearch performs a search request, it will automatically cache a shard's local results to memory. It does not cache the complete document, but rather a bit-string representation. A bit-string is a data structure that has a 1 or a 0 representing whether a document ID matches \n\nThis can still lead to large cache filters if the cluster contains lots of documents. A corpus of a million documents will lead to filter sizes of roughly 125KB. Each unique query will result in a new bit string; if there are also lots of unique searches happening in a short period of time, these can add up quickly.\n\nThe cache Elasticsearch builds is invalidated whenever the shard refreshes (but only when there are changes to the shard). This happens every 1s by default, but the value can be increased. So if your cluster for some reason has a high refresh interval, then the caches can lead to a higher memory footprint.\n\nReducing the [refresh interval ](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html#bulk) will lead to more frequent cache invalidation, which would reduce the amount of memory used by the caches. Alternatively, you can use the [/_cache/clear API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-clearcache.html) to expire the caches manually.\n\n### Upgrade the Subscription\n\nIf you find that you're unable to reindex, disable fielddata or clear your caches -- or that these changes don't yeild a stable resolution -- then you will simply need to upgrade to the next subscription level. You can find more information about updating your plan in the documentation below:\n\n* [Bonsai.io users](doc:managing-your-cluster#manage)\n* [Heroku users](doc:changing-your-plan)\n* [Manifold users](doc:changing-your-manifold-plan)","excerpt":"","slug":"metering-on-bonsai","type":"basic","title":"Metering on Bonsai"}

Metering on Bonsai


The term "metering" in Bonsai refers to the limits imposed on the resources allocated to a cluster. A cluster's limits are determined by its subscription level; higher-priced plans yield higher limits. There are several metered resources that can result in an overage. This document explains what those resources are and how to resolve related overages: * [Shards](doc:metering-on-bonsai#shards) * [Documents](doc:metering-on-bonsai#documents) * [Disk](doc:metering-on-bonsai#disk) * [Memory](doc:metering-on-bonsai#memory) Additionally, Bonsai meters on [Concurrent Connections](doc:metering-on-bonsai#concurrent-connections). Exceeding the connection limit will not result in an overage per se, but rather [HTTP 429's](doc:error-codes#http-429-too-many-requests). [block:api-header] { "title": "Checking on Cluster Status" } [/block] All Bonsai clusters have a dashboard that indicates the cluster's resource usage. The relevant section looks like this: [block:image] { "images": [ { "image": [ "https://files.readme.io/982865c-usage.png", "usage.png", 1200, 234, "#f3f3f3" ], "caption": "This cluster has no indices or data." } ] } [/block] You can find more information about checking on your cluster's status in the documentation: * [Direct Signups](doc:managing-your-cluster#overview) * [Heroku Users](doc:bonsai-elasticsearch-dashboard#overview) * [Manifold Users](doc:managing-your-cluster-1#overview) [block:api-header] { "title": "Overages" } [/block] Overages are indicated on the Cluster dashboard (see [previous section](doc:metering-on-bonsai#checking-on-cluster-status). If your cluster is over its subscription limits, the overage will be indicated in red like so: [block:image] { "images": [ { "image": [ "https://files.readme.io/f74af66-overage.png", "overage.png", 2514, 328, "#7e2c38" ], "caption": "This cluster has some data, and 14 shards out of 6 allowed. This is an overage." } ] } [/block] Bonsai uses "soft limits" for metering. This approach does not immediately penalize or disable clusters exceeding the limits of the subscription. This is a much more gentle way of treating users who are probably not even aware that they're over their limits, or why that may be an issue. When an overages is detected, it triggers a state machine that follows a specially-designed process that takes increasingly firm actions. This process has several steps: **1. Initial notification (immediately).** The owner and members of their team is notified that there is an overage, and provided information about how to address it. **2. Second notification (5 days).** A reminder is sent and warns that the cluster is about to be put into read-only mode. Clusters in read-only mode will receive an [HTTP 403: Cluster Read Only](doc:error-codes#http-403-cluster-read-only) error message. **3. Read-only mode (10 days).** The cluster is put into read-only mode. Updates will fail with a [403](doc:error-codes#http-403-cluster-read-only) error. **4. Disabled (15 days).** All access to the cluster is disabled. Both searches and updates will fail with a [HTTP 403: Cluster Disabled](doc:error-codes#http-403-cluster-disabled) error. [block:callout] { "type": "warning", "body": "Overages that are particularly aggressive are subject to being disabled immediately. Bonsai uses a fairly generous algorithm to determine whether an overage is severe and subject to immediate cut-off. This step is uncommon, but is a definite possibility.", "title": "Extreme Overages Skip the Process" } [/block] [block:callout] { "type": "danger", "title": "Stale Data May Be Lost", "body": "Free clusters that have been disabled for a period of time may be purged to free up resources for other users." } [/block] The fastest way to deal with an overage is to simply upgrade your cluster's plan. Upgrades take effect instantly and will unlock any cluster set to read-only or disabled. For more information on upgrading your plan, see the documentation for your account type: * [Direct Users](doc:managing-your-cluster#manage) * [Heroku Users](doc:changing-your-plan) * [Manifold Users](doc:changing-your-manifold-plan) If upgrading is not possible for some reason, then the next best option is to address the issue directly. This document contains information about how to address an overage in each metered resource. [block:callout] { "type": "info", "title": "Give it a minute!", "body": "The resource usage indicators are not real time displays. Cluster stats are calculated every 10 minutes or so. If you address an overage and the change isn't immediately reflected on the display, don't worry. The changes will be detected, the dashboard will be updated, and any sanction in place on the cluster will be lifted automatically. Please wait 5-10 minutes before [emailing support](mailto:support@bonsai.io)." } [/block] Here are some quick links to questions you may have: * [How do resolve shard overages](doc:metering-on-bonsai#shards) * [How to reduce document counts](doc:metering-on-bonsai#documents) * [How to reduce disk usage](doc:metering-on-bonsai#disk) * [Dealing with concurrency issues](doc:metering-on-bonsai#concurrent-connections) * [Reducing memory footprint](doc:metering-on-bonsai#memory) [block:api-header] { "title": "Concurrent Connections" } [/block] Concurrent connections are simultaneous _active_ connections to a cluster. Bonsai meters on concurrency as one level of defense against [Denial of Service (DoS)](https://en.wikipedia.org/wiki/Denial-of-service_attack) and [noisy neighbor](https://en.wikipedia.org/wiki/Cloud_computing_issues#Performance_interference_and_noisy_neighbors) situations, and to help ensure users are not taking more than their fair share of resources. Bonsai distinguishes between types of connections for concurrency metering purposes: * **Searches.** Search requests. The concurrency allowance for search traffic is generally much higher than for updates and bulk. * **Updates.** Adding, modifying or deleting a given document. Generally has the lowest concurrency allowance due to the high overhead of single document updates. * **Bulk.** Adding, modifying or deleting a batch of documents using the [Bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html). Bulk requests are the preferred way to change data for efficiency reasons. Bonsai also uses connection pools to account for short-lived bursts of traffic. If all available connections are in use, subsequent connections are put into a FIFO queue and served when a connection becomes available. If all connections are in use *and* the request pool is exhausted, then Bonsai will return an immediate [HTTP 429](doc:error-codes#http-429-too-many-requests). [block:callout] { "type": "info", "title": "Concurrency is NOT Throughput!", "body": "Concurrency allowances are a limit on total active connections, not throughput (requests per second). A cluster with a search concurrency allowance of 10 connections can easily serve hundreds of requests per second. A typical p50 query time on Bonsai is ~10ms. At this latency, a cluster could serve a sustained 100 requests per second, _per connection_. That would give the hypothetical cluster a maximum throughput of around 1,000 requests per second before the queue would even be needed. In practice, throughput may be closer to 500 rps (to account for bursts, network effects and longer-running requests).\n\nTo put these numbers in perspective, a sustained rate of 500 rps is over 43M requests per day. StackExchange -- the parent of sites like [StackOverflow](https://stackoverflow.com/), [ServerFault](https://serverfault.com/) and [SuperUser](https://superuser.com/) (and [many more](https://stackexchange.com/sites#)) is performing [34M daily Elasticsearch queries](https://stackexchange.com/performance). By the time your application demands exceed StackExchange by 25%, you will likely already be on a [single tenant configuration](doc:architecture-classes#single-tenant-class) with no concurrency limits." } [/block] [block:callout] { "type": "warning", "title": "Queued Connections Have a 60s TTL", "body": "If the connection queue begins to fill up with connection requests, those requests will only be held for up to 60 seconds. After that time, Bonsai will return a [HTTP 504](doc:error-codes#http-504-gateway-timeout) response. If you are seeing HTTP 429 and 504 errors in your logs, that is an indicator that your cluster has a high volume of long-running queries." } [/block] ## Fixing Concurrency Issues An HTTP 429 error indicates that traffic to the cluster is not being cleared fast enough given the concurrency allowance. This leaves three possibilities for a resolution: 1. Upgrade to a plan with a higher concurrency allowance 2. Reduce the request overhead 3. Reduce the traffic rate If reading over these suggestions doesn't provide enough ideas for resolving the issue, you can always email our support team at [support@bonsai.io](mailto:support@bonsai.io) to discuss options. ### 1. Upgrading Concurrency allowances on Bonsai scale with the plan level. Often the fastest way to resolve concurrency issues is to simply upgrade the plan. Upgrades take effect instantly. You can find more information about updating your plan in the documentation below: * [Bonsai.io users](doc:managing-your-cluster#manage) * [Heroku users](doc:changing-your-plan) * [Manifold users](doc:changing-your-manifold-plan) ### 2. Reduce Overhead The more time a request takes to process, the longer the connection remains open and active. Too many of these requests in a given amount of time will exhaust the connection pool and request queue, resulting in HTTP 429 responses. Reducing the amount of time a request takes to process adds overhead for more traffic. Some examples of requests which tend to have a lot of overhead and take longer to process: * Aggregations on large numbers of results * Geospatial sorting on large numbers of results * Highlighting * Custom scripting * Wildcard matching If you have requests that perform a lot of processing, then finding ways to optimize them with filter caches and better scoping can improve throughput quite a bit. ### 3. Reduce Traffic Volume Sometimes applications are written without considering scalability or impact on Elasticsearch. These often result in far more queries being made to the cluster than is really necessary. Some minor changes are all that is needed to reduce the volume and avoid HTTP 429 responses in a way that still makes the application usable. A common example is autocomplete / typeahead scripts. Often the frontend is sending a query to the Elasticsearch cluster each time a user presses a key. The average computer user types around 3-4 characters per second, and (depending on the query and typist speed) a search launched by one keypress may not even be returned before the next keypress is made. This results in a piling up of requests. More users searching the app exacerbate the problem. Initiating a search every ~500 milliseconds instead of every keypress will be much more scalable without impacting the user experience. Another example might be a site that boosts relevancy scores by page view. Whenever a page is requested, the application updates the corresponding document in Elasticsearch by incrementing a counter for the number of times the page has been viewed. This strategy will boost a document's position in the results in real time, but it also means the cluster is updated every time a user visits any page, potentially resulting in a high volume of expensive single document updates. It would be better to write the page counts to the local database and use a queue and worker system (like [Resque](https://github.com/resque/resque)) to push out updated page counts using the Bulk API every minute or so. This would be a much cheaper and more scalable approach, and would be just as effective. [block:api-header] { "title": "Shards" } [/block] A shard is the basic unit of work in Elasticsearch. If you haven't read the [Core Concept on Shards](doc:basic-glossary#shards), that would be a good place to start. Bonsai meters on the total number of shards in a cluster. That means both primary and replica shards count towards the limit. The relationship between shard scheme and your cluster's usage can sometimes not be readily apparent. For example, if you have an index with a 3x2 sharding scheme (3 primaries, 2 replicas), that's not 5 or 6 shards, it's 9. If this is confusing, read our [Shards and Replicas](doc:what-are-shards-and-replicas) documentation for some nice illustrations. If you have a shard overage, that can mean one of three things: 1. Extraneous indices. 2. Sharding scheme is not optimal. 3. Replication too high It is also possible that the cluster and its data are already configured according to best practices. In that case, you may need to get creative with aliases and data collocation in order to remain on your current subscription. The [Reducing Shard Usage](doc:reducing-shard-usage) document has information about all of these possibilities. If you find that you're unable to reduce shards through the options discussed on the [Reducing Shard Usage](doc:reducing-shard-usage) page, then you basically will need to upgrade to the next plan. You can find more information about updating your plan in the documentation below: * [Bonsai.io users](doc:managing-your-cluster#manage) * [Heroku users](doc:changing-your-plan) * [Manifold users](doc:changing-your-manifold-plan) [block:api-header] { "title": "Documents" } [/block] Bonsai meters on the number of documents in an index. There are several types of "documents" in Elasticsearch, but Bonsai only counts the live Lucene documents in primary shards towards the document limit. Documents which have been marked as deleted, but have not yet been merged out of the segment files do not count towards the limit. The phrase "live Lucene documents" may be a little confusing, but this is due to how Elasticsearch counts [nested documents](https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html). The [Index Stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html) is used to determine how many documents are in a cluster. The Index Stats API counts [nested documents](https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html) by including all associated documents. In other words, if you have a document with 2 nested documents, this is reported as 3 total documents. Elasticsearch has several different articles on how nested documents work, but the simplest answer is that it is creating the illusion of complex object by quietly creating multiple hidden documents. [block:callout] { "type": "info", "body": "A common point of confusion is that the `/_cat/indices` endpoint will show one set of documents, while the `/_stats` endpoint shows a much larger count. This is because the Cat API is counting the \"visible\" documents, while the Index Stats API is counting _all_ documents. The `_stats` endpoint is a more true representation of a cluster's document usage, and is the most fair to all users for metering purposes.", "title": "How Do I Have a Document Overage? I'm Way Under the Limit!" } [/block] There are several strategies for resolving a document overage: ### Remove Old Data If your index has a lot of old data, or "stale" data (documents which rarely show up in searches), then you could simply delete those documents. Deleted documents do not count against your limits. ### Remove an Index Occasionally users are indexing time series data, or database tables that are not actually being searched by the application. Audit your usage by using the Interactive Console to check the `/_cat/indices` endpoint. If you find that there are old or unecessary indices with data, then delete those. ### Compact Your Mappings Changing your mappings to nest less information can greatly reduce your document usage. Consider this sample document: [block:code] { "codes": [ { "code": "{\n \"title\": \"Spiderman saves child from well\",\n \"body\": \"Move over, Lassie! New York has a new hero. But is he also a menace?\",\n \"authors\": [ \n {\n \"name\": \"Jonah Jameson\",\n \"title\": \"Sr. Editor\",\n },\n {\n \"name\": \"Peter Parker\",\n \"title\": \"Photos\",\n }\n ],\n \"comments\": [ \n {\n \"username\": \"captain_usa\",\n \"comment\": \"I understood that reference!\",\n },\n {\n \"username\": \"man_of_iron\",\n \"comment\": \"Congrats on being slightly more useful than a ladder.\",\n }\n ],\n \"photos\": [ \n {\n \"url\": \"https://assets.dailybugle.com/12345\",\n \"caption\": \"Spiderman delivering Timmy back to his mother\",\n }\n ]\n}", "language": "json" } ] } [/block] Note that it's nesting data for authors, comments and photos. The mapping above would actually result in creating 6 documents. Removing the comments and photos (which usually don't need to be indexed anyway) would reduce the footprint by 50%. If you're using nested objects, review whether any of the nested information could stand to be left out, and then reindex with a smaller mapping. ### Upgrade the Subscription If you find that you're unable to remove any documents or indices, or change your mappings, then you will simply need to upgrade to the next subscription level. You can find more information about updating your plan in the documentation below: * [Bonsai.io users](doc:managing-your-cluster#manage) * [Heroku users](doc:changing-your-plan) * [Manifold users](doc:changing-your-manifold-plan) [block:api-header] { "title": "Disk" } [/block] Bonsai meters on the total amount of disk space a cluster can consume. This is for capacity planning purposes, and to ensure [multitenant](doc:architecture-classes#multi-tenant-class) customers have their fair share of resources. Bonsai calculates a cluster's disk usage by looking at the total data store size in bytes. This information can be found in the [Index Stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html). Resolving disk overages can be resolved in a couple different ways: ### Remove Stale Data / Indices There are some cases where one or more indices are created on a cluster for testing purposes, and are not actually being used for anything. These will count towards the data limits; if you're getting overage notifications, then you should delete these indices. [block:code] { "codes": [ { "code": "GET /_cat/indices\ngreen open prod20180101 1 1 1015123 0 32M 64M \ngreen open prod20180201 1 1 1016456 0 35M 70M \ngreen open prod20180301 1 1 1017123 0 39M 78M\ngreen open prod20180401 1 1 1018456 0 45M 90M \ngreen open prod20180501 1 1 1019123 0 47M 94M \ngreen open prod20180601 1 1 1020456 0 51M 102M ", "language": "text" } ] } [/block] Removing the old and unneeded indices in the example above would free up 356MB. A single command could do it: [block:code] { "codes": [ { "code": "# Delete a group of indices:\nDELETE /prod20180101,prod20180201,prod20180301,prod20180401,prod20180501", "language": "text" } ] } [/block] ### Purge Deleted Documents Data in Elasticsearch is spread across lots of files called segments. Segments each contain some number of documents. An index could have dozens, hundreds or even thousands of segment files, and Elasticsearch will periodically merge some segment files into others. When a document is deleted in Elasticsearch, its segment file is simply updated to mark the document as deleted. The data is not actually removed until that segment file is merged with another. Elasticsearch normally handles segment merging automatically, but forcing a segment merging will reduce the overall disk footprint of the cluster by eliminating deleted documents. Normally this is done through the Optimize / Forcemerge API, but this is unfortunately one of Bonsai's [Unsupported API Endpoints](doc:bonsai-unsupported-actions). The same effect can be accomplished however, by simply reindexing. Reindexing will cause the data to be refreshed, and no deleted documents will be tracked by Elasticsearch. This will reduce disk usage. To check whether this will work for you, look at the `/_cat/indices` data. There is a column called `docs.deleted`, which shows how many documents are sitting on the disk and are marked as deleted. This should give a sense of how much data could be freed up by reindexing. For example: [block:code] { "codes": [ { "code": "health status index pri rep docs.count docs.deleted store.size pri.store.size\ngreen open my_index 3 2 15678948 6895795 47.1G 15.7G", "language": "text" } ] } [/block] In this case, the `docs.deleted` is around 30% of the primary store, or around 4.8G of primary data. With replication, this works out to something like 14.4GB of total disk marked for deletion. Reindexing would reduce the cluster's disk footprint by this much. The result would look like this: [block:code] { "codes": [ { "code": "health status index pri rep docs.count docs.deleted store.size pri.store.size\ngreen open my_index 3 2 15678948 0 32.7G 10.9G", "language": "text" } ] } [/block] [block:callout] { "type": "info", "title": "Protip: Queue Writes and Reindex in the Background for Minimal Impact", "body": "Your app's search could be down or degraded during a reindex. If reindexing will take a long time, that may make this option unfeasible. However, you could minimize the impact by using something like Kafka to queue writes while reindexing to a new index.\n\nSearch traffic can continue to be served from the old index until its replacement is ready. Flush the queued updates from Kafka into the new index, then destroy the old index and use an alias to promote the new index.\n\nThe tradeoff of this solution is that you'll minimize the impact to your traffic/users, but you'll need to set up and manage the queue software. You'll also have a lot of duplicated data for a short period of time, so your footprint could be way above the subscription limit for a short time. \n\nTo prevent the state machine from disabling your cluster, you might want to consider temporarily upgrading to perform the operation, then downgrading when you're done. Billing is prorated, so this would not add much to your invoice. You can always [email us](mailto:support@bonsai.io) to discuss options before settling on a decision." } [/block] ### Reindex with Smaller Mappings [Mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) define how data is stored and indexed in an Elasticsearch cluster. There are some settings which can cause the disk footprint to grow exponentially. For example, [synonym expansion](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-synonym-tokenfilter.html) can lead to lots of extra tokens to be generated per input token (if you're using WordNet, see our documentation article on it, specifically [Why Wouldn't Everyone Want WordNet?](doc:using-wordnet-with-bonsai#why-wouldnt-everyone-want-wordnet)). If you're using lots of index-time synonym expansion, then you're essentially inflating the document sizes with lots of data, with the tradeoff (hopefully) being improved relevancy. Another example would be [Ngrams](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-ngram-tokenizer.html). Ngrams are tokens generated from the parts of other tokens. A token like "hello" could be broken into 2-grams like "he", "el", "ll", and "lo". In 3-grams, it would be "hel", "ell" and "llo". And so on. The [Elasticsearch Guide](https://www.elastic.co/guide/en/elasticsearch/guide/current/_ngrams_for_partial_matching.html) has more examples. It's possible to generate multiple gram sizes for a single, starting with values as low as 1. Some developers use this to maximize substring matching. But there is an exponential growth in the number of grams generated for a single token: [block:image] { "images": [ { "image": [ "https://files.readme.io/043ff10-gram-vs-token.png", "gram-vs-token.png", 605, 340, "#163553" ] } ] } [/block] This relationship is expressed mathematically as: [block:image] { "images": [ { "image": [ "https://files.readme.io/3a7a58a-ngram-pattern.png", "ngram-pattern.png", 1500, 554, "#0795fc" ] } ] } [/block] In other words, a token with a length of 5 and a minimum gram size of 1 would result in `(1/2)*5*(5+1)=15` grams. A token with a length of 10 would result in 55 grams. The grams are generated _per token_, which leads to an explostion in terms for a document. As a sample calculation: if a typical document in your corpus has a field with ~1,000 tokens and a [Rayleigh distribution](https://en.wikipedia.org/wiki/Rayleigh_distribution) of length with an average of ~5, you could plausibly see something like a 1,100-1,200% inflation in disk footprint using Ngrams of minimum size 1. In other words, if the non-grammed document would need 100KB on disk, the Ngrammed version would need over 1MB. Virtually none of this overhead would improve relevancy, and would probably even hurt it. [Nested documents](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html) are another example of a feature can also increase your data footprint without necessarily improving relevancy. The point is that there are plenty of features available that lead to higher disk usage than one might think at first glance. Check on your mappings carefully: look for large synonym expansions, make sure you're using Ngrams with a minimum gram size of 3 or more (also look into [EdgeNGrams](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenizer.html) if you're attempting autocomplete), and see if you can get away with fewer nested objects. Reindex your data with the updated mappings, and you should see a definite improvement. ### Upgrade the Subscription If you find that you're unable to remove data, reindex, or update your mappings -- or that these changes don't yeild a stable resolution -- then you will simply need to upgrade to the next subscription level. You can find more information about updating your plan in the documentation below: * [Bonsai.io users](doc:managing-your-cluster#manage) * [Heroku users](doc:changing-your-plan) * [Manifold users](doc:changing-your-manifold-plan) [block:api-header] { "title": "Memory" } [/block] Bonsai meters on the amount of memory that a cluster can use. There are several categories involving memory usage. For metering purposes, Bonsai uses the sum of the following catagories to determine memory footprint: * segments * fielddata * request_cache / query_cache The exact amount of memory used by each category can be found in the [Index Stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html). Reducing memory footprint can be tricky, although fortunately overages of this class are rare. There are a couple of possible resolutions: ### Reduce Segment Files Data in Elasticsearch is spread across lots of files called segments. Segments each contain some number of documents. An index could have dozens, hundreds or even thousands of segment files, and Elasticsearch will periodically merge some segment files into others. An interesting subject is how documents are deleted in Elasticsearch; when a document is deleted, the segment file that contains it is simply updated to mark the document as deleted. The data is not actually removed until that segment file is merged with another. So segment merging not only reduces the number of files Elasticsearch is tracking, but it will also reduce the overall footprint of the cluster, both in memory and on disk. Merging can be forced through the Optimize / Forcemerge API, but this is unfortunately one of Bonsai's [Unsupported API Endpoints](doc:bonsai-unsupported-actions). The same effect can be accomplished however, by simply reindexing. Reindexing will cause the data to be refreshed, and no deleted documents will be tracked by Elasticsearch. This will reduce memory usage. ### Disable fielddata The Elasticsearch documentation has an [awesome article](https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html) explaining fielddata: > Most fields are indexed by default, which makes them searchable. Sorting, aggregations, and accessing field values in scripts, however, requires a different access pattern from search. > > Search needs to answer the question "Which documents contain this term?", while sorting and aggregations need to answer a different question: "What is the value of this field for this document?". > > Most fields can use index-time, on-disk doc_values for this data access pattern, but text fields do not support doc_values. > > Instead, text fields use a query-time in-memory data structure called fielddata. This data structure is built on demand the first time that a field is used for aggregations, sorting, or in a script. It is built by reading the entire inverted index for each segment from disk, inverting the term ↔︎ document relationship, and storing the result in memory, in the JVM heap. > ... > Fielddata can consume a lot of heap space, especially when loading high cardinality text fields. Once fielddata has been loaded into the heap, it remains there for the lifetime of the segment. Also, loading fielddata is an expensive process which can cause users to experience latency hits. This is why fielddata is disabled by default. If you're using fielddata for your text fields, then this can consume an inordinate amount of memory. Disabling it and reindexing should reduce your memory footprint considerably. ### Clear the Caches When Elasticsearch performs a search request, it will automatically cache a shard's local results to memory. It does not cache the complete document, but rather a bit-string representation. A bit-string is a data structure that has a 1 or a 0 representing whether a document ID matches This can still lead to large cache filters if the cluster contains lots of documents. A corpus of a million documents will lead to filter sizes of roughly 125KB. Each unique query will result in a new bit string; if there are also lots of unique searches happening in a short period of time, these can add up quickly. The cache Elasticsearch builds is invalidated whenever the shard refreshes (but only when there are changes to the shard). This happens every 1s by default, but the value can be increased. So if your cluster for some reason has a high refresh interval, then the caches can lead to a higher memory footprint. Reducing the [refresh interval ](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html#bulk) will lead to more frequent cache invalidation, which would reduce the amount of memory used by the caches. Alternatively, you can use the [/_cache/clear API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-clearcache.html) to expire the caches manually. ### Upgrade the Subscription If you find that you're unable to reindex, disable fielddata or clear your caches -- or that these changes don't yeild a stable resolution -- then you will simply need to upgrade to the next subscription level. You can find more information about updating your plan in the documentation below: * [Bonsai.io users](doc:managing-your-cluster#manage) * [Heroku users](doc:changing-your-plan) * [Manifold users](doc:changing-your-manifold-plan)