Upgrading Major Versions

Bonsai makes upgrading major versions of Elasticsearch as painless as possible. There is no need to manage the operational details of deploying software upgrades to nodes or spinning up new servers. With Bonsai, version upgrades are instant and can be performed with zero downtime.

In this document, we’ll cover some general best practices and offer some Bonsai-specific guidelines for migrating your app to a new version of Elasticsearch.

Use one Elasticsearch cluster per environment

This means having one cluster for production, and another for staging, another for development, and so on. Some users like to put staging indices alongside production indices on the same cluster in order to ensure identical behaviors between staging and production applications. First, this is a terrible idea in general; you should never run staging/dev applications on the same resources as production. This is a recipe for disaster.

Second, separating out your environments allows for upgrading them one at a time. While this may sound tedious, it is the most prudent approach and allows you to discover potential problems before they impact production.

Read the Release Notes Carefully

Make sure to perform your due diligence by reading the release notes that accompany the version you’re targeting for the upgrade. It’s also a good idea to read the release notes for the versions between your current version and target version. Bonsai has some tools to facilitate this part, discussed later in this document.

Another thing to investigate is whether your application’s Elasticsearch client supports the upgrade candidate. There have been cases where a popular client or framework was several support versions behind the official Elasticsearch release; some of these have resulted in hours of down time for users who upgraded the Elasticsearch cluster beyond the version supported by their Elasticsearch client.

Validate In Development

Upgrading across major versions sometimes comes with breaking changes, new dependencies, and tweaks in behavior. It is important to validate that the upgrade is safe before pushing it out to production.

One tool Bonsai has to facilitate this is the Migration Checker. This checker will run a test suite against any cluster to confirm compatibility with the upgrade candidate. To use the checker, navigate to your new cluster dashboard and click the ‘Run Migration Tests’ button:

A new window will open up and automatically run tests against your indices:

If the checker yields failing tests, you should correct each issue before upgrading. The checker provides links to documentation to help you make changes to your data and index mappings prepare for migration. Click on the “docs” link next to each line-item test to read about break changes and how to fix them in your integration.

If all checks pass, you can start your upgrade process. We advise starting by upgrading the least critical environment first. Ensure it works as expected, and make any changes as needed. Deploy those changes to the next least critical environment and then upgrade the cluster for that environment. Continue in this fashion until reaching the production environment. By that point, you should be fairly confident that the application and search will work as expected.

Make sure to validate that searches will work as expected in terms of relevancy. Also make sure to test full deletion and reindexing in the least critical environments before upgrading production. Reindexing is something you should be familiar with anyway as a part of normal usage of Elasticsearch, such as changing analyzer settings, backfilling a new field, or - in this case - upgrading to a new major version.

Launch to production!

Once you are satisfied that the candidate version will work in production as expected, the final step is to take it live. There are two ways to do this: upgrading in place, or provisioning a new cluster altogether. The latter is preferred because it not only allows for a zero-downtime switch, but it also provides a fallback in case something breaks.

Provision a new production cluster (preferred)

The first step is to provision a new cluster running on the target version of Elasticsearch. Once you have your new production cluster ready to go, you’ll need to index it. Check the documentation for your language/framework for specifics on how to perform this action.

When the new cluster is fully populated, update the application to serve traffic to the new cluster. This generally involves pointing the Elasticsearch client to the new cluster URL. How this is performed will depend heavily on the application stack, so consult the documentation for your Elasticsearch client.

The old cluster will remain as a back up in case anything goes wrong after the switch. When you have satisfied yourself that the application is running smoothly in production on the new version of Elasticsearch, you can deprovision the old cluster.

We’re Pros at Proration

When you deprovision your cluster, Bonsai will issue a service credit for time unused on the subscription. This ensures that your hard earned cash isn’t wasted in the process of upgrading to a new version of Elasticsearch.

Upgrade the current production cluster There are cases where the preferred approach is not possible. For example, if your application is not able to serve traffic to one cluster while indexing to another in the background. Costs may also be a concern, or the cluster may be running on a subscription that is no longer available and you would like to keep it.

Whatever the reason, the existing cluster can be upgraded in place. A Bonsai cluster can not be upgraded to a new version when it is populated with data. Bonsai will show the following error message if there is data in a cluster to be upgraded:

Data can be removed from the cluster using a tool like curl or the Interactive Console:

# See the current indices:
GET /_cat/indices
green open users_production       1 1 1015123 0  32M  64M
green open notes_production       1 1 1016456 0  35M  70M
green open comments_production    1 1 1017123 0  39M  78M

# Delete the existing indices
DELETE /users_production,notes_production,comments_production

If there is no data in the cluster, the warning message will be a little different and the upgrade button will be blue:

Clicking on the blue upgrade button will upgrade your cluster to the target version. This action is irreversible. Once the cluster is on the new version, it will need to be reindexed.

Ask Support

Upgrading major versions of Elasticsearch while running a production application can be tricky. If you’re unsure of what to do, are concerned about an edge case or special circumstance, or simply want to sanity check a plan, please do not hesitate to reach out to support@bonsai.io. We’re here to help ensure the smoothest upgrade possible.