Skip to main content

Shard Primer

Rob Sears

For people new to Elasticsearch, shards can be a bit of a mystery. Why is it possible to add or remove replica shards on demand, but not primary shards? What’s the difference? How many indices can fit on a shard?

This article explains what shards are, how they work, and how they can best be used.

Where do shards come from?

First, a little bit of background: Elasticsearch is built on top of Lucene, which is a data storage and retrieval engine. What are called “shards” in Elasticsearch parlance are technically Lucene instances. Elasticsearch manages these different instances, spreading data across different instances, and automatically balancing those instances across different nodes.

Whenever an Elasticsearch index is created, that index will be composed of one or more shards. This means that an Elasticsearch index can spread data across several Lucene instances. This architecture is useful for both redundancy and parallelization purposes.

Shards play one of two roles: primary or replica. Primary shards are a logical partitioning of the data in the index, and are fixed at the time that the index is created. Primary shards are useful for parallelization; when a large amount of data is split across several primary shards, a node can run a query on several Lucene instances in parallel, reducing the overall time of the job.

Primary Shards Can Not Be Added/Removed Later

From one of our more popular blog entries: “Elasticsearch uses a naive hashing algorithm to route documents to a given primary shard. This design choice allows documents to be randomly distributed in a reproducible way. This avoids “hot spots” that affect performance and overallocation. However, it has one major downside, which is that the number of primary shards can not be changed after an index has been created. Replicas can be added and removed at will, but the number of primary shards is basically written in stone.

In contrast, replica shards are simply extra copies of the data. They are useful for redundancy or to handle extra search traffic, and can be added and removed on demand.

You can specify how many primary shards and replicas are used when creating a new index.

PUT /my_index/_settings
{
  "number_of_shards": 1,
  "number_of_replicas": 2
}

Replicas and High-Availability

Replica shards are not only capable of serving search traffic, but they also provide a level of protection against data loss. If a node hosting a primary shard is taken offline for some reason, Elasticsearch will promote its replica to a primary role, if a replica exists. However, if the index’s replication is set to 0, then it is not in a High-Availability configuration. In the event of a data loss incident, the data will simply be lost.

Replicas are a multiplier on the primary shards, and the total is calculated as primary * (1+replicas). In other words, if you create an index with 3 primary shards and 2 replicas, you will have 9 total shards, not 5 or 6.

Measuring your cluster’s index and shard usage

Elasticsearch offers some API endpoints to explore the state of your indices and shards. The _cat APIs are helpful for human interaction. You can view your index states by visiting /_cat/indices, which will show index names, primary shards and replicas. You can also inspect individual shard states and statistics by visiting /_cat/shards. See example output below:

$ curl -s https://user:password@bonsai-12345.bonsai.io/_cat/indices?v
health status index  pri rep docs.count docs.deleted store.size pri.store.size
green  open   images   1   0          0            0       130b           130b
green  open   videos   1   0          0            0       130b           130b
green  open   notes    1   0          0            0       130b           130b

$ curl -s https://user:password@bonsai-12345.bonsai.io/_cat/shards?v
index  shard pri rep state   docs store ip              node      
images 0     p       STARTED    0  130b XXX.XXX.XXX.XXX Sugar Man
notes  0     p       STARTED    0  130b XXX.XXX.XXX.XXX Sugar Man
videos 0     p       STARTED    0  130b XXX.XXX.XXX.XXX Sugar Man

We’ve also made this easy by creating a live interactive console for you. Just visit your cluster’s dashboard console, chose GET from the dropdown, and run /_cat/indices?v or /_cat/shards?v.

Frequently Asked Questions About Shards

Q. How Many Indices Can I Put On A Shard?

Short Answer: An index can have many shards, but any given shard can only belong to one index.

Long Answer: It is possible to “fake” multiple indices per shard by using “type” fields and index aliases. There is a section of the Elasticsearch Guide dedicated to this approach. Essentially, if you have several data models, you could put two or more into a single index instead of separate indices, and use aliases to perform automated filtering of queries.

The downside to this approach is that it requires a fair amount of work. Most frameworks and content management systems with integrated Elasticsearch clients will not take this approach by default. Developers will need to create and manage the index settings manually instead of relying on automated tools.

Q. How Many Shards Do I Need?

Answer: It depends. Because replicas can be added/removed at will, the real question is how many primary shards are needed. Increasing the number of primary shards for an index is one way to improve performance because it allows the query to be processed in parallel. But there are a lot of reasons to not have a kajillion primary shards.

Generally, we recommend that if you don’t expect data to grow significantly, then:

  • One primary shard is fine if you have less than 100K documents
  • One primary shard per node is good if you have over 100K documents
  • One primary shard per CPU core is good if you have at least a couple million documents

If you anticipate lots of growth – orders of magnitude within a short time – then the problem is a little more complicated. Take a read through the Ideal Elasticsearch Index post, especially the benchmarking section. Or shoot an email to support@bonsai.io.

Q. How Do I Reduce The Number of Shards I’m Using?

Answer: There is an article dealing with this very subject: Reducing Shard Usage.

Related articles