When setting up a Splunk indexer cluster, you stipulate the number of copies of data that you want the cluster to maintain. Peer nodes store incoming data in buckets, and the cluster maintains multiple copies of each bucket. The cluster stores each bucket copy on a separate peer node. The number of copies of each bucket (that the cluster maintains) is known as the Splunk replication factor.
Let's try to explain this concept (of the replication factor) with a highly simplified example.
Keep in mind that a cluster (of indexes) can tolerate a failure of 1 less than your total replication factor. So, if you have configured 3 peer indexes, you have a replication factor of 3 so your failure tolerance is 2.
By configuring additional peer nodes, you are telling Splunk to store identical copies of each bucket (of indexed data) on separate nodes, and therefore, increase your failure tolerance (that is, concurrent failures). For the most part, this seems logical and straightforward; more the peer nodes—higher is the failure tolerance.
The trade-off is that you need to store and process all those copies of data. From the Splunk website, we have:
Of course, in reality, a Splunk environment is nearly never so simplistic. In most production Splunk environments, the following should be considered:
In most clusters, each of the peer nodes would be functioning as both source and target peers, receiving external data from a forwarder as well as replicated data from other peers.
To accommodate horizontal scaling, a cluster with a replication factor of 3 could consist of many more peers than three. At any given time, each source peer would be streaming copies of its data to two target peers, but each time it started a new hot bucket, its set of target peers could potentially change.