In this scenario, servers write their logs to a local drive, and a forwarder process monitors these logs. This is the typical Splunk installation.
The advantages of this approach include the following:
- This process is highly optimized. If the indexers are not overworked, events are usually searchable within a few seconds.
- Slowdowns caused by network problems or indexer overload are handled gracefully. The forwarder process will pick up where it left off when the slowdown is resolved.
- The agent is light, typically using less than 100 megabytes of RAM and a few percent of one CPU. These values go up with the amount of new data written and the number of files being tracked. See inputs.conf in Chapter 11, Configuring Splunk, for details.
- Logs without a time zone specified will inherit the time zone of the machine running the forwarder. This is almost always what you want.
- The hostname will be picked up automatically from the host. This is almost always what you want.
The disadvantages of this approach include the following:
- The forwarder must be installed on each server. If you have a system for distributing software already, this is not a problem. We will discuss strategies in the Deploying the Splunk binary section.
- The forwarder process must have read rights to all the logs to be indexed.
This is usually not a problem but does require some planning. This typical deployment looks like the following diagram:
If your log volume exceeds 100 gigabytes of logs produced each day, you need to think about multiple indexers. We will talk about this further in the Sizing indexers section.