Configurations being used outside of search are merged in a fairly simple order. These configurations include the files to read, the indexed fields to create, the indexes that exist, deployment server and client configurations, and other settings. These configurations merge in this order:
- $SPLUNK_HOME/etc/system/default: This directory contains the base configurations that ship with Splunk.
Never make changes in $SPLUNK_HOME/etc/system/default as your changes will be lost when you upgrade Splunk.
- $SPLUNK_HOME/etc/apps/*/default: Configurations are overlaid in the reverse ASCII order by app directory name, that is, a beats z.
- $SPLUNK_HOME/etc/apps/*/local.
- $SPLUNK_HOME/etc/system/local:
- The configurations in this directory are applied last.
- Outside of search, these configurations cannot be overridden by an app configuration. Apps are a very convenient way to compartmentalize control and distribute configurations. This is particularly relevant if you use the deployment server, which we will cover in Chapter 12, Advanced Deployments.
Don't edit configurations in $SPLUNK_HOME/etc/system/local even if you have a very specific reason. An app is almost always the correct place for configuration.
A little pseudocode to describe this process might look like this:
$conf = new Configuration('$SPLUNK_HOME/etc/')
$conf.merge( 'system/default/$conf_name' )
for $this_app in reverse(sort(@all_apps)):
$conf.merge( 'apps/$this_app/default/$conf_name' )
for $this_app in reverse(sort(@all_apps)):
$conf.merge( 'apps/$this_app/local/$conf_name' )
$conf.merge( 'system/local/$conf_name' )