Splunk's configuration files live in $SPLUNK_HOME/etc. This is reminiscent of Unix's /etc directory but is instead contained within Splunk's directory structure.
This has the advantage that the files don't have to be owned by root. In fact, the entire Splunk installation can run as an unprivileged user (assuming you don't need to open a port below 1024 or read files only readable by another user).
The directories that contain configurations are as follows:
- $SPLUNK_HOME/etc/system/default: These are the default configuration files that ship with Splunk. Never edit these files as they will be overwritten each time you upgrade.
- $SPLUNK_HOME/etc/system/local: This is the location of the global configuration overrides specific to this host. There are very few configurations that need to live here—most configurations that do live here are created by Splunk itself. In almost all cases, you should make your configuration files inside an app.
- $SPLUNK_HOME/etc/apps/$app_name/default: This is the proper location for configurations in an app that will be shared either through Splunkbase or otherwise.
- $SPLUNK_HOME/etc/apps/$app_name/local: This is where most configurations should live and where all non-private configurations created through the web interface will be placed.
- $SPLUNK_HOME/etc/users/$user_name/$app_name/local: When a search configuration is created through the web interface, it will have a permission setting of Private and will be created in a user- or app-specific configuration file. Once permissions are changed, the configuration will move to the corresponding directory named $app_name/local.
There are a few more directories that contain files that are not .conf files. We'll talk about those later in this chapter, in the User interface resources section.