Creating indexes

Indexes are where Splunk Enterprise stores all the data it has processed. It is essentially a collection of databases that are, by default, located at $SPLUNK_HOME/var/lib/splunk. Before data can be searched, it needs to be indexed—a process we describe here.

Tip from the Fez: There are a variety of intricate settings which can be manipulated to control size and data management aspects of an index.  We will not cover those in this book, however as your situation requires complexity, be sure to consider a variety of topics around index management, such as overall size, buckets parameters, archiving and other optimization settings. 

There are two ways to create an index, through the Splunk user interface or by creating an indexes.conf file. You will be shown here how to create an index using the Splunk portal, but you should realize that when you do that, it simply generates an indexes.conf file.

When adding and making changes to configurations in the Splunk user interface, those updates will commonly be stored in a configuration file (.conf) somewhere under the $SPLUNK_HOME directory.

You will be creating an index called winlogs to store a sample Windows perfmon log. To do this, take the following steps:

  1. In the Splunk navigation bar, go to Settings.
  2. In the Data section, click on Indexes, which will take you to the Indexes page.
  3. Click on the New Index button in the upper-right corner.
  4. Fill out the information for this new index as seen in the following screenshots, carefully going through steps 1 to 6. You will need to scroll down in the window to complete all the steps.

The following screenshot displays the first three steps to be followed:

The next screenshot indicates step 4 and step 5 to be followed:

  1. Be sure to Save when you are done.

You will now see the new index in the list as shown here:

The preceding steps have created a new indexes.conf file.

Now go ahead and inspect this file. In Windows this can be done through Notepad. In Linux, you can use a visual text editor such as Notepad++ to connect to your Linux server or, at the command line, use vi.

The specific indexes.conf to open will be found in $SPLUNK_HOMEetcappsdestinationslocal. Specifying the destinations app for the index is what placed the indexes.conf file below the destinations directory.

Tip from the Fez: As you build your Splunk environment, organize your content by application. This will ensure that the configurations can be stored and managed consistently as needed, as opposed to storing all configurations inside a single application. The challenges with this approach will become more evident as your Splunk environment grows.

Every index has specific settings of its own. Here is how your index looks when automatically configured by the portal. In production environments, this is how Splunk administrators manage indexes:

[winlogs] 
coldPath = $SPLUNK_DBwinlogscolddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DBwinlogsdb 
maxTotalDataSizeMB = 100 
thawedPath = $SPLUNK_DBwinlogsthaweddb
Note that the maximum size value of 100 that you specified is also indicated in the configuration.

The complete indexes.conf documentation can be found at http://docs.splunk.com/Documentation/Splunk/latest/admin/indexesconf.