Summary indexing

In a matter of days, Splunk will accumulate data and start to move events through the bucketing process. With the millions or billions of events that are typical with a robust Splunk implementation, you can start to understand how searches run over long-time horizons can slow down.

There are two ways to circumvent this problem. In addition to search acceleration, completed earlier in this chapter, faster search results on large amounts of data can be achieved through summary indexing.

With summary indexing, you run a scheduled search and output the results into a different index, often called summary. The result will only show the computed statistics of the search. This results in a very small subset of data that will be much faster to retrieve and report on than going through a large set of detailed event records and summarizing results on the fly. This concept is not unlike the notion of creating aggregate tables or pre-calculated views in a relational database management system, where data is pre-calculated and results stored for access at some point in the future.

Say, for example, you wish to keep track of payment error counts and you wish to keep the data in the summary index. Follow these steps:

  1. From your Destinations app, go to Settings | Searches, reports, and alerts.
  2. Click on the New Report button to create a new scheduled search.
  3. Use the following information as a guide:
    • Title: Summary of Payment Errors
    • Search: eventtype=bad_payment | stats count
    • Earliest time: -2m@m
    • Latest time: now
    • App: Destinations
    • Time Range Picker: No
  4. Click on Save when complete. Use the following screenshot as a guide:

Now, perform the following steps:

  1. Click on the Edit drop-down menu next to the report you just created and select Edit Schedule
  2. Change Schedule type to Cron
  3. Set Cron schedule to */2 * * * *
  4. Click on Save

Use the following screenshot as a guide:

Now perform the following steps:

  1. Again, click on the Edit drop-down menu for the report and select Edit Summary indexing.
  2. Check the box to start the process of enabling summary indexing.
  3. Add a new field in the Add fields section, where values will be summaryCount equals to count.

Use the following information as a guide:

  1. Save when you are ready to continue.
  2. Wait just a few minutes, and then go back to the Destinations app's search page. Type in and execute the following search:
SPL> index=summary search_name="Summary of Payment Errors" | table _time,count

The statistics tab will appear first when showing the results of the search, due to the table command. Clicking on the Events table will show the events loaded into the summary index. Notice these events are stripped of the original event fields. Also notice that even if there were zero results returned during the two minute window, an event is stored with count=0. We will use this information in later chapters to create optimized dashboards.