Always remember to filter your searches by index. Not restricting your search to a specific index makes Splunk go through all available indexes, consuming unnecessary time and resources. The same can be said about filters for sourcetype if your searches only need to consider a specific set of data that resides in an index with many sourcetypes.
A normal question arises when designing your Splunk implementation about how many indexes to have and what data goes into each. Careful thought needs to be taken when planning for indexes and when you create a new index.
For example, all web server logs for the same software application can be placed in one index. You may then split the log types by sourcetype but keep them within the same index. This will give you a generally favorable search speed even if you have to search between two different source types.
Consider this example:
| Index name | Source type |
| App1 | Logs.Error |
| App1 | Logs.Info |
| App1 | Logs.Warning |
| App2 | Logs.Error |
| App2 | Logs.Info |
| App3 | Logs.Warning |
As you can see, we have indexed by app number first and then created various sourcetypes. You may then create a search within the same index, even if you have to combine two sourcetypes:
- A good search in this example:
SPL> index=App1 sourcetype=Logs.Error OR Logs.Warning
- A bad search:
SPL> sourcetype=Logs.* Error
The way we have set it up here, if you ever have to retrieve data from both indexes, you can combine them with the following query. It is not as efficient as searching against a single index, but it is better than going through all other available indexes:
SPL> index=App1 OR index=App2 sourcetype=Logs.Error