Dropping events

Some events are simply not worth indexing. The hard part is figuring out which ones these are and making very sure you're not wrong. Dropping too many events can make you blind to real problems at critical times, and can introduce more problems than tuning Splunk to deal with the greater volume of data in the first place.

With that warning stated, if you know what events you do not need, the procedure for dropping events is pretty simple. Say we have an event such as this one:

2012-02-02 12:24:23 UTC TRACE Database call 1 of 1,000. [...] 

I know absolutely that, in this case and for this particular source type, I do not want to index TRACE-level events.

In props.conf, I will create a stanza for my source type, as shown here:

[mysourcetype] 
TRANSFORMS-droptrace=droptrace 

Then, I will create the following transform in transforms.conf:

[droptrace] 
REGEX=^d{4}-d{2}-d{2}s+d{1,2}:d{2}:d{1,2}s+[A-Z]+sTRACE 
DEST_KEY=queue 
FORMAT=nullQueue 

Splunk compares nullQueue to nulldevice, which (according to the product documentation) tells Splunk not to forward or index the filtered data.

This REGEX attribute is purposely as strict as I can make it. It is vital that I do not accidentally drop other events, and it is better for this brittle pattern to start failing and to let through TRACE events rather than for it to do the opposite.