As a Splunk administrator, one of the most important responsibilities is to be responsible for the data. As a custodian of data, a Splunk admin has significant influence over how to interpret and present information to users. It is common for the administrator to create the first few dashboards. A more mature implementation, however, requires collaboration to create an output that is beneficial to a variety of user requirements, and may be completed by a Splunk development resource with limited administrative rights.
Make it a habit to consistently request users input regarding the Splunk delivered dashboards and reports and what makes them useful. Sit down with day-to-day users and lay out, on a drawing board for example, the business process flows or system diagrams to understand how the underlying processes and systems you're trying to measure really work. Look for key phrases like these, which signify what data is most important to the business:
- If this is broken, we lose tons of revenue...
- This is a constant point of failure...
- We don't know what's going on here...
- If only I can see the trend, it will make my work easier...
- This is what my boss wants to see...
Splunk dashboard users may come from many areas of the business. You want to talk to all the different users, no matter where they are on the organizational chart. When you make friends with the architects, developers, business analysts, and management, you will end up building dashboards that benefit the organization, not just individuals. With an initial dashboard version, ask for users thoughts as you observe them using it in their work and ask what can be improved upon, added, or changed.
We hope that at this point, you realize the importance of dashboards and are ready to get started creating some, as we will do in the following sections.