Let the Data speak for itself. A better Grafana dashboard
Why you should have a dashboard
- Make better decisions based on data
- Consolidate signals into a single location
- Important information is front-and-center
- Even non-technical audiences can use it
- You can solve problems in your system faster
- You will see problems you never knew about before, before they cause lasting harm
- Reduces the need for ad-hoc querying
Our objective
Use Grafana dashboards to track your performance metrics and easily visualize the health of your system
How to create better dashboards
5 Second Rule
The most important metrics should immediately pop from the screen
Be Picky
- Don’t overload!
- What will help you spot issues quickly?
- Move the more detailed panels to the bottom
Tell A Story
- Choose metrics that work together
- Consider multiple audiences
What is worth monitoring?
Okay Examples
- Any time-series data you can get your hands on
- General metrics that are available, but that you’re unfamiliar with
- Really cool looking charts
Better Examples
- Use metrics that tell a story when put together
- What questions will they answer?
- Cover the Four Golden Signals [1]
- Latency, Traffic, Errors, Saturation
- Indicator of issues
- High level metrics
- Detailed views are okay, but should be lower on the page or just linked to
- Don’t assume you’ll know the problem you’re looking for
- Quickly point the user in the right direction
Drilling in with variables
- Variables add interactivity to your dashboard
- Toggle data sources, environments, regions, customers, etc.
- Hide the data you don’t care about - focus on what you do.
- Can be static or dynamic list based on live queries against your data sources
- Can be nested to drill-in deeper
Use the metrics you’re already alerting on
- Cloudwatch Metrics
- Many of the graphs you find in the AWS console can be easily recreated in Grafana.
- Multiple metrics can be combined into a single chart.
Your DB as a Data Source
- Connect to Postgres, MySQL, or other DB
- Design your own queries around tabular or time series data
- Lots of flexibility
- Write your own SQL with template variables to build visualizations right on top of your data
Other Interesting Data Sources
Grafana supports many additional data sources. Here’s a few more to try:
- Tracking MRs in GitLab
- Collect your own metrics
- Additional data source plugins can be added to Grafana. Listed here:
- Additional visualization panel types are also available. Listed here:
Resources
- https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/
- Monitoring Distributed Systems
- https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/#xref_monitoring_golden-signals
- Four Golden Signals
- https://www.sisense.com/blog/4-design-principles-creating-better-dashboards/
- 5 second rule
- tell a clear story
- express the meaning of the data
- anti-patterns
- too many widgets, basic questions take too long to answer, no organizing principle
- https://techbeacon.com/devops/10-monitoring-talks-every-developer-should-watch
- "10 monitoring talks that every developer should watch"
- https://youtu.be/czes-oa0yik
- metrics, metrics everywhere
- importance of metrics, improve mental model of software using data by measuring
- 2nd half of talk is more focused on how to use metrics better
- https://youtu.be/zLjhFrUhqxg
- What should I monitor, and how should I do it?
- metrics, metrics everywhere
- https://grafana.com/blog/2019/06/10/how-not-to-fail-at-visualization/
- How Not to Fail at Visualization
- Points out lots of anti-patterns
- How Not to Fail at Visualization
- https://grafana.com/blog/2019/05/16/worth-a-look-public-grafana-dashboards/