Let the Data speak for itself. A better Grafana dashboard

Make better decisions based on data. Consolidate signals. Important information front-and-center. Use Grafana dashboards to track your performance metrics and easily visualize the health of your system.

Why you should have a dashboard

● Make better decisions based on data
● Consolidate signals into a single location
● Important information is front-and-center
● Even non-technical audiences can use it
● You can solve problems in your system faster
● You will see problems you never knew about before, before they cause lasting harm
● Reduces the need for ad-hoc querying

Our objective

Use Grafana dashboards to track your performance metrics and easily visualize the health of your system

How to create better dashboards

5 Second Rule
The most important metrics should immediately pop from the screen

Be Picky
● Don’t overload!
● What will help you spot issues quickly?
● Move the more detailed panels to the bottom

Tell A Story
● Choose metrics that work together
● Consider multiple audiences

What is worth monitoring?

Okay Examples
● Any time-series data you can get your hands on
● General metrics that are available, but that you’re unfamiliar with
● Really cool looking charts

Better Examples
● Use metrics that tell a story when put together
○ What questions will they answer?
● Cover the Four Golden Signals [1]
○ Latency, Traffic, Errors, Saturation
● Indicator of issues
● High level metrics
○ Detailed views are okay, but should be lower on the page or just linked to
● Don’t assume you’ll know the problem you’re looking for
○ Quickly point the user in the right direction

Drilling in with variables

Variables add interactivity to your dashboard
Toggle data sources, environments, regions, customers, etc.
Hide the data you don’t care about - focus on what you do.
● Can be static or dynamic list based on live queries against your data sources
● Can be nested to drill-in deeper

Use the metrics you’re already alerting on

Cloudwatch Metrics
Use the metrics you’re already alerting on
Many of the graphs you find in the AWS console can be easily recreated in Grafana. Multiple metrics can be combined into a single chart.

Your DB as a Data Source

Connect to Postgres, MySQL, or other DB
Design your own queries around tabular or time series data
● Lots of flexibility
● Write your own SQL with template variables to build
visualizations right on top of your data

Other Interesting Data Sources

Grafana supports many additional data sources. Here’s a few more to try:
● Tracking MRs in GitLab
● Collect your own metrics
● Additional data source plugins can be added to Grafana. Listed here:
https://grafana.com/grafana/plugins?type=datasource
● Additional visualization panel types are also available. Listed here:
https://grafana.com/grafana/plugins?type=panel

Resources

https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/
a. Monitoring Distributed Systems
b. https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/#xref_monitoring_golden-signals
i. Four Golden Signals
https://www.sisense.com/blog/4-design-principles-creating-better-dashboards/
a. 5 second rule
b. tell a clear story
i. express the meaning of the data
c. anti-patterns
i. too many widgets, basic questions take too long to answer, no organizing principle
https://techbeacon.com/devops/10-monitoring-talks-every-developer-should-watch
a. "10 monitoring talks that every developer should watch"
b. https://youtu.be/czes-oa0yik
i. metrics, metrics everywhere
1. importance of metrics, improve mental model of software using data by measuring
2. 2nd half of talk is more focused on how to use metrics better
c. https://youtu.be/zLjhFrUhqxg
i. What should I monitor, and how should I do it?
https://grafana.com/blog/2019/06/10/how-not-to-fail-at-visualization/
a. How Not to Fail at Visualization
i. Points out lots of anti-patterns
https://grafana.com/blog/2019/05/16/worth-a-look-public-grafana-dashboards/