Overview

I tried introducing a continuous profiling tool called Pyroscope.

For more about continuous profiling, refer to this article: What is continuous profiling?

Apparently, Grafana acquired it earlier this year.

Grafana Labs Acquires Pyroscope to Add Code Profiling Capability

Since the acquisition, it seems the official name is Grafana Pyroscope?

There is a plugin available for Grafana, so integration is possible, but Pyroscope also has its own UI.

A Demo is available, which might make it easier to understand what you can see.

The code is open-source, so if you're interested in the implementation, you can check it out.

grafana/pyroscope

Introduction

It's good to first look over the configuration.

Pyroscope Agent

1. Installing Pyroscope Server

An image is pushed to DockerHub, so you can use it.

pyroscope/pyroscope

Here's the Docker installation guide.

Docker Guide

There's also a Kubernetes installation guide.

Kubernetes/Helm

2. Enabling Profiling on the Application Side

Set up profiling and install the agent on the application side.

The basic approach is push-based, but for Go, there's a pull-based option, allowing target management on the Pyroscope server.

Go Pull Mode

When implementing in pull mode with Go, you can manage targets like this:

---
scrape-configs:
  - job-name: pyroscope
    scrape-interval: 60s
    enabled-profiles: [cpu, mem, goroutines, mutex, block]
    static-configs:
      - application: foo
        spy-name: gospy
        targets:
          - foo:80
      - application: bar
        spy-name: gospy
        targets:
          - bar:81

Although not set here, it's advisable to configure data retention periods. By default, data is retained indefinitely. cf. Data retention

Sample Code

Various examples are available in examples.

Here's an example implemented in an application I manage:

gobel-example

Challenges

Initial Password Authentication Setup

Pyroscope offers API KEY, OAuth2, and password authentication mechanisms.

I misunderstood the documentation when setting up initial authentication information with password authentication, which caused some confusion.

To set initial authentication information, include it in the configuration file like this:

auth:
  internal:
    admin:
      name: USERNAME
      password: PASSWORD
    enabled: true

If you read carefully, it's mentioned in configuring-built-in-admin-user, but I mistakenly thought it could only be changed via CLI, which led to unnecessary confusion...

pprof

This is not about Pyroscope, but about Go applications. I encountered issues with pprof settings.

I wrote an article about it, so refer to Using pprof with something other than DefaultServeMux.

Impressions

I've been searching for a profiling tool available as OSS, and Pyroscope seems easy to implement with a user-friendly UI. It's also great that it supports pull mode for Go.