-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add grafana example #290
Add grafana example #290
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,4 +3,5 @@ laika.title = Examples | |
laika.navigationOrder = [ | ||
jaeger-docker | ||
honeycomb | ||
grafana | ||
] |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,300 @@ | ||
# Grafana - All-in-one | ||
|
||
In this example, we are going to use [Prometheus](https://prometheus.io/) and [Jaeger](https://jaegertracing.io/), both displayed inside [Grafana](https://grafana.com/grafana/). | ||
Just like [Jaeger example](../jaeger-docker/README.md), we will need to set up an [Open Telemetry Collector](https://opentelemetry.io/docs/collector/) to gather both metrics and traces. | ||
|
||
### Project setup | ||
|
||
Configure the project using your favorite tool, once again setup is similar to the Jaeger example, only difference being that we export metrics as they will be stored inside Prometheus: | ||
|
||
@:select(build-tool) | ||
|
||
@:choice(sbt) | ||
|
||
Add settings to the `build.sbt`: | ||
```scala | ||
libraryDependencies ++= Seq( | ||
"org.typelevel" %% "otel4s-java" % "@VERSION@", // <1> | ||
"io.opentelemetry" % "opentelemetry-exporter-otlp" % "@OPEN_TELEMETRY_VERSION@" % Runtime, // <2> | ||
"io.opentelemetry" % "opentelemetry-sdk-extension-autoconfigure" % "@OPEN_TELEMETRY_VERSION@" % Runtime // <3> | ||
) | ||
run / fork := true | ||
javaOptions += "-Dotel.java.global-autoconfigure.enabled=true" // <4> | ||
javaOptions += "-Dotel.service.name=grafana-example" // <5> | ||
javaOptions += "-Dotel.exporter.otlp.endpoint=http://localhost:4317" // <6> | ||
``` | ||
|
||
@:choice(scala-cli) | ||
|
||
Add directives to the `grafana.scala`: | ||
```scala | ||
//> using scala 3.3.0 | ||
//> using lib "org.typelevel::otel4s-java:@VERSION@" // <1> | ||
//> using lib "io.opentelemetry:opentelemetry-exporter-otlp:@OPEN_TELEMETRY_VERSION@" // <2> | ||
//> using lib "io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:@OPEN_TELEMETRY_VERSION@" // <3> | ||
//> using `java-opt` "-Dotel.java.global-autoconfigure.enabled=true" // <4> | ||
//> using `java-opt` "-Dotel.service.name=grafana-example" // <5> | ||
//> using `java-opt` "-Dotel.exporter.otlp.endpoint=http://localhost:4317" // <6> | ||
``` | ||
|
||
@:@ | ||
|
||
1) Add the `otel4s` library | ||
2) Add an OpenTelemetry exporter. Without the exporter, the application will crash | ||
3) Add an OpenTelemetry autoconfigure extension | ||
4) Enable OpenTelemetry SDK autoconfigure mode | ||
5) Add the name of the application to use in the traces | ||
6) Add the OpenTelemetry Collector endpoint | ||
|
||
|
||
### OpenTelemetry SDK configuration | ||
|
||
As mentioned above, we use `otel.service.name` and `otel.metrics.exporter` system properties to configure the | ||
OpenTelemetry SDK. | ||
The SDK can be configured via environment variables too. Check the full list | ||
of [environment variable configurations](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md) | ||
for more options. | ||
|
||
### Observability stack configuration | ||
|
||
Use the following `docker-compose.yaml` file: | ||
|
||
```yaml | ||
version: '3.7' | ||
services: | ||
otel-collector: # receives application metrics and traces via gRPC or HTTP protocol | ||
image: otel/opentelemetry-collector-contrib | ||
command: [--config=/etc/otel-collector-config.yaml] | ||
volumes: | ||
- "./config/otel-collector-config.yaml:/etc/otel-collector-config.yaml" | ||
ports: | ||
- "8888:8888" # Prometheus metrics exposed by the collector | ||
- "8889:8889" # Prometheus exporter metrics | ||
- "4317:4317" # OTLP gRPC receiver | ||
- "4318:4318" # OTLP http receiver | ||
networks: | ||
- static-network | ||
|
||
jaeger: # stores traces received from the OpenTelemetry Collector | ||
image: jaegertracing/all-in-one:latest | ||
volumes: | ||
- "./config/jaeger-ui.json:/etc/jaeger/jaeger-ui.json" | ||
command: --query.ui-config /etc/jaeger/jaeger-ui.json | ||
environment: | ||
- METRICS_STORAGE_TYPE=prometheus | ||
- PROMETHEUS_SERVER_URL=http://prometheus:9090 | ||
ports: | ||
- "14250:14250" | ||
- "16685:16685" # GRPC | ||
- "16686:16686" # UI | ||
networks: | ||
- static-network | ||
|
||
prometheus: # scrapes metrics from the OpenTelemetry Collector | ||
image: prom/prometheus:latest | ||
volumes: | ||
- "./config/prometheus.yml:/etc/prometheus/prometheus.yml" | ||
ports: | ||
- "9090:9090" | ||
networks: | ||
- static-network | ||
|
||
grafana: # queries Jaeger and Prometheus to visualize traces and metrics | ||
image: grafana/grafana-oss | ||
restart: unless-stopped | ||
ports: | ||
- "3000:3000" | ||
networks: | ||
- static-network | ||
|
||
networks: | ||
static-network: | ||
``` | ||
|
||
### Configuration files | ||
|
||
#### ./config/otel-collector-config.yaml | ||
|
||
OpenTelemetry Collector configuration: receivers, exporters, and processing pipelines. | ||
|
||
```yml | ||
receivers: | ||
otlp: | ||
protocols: # enable OpenTelemetry Protocol receiver, both gRPC and HTTP | ||
grpc: | ||
http: | ||
|
||
exporters: | ||
jaeger: # export received traces to Jaeger | ||
endpoint: jaeger:14250 | ||
tls: | ||
insecure: true | ||
|
||
prometheus: # run Prometheus exporter server on port 8889, so Prometheus can scrape the metrics | ||
endpoint: 0.0.0.0:8889 | ||
send_timestamps: true | ||
|
||
processors: | ||
batch: | ||
timeout: 10s | ||
|
||
service: | ||
pipelines: | ||
traces: | ||
receivers: [otlp] | ||
processors: [batch] | ||
exporters: [jaeger] | ||
metrics: | ||
receivers: [otlp] | ||
processors: [batch] | ||
exporters: [prometheus] | ||
``` | ||
|
||
#### ./config/prometheus.yml | ||
|
||
Prometheus server configuration: scrape interval and targets. | ||
|
||
```yml | ||
global: | ||
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. | ||
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. | ||
|
||
scrape_configs: | ||
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. | ||
- job_name: "prometheus" # scrape prometheus itself to collect the internal metrics (e.g. scrape stats, etc) | ||
static_configs: | ||
- targets: ["localhost:9090"] | ||
|
||
- job_name: "otel-collector" # scrape metrics from the OpenTelemetry collector | ||
static_configs: | ||
- targets: ["otel-collector:8889"] | ||
``` | ||
|
||
#### ./config/jaeger-ui.json | ||
|
||
Jaeger configuration: enable [Service Performance Monitor (SPM)](https://www.jaegertracing.io/docs/1.48/spm/). | ||
|
||
```json | ||
{ | ||
"monitor": { | ||
"menuEnabled": true | ||
}, | ||
"dependencies": { | ||
"menuEnabled": true | ||
} | ||
} | ||
``` | ||
|
||
### Application example | ||
|
||
Example service mocking a call to a remote API: here the remote API returns apples or bananas. | ||
We're using the metrics to measure the apple/banana ratio returned by the API, | ||
and the traces to measure the latency of this API. | ||
|
||
```scala mdoc:silent | ||
import cats.effect.{Async, IO, IOApp} | ||
import cats.effect.std.Random | ||
import cats.syntax.apply._ | ||
import cats.syntax.flatMap._ | ||
import cats.syntax.functor._ | ||
import org.typelevel.otel4s.Attribute | ||
import org.typelevel.otel4s.java.OtelJava | ||
import org.typelevel.otel4s.metrics.Meter | ||
import org.typelevel.otel4s.trace.Tracer | ||
|
||
import java.util.concurrent.TimeUnit | ||
import scala.concurrent.duration.FiniteDuration | ||
|
||
case class ApiData(result: String) | ||
|
||
trait ApiService[F[_]] { | ||
def getDataFromSomeAPI: F[ApiData] | ||
} | ||
|
||
object ApiService { | ||
def apply[F[_]: Async: Tracer: Meter: Random]( | ||
minLatency: Int, | ||
maxLatency: Int, | ||
bananaPercentage: Int | ||
): F[ApiService[F]] = | ||
Meter[F] | ||
.counter("RemoteApi.fruit.count") | ||
.withDescription("Number of fruits returned by the API.") | ||
.create | ||
.map { remoteApiFruitCount => | ||
new ApiService[F] { | ||
override def getDataFromSomeAPI: F[ApiData] = for { | ||
latency <- Random[F].betweenInt(minLatency, maxLatency) | ||
isBanana <- Random[F].betweenInt(0, 100).map(_ <= bananaPercentage) | ||
duration = FiniteDuration(latency, TimeUnit.MILLISECONDS) | ||
fruit <- Tracer[F].span("remoteAPI.com/fruit").surround( | ||
Async[F].sleep(duration) *> | ||
Async[F].pure(if (isBanana) "banana" else "apple") | ||
) | ||
_ <- remoteApiFruitCount.inc(Attribute("fruit", fruit)) | ||
} yield ApiData(s"Api returned a $fruit !") | ||
} | ||
} | ||
} | ||
|
||
object ExampleService extends IOApp.Simple { | ||
def run: IO[Unit] = | ||
OtelJava.global.flatMap { otel4s => | ||
( | ||
otel4s.tracerProvider.get("com.service.runtime"), | ||
otel4s.meterProvider.get("com.service.runtime"), | ||
Random.scalaUtilRandom[IO] | ||
).flatMapN { case components => | ||
implicit val (tracer: Tracer[IO], meter: Meter[IO], random: Random[IO]) = | ||
components | ||
|
||
for { | ||
service <- ApiService[IO]( | ||
minLatency = 40, | ||
maxLatency = 80, | ||
bananaPercentage = 70 | ||
) | ||
data <- service.getDataFromSomeAPI | ||
_ <- IO.println(s"Service data: $data") | ||
} yield () | ||
} | ||
} | ||
} | ||
``` | ||
|
||
### Run the application | ||
|
||
@:select(build-tool) | ||
|
||
@:choice(sbt) | ||
|
||
```shell | ||
$ sbt run | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't see an entry point in the example above, so will There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just tried it on my example project, works fine 👌 |
||
``` | ||
|
||
@:choice(scala-cli) | ||
|
||
```shell | ||
$ scala-cli run grafana.scala | ||
``` | ||
|
||
@:@ | ||
|
||
### Setup your metrics dashboard | ||
|
||
Connect to grafana and Add your Jaeger and Grafana data sources. Once done, you can create your [first dashboard](https://github.com/typelevel/otel4s/tree/main/docs/examples/grafana/dashboards/metrics-dashboard.json). | ||
Now after making a few calls to `ApiService.getDataFromSomeAPI`, you should get some data points: | ||
|
||
@:image(metrics-grafana.png) { | ||
alt = Grafana Metrics Example | ||
} | ||
|
||
### Adding your traces | ||
|
||
Just like you just did with Prometheus, you can use Jaeger as data source and display traces using the appropriate data visualization in your [new dashboard](https://github.com/typelevel/otel4s/tree/main/docs/examples/grafana/dashboards/traces-dashboard.json): | ||
|
||
@:image(traces-grafana.png) { | ||
alt = Grafana Traces Example | ||
} | ||
|
||
> Note: a complete example is available [here](https://github.com/keuhdall/otel4s-grafana-example/) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That way we can ensure users will have a correct Scala version and the example will compile