How to add support for prometheus?

[bjorntj]

When running the console, the following errors appears:

"Prometheus can't be reached, or is not configured. Please check your configuration."

So I am trying to configure the instance with prometheus support but I can not find any examples on how to do this. What I can find, is a mentioning of adding this: metricsSources: [
],
under spec but I am sure what this array should contain?

Also, the IDE is complaining when adding metricSources so not sure if this is supported yet?

[MikeEdgar]

@bjorntj, the metricsSources should work in v0.5.0, together with a metricsSource: <name> specified on the kafkaClusters entry. Below is a template for the metricsSources and how to reference one of them.

If you are in OpenShift with user workload monitoring enabled, you can use the openshift-monitoring type. The embedded type will deploy a Prometheus in the same namespace as the console deployment with configuration to scrape all Strimzi Kafka clusters in the Kube cluster (intended for development/testing scenarios). Otherwise, the standalone type allows you to reference an existing Prometheus and optionally provide credentials. The upcoming 0.6.0 release will also allow for a CA/trusted certificate configuration in case you have self-signed certs protecting your Prometheus endpoint.

metricsSources:
 - name: my-prometheus
   type: <one of 'embedded', 'standalone', or 'openshift-monitoring'>
   url: <url to the Prometheus host, only for 'standalone'>
   authentication:
     # optional, only if needed for 'standalone'
     username: <for the prometheus api>
     password: <for the prometheus api>
kafkaClusters:
  - name: my-kafka
    metricsSource: my-prometheus
    # other kafka configurations

As far as the IDE error/warning you have, I'm not entirely sure about that. Is the IDE using the CRD somehow for validation?

[bjorntj]

And I see inside the config map, there is only a few zookeeper rules.. So I guess that is the reason for missing metrics..

[MikeEdgar]

You may be able to work around that exception for the messages screen adding this to your Console CR. Note it's just a work-around and it will be properly fixed shortly.

spec:
  env:
  - name: JAVA_OPTS_APPEND
    value: "-Djava.io.tmpdir=/deployments/work"

[bjorntj]

Not quite the same exception but almost, after applying the fix:

at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.kafka.common.KafkaException: java.lang.ExceptionInInitializerError: Cannot unpack libzstd-jni-1.5.6-3: Read-only file system
at org.apache.kafka.common.compress.ZstdFactory.wrapForInput(ZstdFactory.java:70)
at org.apache.kafka.common.record.CompressionType$5.wrapForInput(CompressionType.java:155)
at org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:273)
at org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:277)
at org.apache.kafka.common.record.DefaultRecordBatch.streamingIterator(DefaultRecordBatch.java:358)
at org.apache.kafka.clients.consumer.internals.CompletedFetch.nextFetchedRecord(CompletedFetch.java:223)
at org.apache.kafka.clients.consumer.internals.CompletedFetch.fetchRecords(CompletedFetch.java:274)
... 49 more
Caused by: java.lang.ExceptionInInitializerError: Cannot unpack libzstd-jni-1.5.6-3: Read-only file system
at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.base/java.io.File.createTempFile(File.java:2170)
at com.github.luben.zstd.util.Native.load(Native.java:130)
at com.github.luben.zstd.util.Native.load(Native.java:86)
at com.github.luben.zstd.ZstdInputStreamNoFinalizer.<clinit>(ZstdInputStreamNoFinalizer.java:23)
at org.apache.kafka.common.compress.ZstdFactory.wrapForInput(ZstdFactory.java:68)
... 55 more

[MikeEdgar]

This should work instead - I misinterpreted how the JAVA_OPTS_APPEND is used.

spec:
  env:
  - name: JAVA_OPTS
    value: "-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Djava.io.tmpdir=/deployments/work"

[bjorntj]

Yes, that worked.. 🙂