Possibility to get metrics in code

[darkjh]

Hi,

I'm wondering with this library or spark's metric system itself, is it possible to get the metrics directly in code?
For example I've an spark application:

val sc = new SparkContext(conf)
myJob(sc, params, ...)

val myGauge = UserMetricSystem.getGauge("myGauge")
println(myGauge.value)

[akakitani]

Hi @darkjh, there's currently no way to get metrics from individual instances of metrics like this, but it should be possible to get the aggregated metrics using Spark's API. It's a bit convoluted at the moment though, so I'll make a note of this to see if I can add a cleaner API for this.

Here's a quick example which should work, I haven't ran it myself though so let me know if there are any issues:

import collection.JavaConverters._

val sc = SparkContext.getOrCreate()
val metricNamespace = "myMetricNamespace"
UserMetricsSystem.initialize(sc, metricNamespace)

UserMetricSystem.getGauge("myGauge").set(1)

// Obtained from here: https://github.com/groupon/spark-metrics/blob/spark-metrics-1.0/src/main/scala/org/apache/spark/groupon/metrics/MetricsReceiver.scala#L172
val sourceName = s"${sc.appName}.$metricNamespace"
val gauges = for {
  source <- sc.metricsSystem.getSourcesByName(sourceName)
  (gaugeName, gauge) <- source.metricRegistry.getGauges.asScala
} yield {
  gaugeName -> gauge
}

val myGaugeValue = gauges.toMap.get("myGauge").map(gauge => gauge.getValue)

myGaugeValue is going to be an Option[Any] here as we lose the type information, so you'll have to cast it to the appropriate value yourself.

One caveat with this though is that calling getSourcesByName needs to happen after myGauge has a value. This won't work if getSourcesByName is called before the metric is created, since it's yet not registered with Spark's internal metric registry.

I'll keep this use case in mind for a future release, thanks for bringing it up!

[bernhardschaefer]

I used your approach and it works perfectly.
One minor thing: In Spark 2.x metricsSystem was removed from SparkContext (see SPARK-13413). It is still accessible using sc.env.metricsSystem though.