Monitor and report your build

Monitor and report your build

There are several senses of "how to track" your build:

• (monitoring) I want to see build results locally.
  Your local command line or IDE does that now, but maybe you focus elsewhere for a time, and would like notification. The Maven Notifier project is helpful. Note that it provides a good list of alternatives.
• (monitoring) I want to see my build work in CI.
  You can monitor the web page for your build in CI, but again, you'd like to focus elsewhere.
• (reporting) When you check your build outcome, you'd like relevant information when it fails, and ignore things when it passes.
• (reporting) You'd like build reports that you can share, or use as inputs to other tools.

Logging

Logging is a key topic for monitoring and reporting. The easiest sense of "logging" is did it pass or fail? All tooling supports pass/fail checking: it is simply checking the exit status of Gradle or Maven.

Note

Exit status is simply a numeric code. For containers and command line, with Linux/Mac/Windows, a 0 exit code is success, and any non-zero exit code is a failure.

Typically a failure exit code refers to some kind of message with more information, however this is hit-or-miss as systems and tools disagree on what non-0 codes should indicate. This is akin to the xkcd Traffic Lights joke.

Frontpage badges

One of the easiest means is showing status in your frontpage README.md for your project with badges: short images that highlight statuses. Immediately intuitive is this image:

The screen capture for this example project provides quick visuals for:

• Build status with Gradle — for Gradle projects; provided by GitHub
• Build status with Maven — for Maven projects; provided by GitHub
• CodeQL status — if you choose to use CodeQL in GitHub
• Snyk security status — if you use Snyk security scans
• Code coverage percentile — your coverage; JaCoCo for this project
• Pull request count — provided by GitHub
• Issue count — provided by GitHub
• Licensing — relevant for public projects

You may not want all of these, and may have others you'd like to add. If your project is shared with others, a good badge to show is "Release" with a version: this helps others know if they are up-to-date, and the badge points to how to download or use the latest version. (This example project does not have "releases".)

At a minimum you should show:

• Build status, ie, latest build result (Actions in GitHub, et al for other CI systems)
• Security check status (varying depending on how you monitor and report security)
• Coverage checks such as JaCoCo (if you implement them, but really you should)
• Issue count to help others to share with you feedback and bugs, and check if your project is keeping up to date. (This could be a count from non-CI systems such as JIRA)

Look at this project's README.md for example Markdown for badges.

Monitoring

A useful feature of modern CI systems is chatops, that is, interacting with remote systems over online chat. Some examples (and the landscape of options is huge):

• Notifications for build status in CI
• Creating project issues "on the fly" over chat
• Using chat conversations with others as comments on PRs or issues
• Deploying from CI without going to a web page

There are too many features and combinations of systems to mention them all (the topic is worth a book in itself). You should research what add-ons or plugins are available for your chat system.

An obvious one to try is integration with your devops platform (ADO, GitHub, GitLab, etc) or CI system (Jenkins, etc.). GitHub itself provides chatops features in PRs and Issues, for example, asking Dependabot to rescan your dependencies.

One example is GitHub+Slack integration. You should explore your options based on combination of devops platform (ADO, GitHub, GitLab, et al), and your chat client. This project uses GitHub+Slack to communicate among authors.

Earthly monitoring

Earthly provides good resources for monitoring your build. In addition to expected writing to STDOUT/STDERR of the build for local and CI, it saves build outputs to the cloud. An example from the output of earthly +build-with-maven is (the link shown is an example, and not valid):

========================== 🌍 Earthly Build  ✅ SUCCESS ==========================

View logs at https://cloud.earthly.dev/builds/171f1e57-d5a6-4653-8942-f705e1287373

Note

The example is not quite what you'd run for earthly but was simplified. You would provide --secret OWASP_NVD_API_KEY to speed up OWASP checks for dependency vulnerabilities (CVEs).

Tips

See https://github.com/earthly/earthly/issues/4065 for a discussion on disabling default Earthly monitoring to the cloud.

Reporting

Both Gradle and Maven plugins produce reports in multiple formats for you to peruse and reuse, however they do not notify you on your desktop or mobile device. And there will be a wealth of information in the reports, so you need to scan carefully to find reasons for a build breaking, or for ways to improve the build.

If you are unsure where a report appears, try searching in the build (Gradle) or target (Maven) directories. An example with the command line after a full local build:

$ find ./build -name \*.html  # Gradle
# OR
$ find ./target -name \*.html  # Maven

Exact reporting formats and file extensions in your search depend on how you configured your build.

Generating good reports

All the plugins discussed in other pages provide means to generate reports in various formats. The general pattern the example project follows on configuring plugins asks these questions:

1. Are reports HTML?
   This this the key goal.
2. Are reports available as SARIF or XML (or other formats such as JSON)?
   Add these for flexibility with integration to other tools and dashboards.
3. Does the tool generate a directory?
   Then save the directory during CI build to create a ZIP artifact.
4. Does the tool generate just files?
   Then save the files directly during CI builds as artifacts.

The example project configures plugins to provide maximum report formats, and we save the results for each CI build run as artifacts.

Logging

There are two kinds of logging from the build:

1. Output you want to see
2. Output you do not want to see

That sounds easy, right? Of course not.

The first ouput is the tricky one. When a build is RED, you may want as much output as possible, but only for output around failures; when a build is GREEN, you may only want to see the success and not the details; and even for a green build, if you are working on tuning build performance, you may want to see everything. How is are build scripts to know what you want to see?

Common report formats

SARIF

You may want to save build reports for sharing with other projects or teams, or for aggregated reporting, for example to share with a security team at your company. There has been progress on this in the open source community (OASIS) with support from signicant software companies (Microsoft and others): SARIF specification and documentation. SARIF writes build result files in JSON.

Tooling is catching up to provide their results as SARIF format when projects decide to use this format. It is a good way to provide common result files usable and interchangeable independent of source language (such as JavaScript) and tools.

Current tooling that support SARIF report format includes:

• Checkstyle (Gradle only)
• SpotBugs (Gradle and Maven)

When looking for SARIF files in your build directory, typical file names are *.sarif.

SARIF Examples

Here are sample files from Gradle:

• Checkstyle main
• Checkstyle test
• Checkstyle integration test
• SpotBugs main
• SpotBugs test
• SpotBugs integration test

Note

The Checkstyle SARIF files show no results as there were no style violations.

And from Maven:

• SpotBugs main

There are many sources for reading more, some are:

• SARIF support for code scanning (GitHub documentation)
• What is SARIF and how it could revolutionize software security
• How to Get Nice Error Reports Using SARIF in GitHub

Gradle

Canonically all reports appear somewhere underneath $PROJECT_ROOT/build unless you have configured them for elsewhere.

Maven

Canonically all reports appear somewhere underneath $PROJECT_ROOT/target unless you have configured them for elsewhere.

Tips

TODO: Placeholder section

Going further

There are lots of interesting things you can do with a build monitor, and integrating with monitoring for deployed system. There are many examples: this is a popular area for projects and startups, and from big-name cloud vendors.

Two quick examples are:

• Apache Skywalking
• Grafana

All modern options (not just the above two) support common log and metrics aggregation formats and platforms. Your build is just one kind of metric.