Shift security left

Shift security left

Special thanks to Dan Wallach for ideas and review of this page.

Note

The image for this page is from economics, and shows the impact of shifting concerns "to the left".

Security as a first-class concern

If you look at the badges for the project code, you may note that "Snyk security" comes immediately after build status. This is intentional. But this badge is only one way to "shift security left". This example project should show you other ways to improve security, and make security a first-class concern for your project.

After the CI build successfully passing, there is probably no other check more important in terms of reliability than security checks. These checks validate as best you can that no unexpected or malicious processes are affecting your build or software. You want your build to fail if you detect potential security issues.

There are many ways problems arise and vulnerabilities surface in your project. To list only some:

• Security bugs in library dependencies may hide after updating. CVEs and other publicly reported flaws are something you can catch by keeping dependencies current.
• Corner cases for functional code often include security angles as well, such as a malicious user trying "unexpected" values.

We recommend that your project always addresses concerns about security and restricted information, and treat security a first-class concern. Breaches of security and private information should be communicated to stakeholders (for open source projects the public after a suitable delay for problems to be fixed). This grows confidence in your software.

Aside: an example of going wrong

The well-known XKCD cartoon about SQL injection perfectly illustrates why your build should catch as many security and privacy concerns as early as possible before they could happen in released software.

Leverage your build container for security

So what happens when a dependency or a build plugin has been hijacked, and wants to be malicious? A key defense is to run containerized builds as your default practice: this limits damage to inside the build container. Locally developers build from the IDE or command-line; but for CI builds only changes pushed to share are run from containers in your CI system (such as GitLab or GitHub).

A theoretical example (there are several real world examples actually) is a dependency that has been tinkered with to probe the build machine (which is your machine when building locally) and send back telemetry to bad actors. Running the build in a container leaves this dependency to only probe the container rather than the host machine.

Leverage static analysis for security

Many top-level security issues can be spotted during your build! This project strongly recommends the Find Security Bugs plugin for Spot Bugs which both fails your build for problems, and produces helpful reports (141 security bug patterns as of Aug 2024). See Use static analysis for more on finding concerns during your build rather than at runtime.

Many of the bug patterns spotted by "find-sec-bugs" are directly related to SQL and command injection issues: don't let "Bobby Tables" happen to you.

Security problems come in many flavors. Conversations with Dan Wallach, a security researcher who kindly provided input for this writing, highlighted these areas—but they will vary for you depending on your project!—:

• Login concerns
• SQL injection
• Webapp flaws

This topic is too large to cover in this page!

Further reading

There are many tools that perform security scans at build. Some are commercial, and some are available for open-source as well. One example available for open-source is Coverity developed for US Homeland Security (this is not an endorsement).

You should do your own research: circumstances are specific to your project and environment, and the best choices differ for different circumstances. For example, this project does not look at web security but focuses on JVM builds.

Use secrets wisely

There are many examples of builds needing secrets such as cloud credentials, passwords, sensitive user information, etc. This project uses the OWASP_NVD_API_KEY as an example to both illustrate safe passing of secrets to local and CI builds, and how a containerized build (Earthly) can help you with this. This is worth your time to study and get right: One example bad outcome from doing it wrong is to save secrets into the Earthly image for your build.

For Earthly and GitHub, some helpful documentation:

• https://docs.earthly.dev/docs/guides/best-practices#use-secret-not-args-to-pass-secrets-to-the-build
• https://docs.earthly.dev/docs/guides/secrets
• https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions

Secrets sample

The sample build scripts for Gradle and Maven assume an OWASP_NVD_API_KEY environment variable — but work without one, just more slowly when updating CVEs for dependency checking.

The sample Earthfile configuration and sample command-line scripts show how to pass secrets from an environment variable into your containerized build, and the sample GitHub workflow files for CI bring the secret from your GitHub project configuration into the build pipeline.

Tip

Make sure you are passing the DependencyCheck key through from the shell environment to your build scripts, including passing through the containerized build. And update your build.gradle (build.gradle.kts) or pom.xml. You will need to update your CI configuration with this key to get the same results as local devs who have a shell export for the key.

The end result is that OWASP_NVD_API_KEY is handled as a secret when:

• Building locally directly (calling ./gradlew or ./mvnw)
• Building locally with a containerized build (Earthly)
• Building in CI with a containerized build (Earthly)

The setup for this example looks like:

1. Update Earthfile so that RUN some-command becomes RUN --secret OWASP_NVD_API_KEY some-command.
2. Update your local environment with the OWASP_NVD_API_KEY environment variable.
3. Update your CI secrets with the OWASP_NVD_API_KEY=value pair.
4. Update your build script (eg, .github/workflows/ci.yml) to include the secret:

   env:
     OWASP_NVD_API_KEY: ${{ secrets.OWASP_NVD_API_KEY }}

   and use it when calling Earthly: earthly --secret OWASP_NVD_API_KEY +your-target.

Important

After this change, you need to invoke Earthly locally just as CI does with the --secret command-line option. The build-as-ci-does.sh helper script in this project is an example.

Note

An alternative to a dedicated build script is a shell alias such as alias earthly='earthly --secret=OWASP_NVD_API_KEY.

Private information in images

You may find that you have images in your code repository that provides revealing information about the individuals that created them. An example is GPS coordinates for a photo. The example code repo scanned the code in all previous commits, and the wiki pages, for "PII" (personal identifiable information). Screenshots from a phone or personal devices are especially prone to leaking details on individuals.

There are tools for checking images in a code repository for PII. This project uses ExifTool to check for image tags that may say more than we intend.

Checking dependencies

This is CRITICAL if you have any direct, indirect, or through-plugin dependencies on Log4j. Beyond your project, you may be impacted by services you call, so check with your organization or external services

DependencyCheck is the current best tool for JVM projects to verify that your project does not rely on external code with known security vulnerabilities (CVEs) from the NVD. That said, DependencyCheck does impact build times. It is smart about caching, but will once a day may take time to download data on any new NVD CVEs, and occasionally the site is down for maintenance. You may consider leaving this check in CI-only if you find local build times overly impacted. Leaving these checks to CI-only is a tradeoff between "shifting security left", and speed for local builds. I lean towards security first; however, you know your circumstances best.

Important

This project fails the build if finding any CVEs for the current version of any dependency.

Your build should fail, too. It is a red flag to you to consider the CVE, what impact the vulnerable dependency has, and if you are comfortable with a vulnerable dependency. It is rarely (if ever) the case you keep a vulnerable version of a dependency. However, there are also "false positives": see config/owasp-suppressions.xml for examples of handling these.

The best current tooling with Gradle or Maven uses the DependencyCheck plugin.

NVD API key

Note

As per the Attribution request from NVD: This product uses the NVD API but is not endorsed or certified by the NVD.

DependencyCheck works out of the box. However, for faster build times (especially when local), consider using an API key for pulls of NVD data (CVE security issues).

Requesting a key is simple: 0. Check if your project already has an NVD API key to use. If not:

1. Go to the linked request page.
2. Fill in the fields.
3. Accept the terms.
4. Check for an email to the address you provided.
5. In the email, use the link and save the API key in the resulting web page.
6. Include the key in your build but do not share the key and do not commit it to source.

Repeat for each separate project.

The sample code uses the OWASP_NVD_API_KEY environment variable and Gradle and Maven automatically pick this up. In single projects you can use the key in your ~/.bashrc file (loaded each time you log in):

export OWASP_NVD_API_KEY=<the key you got from NVD>

Note

I had an email conversation with NVD on how best to use the key, and their recommendation was along the lines of:

• Use separate NVD API keys for each public or shared project.
• For personal projects, it is OK to use a single key for yourself across your personal projects.

Gradle

Read more at Verifying dependencies.

Note

When running a Gradle build, you may see the dependencyCheckAnalyze task take a while. This is as the DependencyCheck plugin pulls down updates for CVEs, the plugin does not notify you of progress. (Contrast with the equivalent Maven plugin which does notify you of progress.)

Maven

Always run with the --strict-checksums (or -C) flag. See Maven Artifact Checksums - What? for more information. This is easy to forget about at the local command line. The .mvn/maven.config file helps this be automatic, and can be checked into your project repository.

An alternative is to declare each repository in your user settings.xml and set the checksum policy to "fail".

(Earthly and GitHub Actions are discussed both above.)

Automate scanning for secrets

One key to shifting security left is avoiding secrets (passwords, private identifiers, etc.) in your source code, commit history, and so on.

GitHub and other repository services offer secrets scanning out of the box: Secret scanning alerts are now available (and free) for all public repositories.

Notes

DependencyCheck may be your slowest quality check in local builds (competing with mutation testing for that ignominious title). Sometimes it may fail when the upstream source for CVEs is offline. If this is a recurring problem for you, consider moving this check into CI. The downside that local work might use an insecure dependency for a while. Checking daily for updated dependencies can lessen this risk:

• Gradle: ./gradlew dependencyUpdates
• Maven: ./mvnw versions:update-properties

For non-Windows platforms, you may see this warning when DependencyCheck runs:

.NET Assembly Analyzer could not be initialized and at least one 'exe' or 'dll' was scanned. The 'dotnet' executable could not be found on the path; either disable the Assembly Analyzer or add the path to dotnet core in the configuration.

In most situations, you are running in a Linux-based Docker container, or using local Linux or Mac command line. In a native Windows project, this is an issue to address, and may be a serious security concern indicating you are missing critical Windows components or updates. For other platforms, this is a nuisance message.

On Gradle when updating to version 7.x.x of DependencyCheck from 6.x.x or earlier, first run ./gradlew dependencyCheckPurge to clear out the local cache schema of CVEs. DependencyCheck moved to schema v2 in 7.x.x from v1 in 6.x.x and earlier, and the 7.0.0 Gradle plugin fails with the older schema version.

Verifying your project to others

Use checksums and signatures: verify what your build and project downloads! When publishing for consumption by others, provide MD5 (checksum) files in your upload: be a good netizen, and help others trust code downloaded from you.

Dependabot

GitHub provides Dependabot (other systems than GitHub may have similar robot tools) which, among other things, can automatically issue PRs to your repository when security issues are discovered. This project uses Dependabot for Gradle and Maven.

NB — Dependabot is more reliable than either the Gradle or Maven plugins for dependencies.

It is tempting to hard-code the API key into your Gradle or Maven build scripts: avoid this; better is to put it into your CI environment configuration, and help local developers in their setups. (This is a mistake I've made, and now have to figure out how to get the API key out of my git history.)

• To open the report for DependencyCheck, build locally and use the <project root>/build/reports/ (Gradle) or <project root/target/ (Maven) path. The path shown in a Docker build is relative to the interior of the container
• Sometimes you may want to refresh your local cache of the NVD files (DependencyCheck may suggest this):
  • Gradle: ./gradlew dependencyCheckPurge depenedencyCheckUpdate
  • Maven: ./mvnw dependency-check:purge dependency-check:update-only
• See Smart configuration for more discussion on configuring your API key.
• See the "Tips" section of Gradle or Maven.
• With GitHub actions, consider adding a tool such as Dependabot, which automatically files GitHub issues for known dependency vulnerabilities. See earlier in this document for an example.
• You can temporarily disable OWASP dependency checking via -Dowasp.skip=true for either Gradle or Maven, for example if the OWASP site is down for maintenance, and you cannot update the local CVE cache.
• The log4shell security vulnerabilities (CVE-2021-44228, CVE-2021-45046, CVE-2021-45105 are extremely severe. They are so severe, this should be a top priority for you to address regardless of other priorities. Although this project does not use log4j, local testing shows that the DependencyCheck plugin for either Gradle or Maven fails build when you use an older, insecure version of log4j-core indirectly. Note that Gradle 7.3.3+ itself fails your build if it detect a dependency on a vulnerable version of log4j-core.
• The BCEL security vulnerability (CVE-2022-42920) is critical, and should be a top priority to address. In this project, BCEL is used by SpotBugs: you should update to recent versions of SpotBugs that resolve the dependency on BCEL.

When Dependabot fails in a PR

Dependabot for GitHub and others may fail when running builds on PRs offered. If the problem is not on the suggested branch dependency change, you can fix the issue in your main/master branch, and then go to the PR and ask CI Dependabot to pull and retry. You do this with a comment in the PR:

@dependabot rebase

Note that this asks Dependabot to:

1. Update the PR branch with changes from the main branch.
2. Apply the PR changes against the updated main branch.
3. Rerun checks to show if the PR is easily merged.

CodeQL

CodeQL is also provided by GitHub for helping you analyze and search your codebase for vulnerabilities. The way it works is it builds a database of your codebase that can be scanned for potential problems using its own query language. It can easily be setup in github to scan your codebase using query suites. There is a default one, which focuses on finding high risk issues, an extended one which broadens to include ones with lower severity or you can create your own custom suite. These can either be configured to run either on your pipeline or using their CLI tool.

For further reading, you can start here or there is also a built in tutorial which can be found here.

Going further

OK, you have built software that you trust. How do you share that trust with others, so that they also trust your software?

A good overview comes from GitHub: Where does your software (really) come from?.