VMware: Streamline Open Source Software Discovery with Bazel

0

Description of the icon if necessary8 minute read

There are hundreds of thousands of open source software projects to choose from today, making open source software (OSS) an essential component of nearly every codebase. Regardless of the OSS used, product releases must comply with the legal and licensing requirements of all open source software. It’s true for VMware and it’s true for anyone who uses open source components in their products.

To be compliant, you need to know which OSS you included in your build. But compiling this exhaustive list of components can add time and friction to the development cycle and can delay product releases. The ideal scenario is an effective process for identifying and tracking open source software so that your company’s legal or open source team can further investigate and verify license compliance. .

Building in Bazel

At VMware, we’ve moved some of our larger builds to Bazel, a build system with precise knowledge of all build entries. Bazel includes many additional benefits such as fast and deterministic builds with multilingual support. The story of how we chose Bazel and the benefits it brought us would fill an entire blog, or two. In this blog, we take a look at how our team developed a new way to integrate the existing OSS compliance process with Bazel.

The new developer workflow to meet OSS compliance

As part of the move to Bazel, we set out to improve the efficiency of OSS license identification, tracking, and compliance. By creating and applying a new Bazel to reignduring the build process, we can accurately identify direct and transitive OSS dependencies. We combine information about OSS dependencies from Bazel with information about licensing and usage from VMware’s internal OSS Review and Compliance Tool. This benefits our developer workflow to meet OSS compliance in several ways and has been adopted by a growing subset of VMware product releases.

First, developers get a deterministic BOM with each release. The BOM contains all direct and transitive dependencies of a product associated with license information and approval status of our OSS tool. VMware’s current compliance process OSS scanners verify the accuracy and completeness of the BOM generated by Bazel. In case the BOM is incomplete, it can be supplemented by construction steps that analyze the individual entries.

Second, developers are notified at build time if their product contains OSS dependencies that have been denied use by legal or security teams, a process that previously could take days. This is done by checking each OSS dependency against its approval status in our OSS tool.

Third, Bazel’s multilingual support allows us to have a tool that works across multiple platforms. oss_auditsupports OSS identification in Java projects built by Bazel. We are currently adding support for C++ versions and will soon support other languages.

Because we create a BOM and validate OSS approval status with each release, OSS decisions are made earlier in the development and review process, making it less costly and less likely to impose publication delays due to required corrective actions.

Let’s look at an example of this new developer workflow using the oss_auditto reign.

Use oss_audit

under the hood, oss_auditconsumes a Bazel aspectwhich analyzes a build’s dependency graph and collects licensing information about each package it finds. Additionally, oss_audituses a list of approved and denied OSS packages from our OSS tool to alert developers of the case when denied packages are used and need to be removed.

Figure 1: Applying oss_audit to an example project packaged in an RPM.

The Bazel rule generates two files. First, a BOM yaml file, which includes information about each OSS dependency. Second, a BOM issues file, containing a subset of OSS dependencies that have been denied use or are still awaiting approval. Release managers and developers can use the BOM-issues file to identify issues that could block or delay a product release.

To track OSS for license compliance, an automated task uses the BOM to file new OSS packages with the OSS tool for legal and security review. Updated data from the review process is then fed back into source control as approved and denied lists for use by the next build.

Example code

Here is an example code snippet showing how to apply oss_auditto a project.

Figure 2: Code for applying oss_audit to an RPM.

oss_auditcan be added to a BUILD file to audit a target, such as an RPM.In the example above, the code uses the pkg_rpmrule, the creation of the RPM and the oss_auditruler, RPM check.You can use oss_auditin your own project fromoss_audit_rulesrepository on GitHub (github.com/vmware/rules_oss_audit).

Parts lists

At VMware, there are many complex products made up of several sub-components. For such products, oss_auditcan be used to generate a BOM for each subcomponent, and the subcomponent BOMs are then merged to create a higher level cumulative BOM and BOM issue files for the entire product. This gives product and component teams insight into what they are shipping at varying degrees of granularity.

Conclusion

OSS is an essential part of almost every code base. With Bazel as a base, we are able to identify all OSS used directly and transitively in a build. With this information, we can create a Bill of Materials and cross-reference all OSS with their approval status in our OSS Review and Compliance tool at build time. We use this Bazel-based tool in a number of projects at VMware and work closely with our colleagues in the Open Source Program Office and legal teams to ensure complete, documented, and validated license compliance. Although we have discovered that Bazel can be used to populate your data with open source components, it is essential that your company’s internal tools and processes verify license compliance and create the required open source license document.

Please visit our oss_audit_rulesrepository on GitHub (github.com/vmware/rules_oss_audit), check out the code and open an issue to continue this discussion!

Stay tuned to Open Source Blogand follow us on Twitterfor deeper dives into the world of open source contributing.

Share.

Comments are closed.