Scalable Cloud-based CI/CD HDL Verification Environment

Verification is the cornerstone of digital design, ensuring high reliability and functional correctness of FPGA and SoC designs. By integrating Azure’s scalable cloud computing, the open-source unit testing capabilities of VUnit, and the high-performance simulation engine of Riviera-PRO, developers can revolutionize simulation workflows and environments. This powerful trio provides a modern, efficient, and collaborative approach to unit testing, tailored to meet the demands of today’s complex projects.

In this blog we will talk about the top benefits of connecting these tools together. For a better understanding of this topic, we will explore the details of the demonstration. Firstly, let’s examine what we are dealing with.

 

Understanding the Tools

Azure: Scalable Cloud Computing

The Azure cloud platform has industry-leading data capabilities and is designed to help developers to bring new solutions to life, empowering cloud, hybrid and edge environments. Build, run, test and manage applications with the tools and framework of your choice.

Key Benefits of Azure

• Global Scalability: Azure’s vast network of data centers supports workloads of all sizes.
• Enhanced Security: Multilayered protections ensure compliance and data integrity.
• Cost Efficiency: Pay-as-you-go pricing and autoscaling reduce operational costs.
• Hybrid Seamless Operate: Integration and management environments with services designed for hybrid cloud.
• Flexibility: Azure supports a variety of languages, tools, and frameworks.
• Revision Control: Azure provides source revision control.

 

VUnit: Open-Source Verification Framework

VUnit is an open-source unit testing framework for VHDL and SystemVerilog. It simplifies HDL verification with automation and continuous integration (CI) capabilities, enabling a “test early and often” approach.

 

Benefits of VUnit

• Cost-Effective: Free to use, making it accessible for teams and projects of all sizes.
• Community-Driven: Global contributors enhance functionality with regular updates.
• Seamless Integration: Works well with CI/CD pipelines and simulation tools like Riviera-PRO.
• Extensive Features: Includes parallel test execution, assertion libraries, logging frameworks, and customizable workflows.

For more details you can refer to the blog about VUnit on the Aldec website.

 

Riviera-PRO: Advanced Simulation and Debugging

Aldec’s Riviera-PRO enhances VUnit’s capabilities with:

• High-Performance Simulation: Mixed VHDL/SystemVerilog/SystemC
• Advanced Debugging: Intuitive GUI, waveform viewer/analysis and macro scripting.
• Incremental Compilation: Saves time by recompiling only updated files.
• Scalable Execution: Runs VUnit testcases across multiple instances in parallel.
• Coverage: Enables HDL structural and functional coverage with intuitive coverage results viewers.

For more information about Riviera-PRO, please visit the website.

 

Connecting the Dots: Azure, VUnit, and Riviera-PRO

Why Combine These Tools?

Combining Azure’s cloud computing resources, VUnit’s testing capabilities, and Riviera-PRO’s advanced simulation features enable the following:

• Scalability: Utilizing Azure to execute VUnit testbenches across multiple compute instances, achieving quicker results without manual intervention.
• Automation: VUnit automates test execution. YAML-based configuration to manage tests and artifacts.
• Accuracy: Riviera-PRO provides advanced debugging and simulation capabilities.
• Faster Results: without increasing costs, as the cost of 20 compute instances running for 1 day is the same as the cost of 1 local machine running for 20 days.

 

Example Project: A Demonstration of Azure, VUnit, and Riviera-PRO in Action

This practical demonstration shows how Azure, VUnit, and Riviera-PRO enhance verification workflows by streamlining test execution, improving coverage analysis, and utilizing scalable cloud-based workflows. For better understanding the topic, the project flow has been depicted in Figure 1.

 

Figure 1. Project Flow Scheme

 

This flowchart summarizes the integration process of Azure DevOps, VUnit, and Riviera-PRO for automated testing and coverage analysis.

1. Azure DevOps Repository: Contains source files, testbench files, scripts (e.g., run.py), YAML file responsible for configuration, and pipelines descriptions.
2. Pipeline Execution: Initiates the workflow based on the YAML configuration.
3. Testing and Coverage Analysis: The run.py script runs VUnit tests, while the acdb.do merges acdb files and generates coverage report.
4. Conversion Process: The coverage report is imported and converted to Cobertura format.
5. Outputs: VUnit tests results and coverage report are generated.

 

Setting Up the Environment

The project files for this demo are hosted on Aldec’s GitHub repository. These files are based on Aldec’s VUnit blog series, with enhancements to showcase additional functionalities. A key update is the declaration of the CONTRIBUTION environment variable in the run.py script, allowing users to focus testing efforts on a user-defined group of tests.

 

# Ask for contribution tests or all tests
CONTRIBUTION='0'
if "CONTRIBUTION" in os.environ:
CONTRIBUTION=os.environ["CONTRIBUTION"]
if CONTRIBUTION=='1':
print("Following contribution tests will be conducted:")

In Azure, environment variables can be configured in a YAML file (explained later). In this example, the environment variable "CONTRIBUTION" allows developers to conduct only specific tests. There is a possibility to set up different environment variables for each job in pipeline run. While executing the pipeline via the GUI, you can assign the desired value to the environment variable for each job in the pipeline.

 

Figure 2. Selecting the Value of the Environment Variable While Running the Pipeline

 

Setting the Virtual Machine (VM) with Azure Pipelines Agent and Uploading Docker Image

In order to run the demo, it’s obligatory to set up a Docker Image for Azure. This process is similar to the preparing Docker Container with Riviera-PRO, but also involves installing Python, VUnit_HDL, and Cobertura add-ons. Once Docker Image is built, upload it to the Azure Container Registry.

 

After that, set up the VM with the Azure Pipelines Agent. You have the option to set up the agent either manually or automatically. Once completed, the setup should look similar to the one shown in Figure 3.

 

Figure 3. View After Setting Up the VM with the Agent and Upload of the Docker Image

 

Uploading Files to Azure DevOps

To upload files to the Azure DevOps repository, first ensure that the Container Registry and VM with the installed agent are prepared. You can either import the existing GitHub repository with files or create a new one in Azure DevOps. Once the files are uploaded, your project should resemble the layout depicted in Figure 4.

 

Figure 4. Files Uploaded into Azure DevOps Repository

 

Configuring Azure Pipelines with YAML

Azure Pipelines combines continuous integration (CI) and continuous delivery (CD) to test, build and deliver code to any destination. Azure DevOps pipelines support various types of triggers, such as code commits, pull requests, or scheduled triggers.

In order to manage pipelines, you will require a configuration YAML file. An example YAML file can be found in the project GitHub repository, with a snippet below containing a sample YAML file.

 

trigger:
- master

parameters:
- name: CONTRIBUTION
displayName: "Run only contribution tests"
type: string
default: 0
values:
- 0
- 1
- name: VUNIT_P
displayName: "VUNIT parallel jobs"
type: string
default: 1
- name: tests_jobs
type: object
default:
- job_name: 'run_tests_1'
tests: 'lib.tb_enc_generics.*'
- job_name: 'run_tests_2'
tests: 'lib.tb_enc_hardcoded.*'

jobs:
- ${{ each tests_job in parameters.tests_jobs }} :
- template: templates/run_tests_jobs.yml
parameters:
job_name: ${{ tests_job.job_name }}
tests: ${{ tests_job.tests }}
contribution: ${{ parameters.CONTRIBUTION }}
vunit_p: ${{ parameters.VUNIT_P }}

- job: publish_coverage
pool:
name: Default

container:
image: aldecdemo.azurecr.io/aldec/riviera-pro:vunit
endpoint: aldec-demo-acr-connection

dependsOn:
- ${{ each tests_job in parameters.tests_jobs }} :
- ${{ tests_job.job_name }}

steps:
- ${{ each tests_job in parameters.tests_jobs }} :
- task: DownloadPipelineArtifact@2
displayName: 'Download artifact ${{ tests_job.job_name }}_acdb'
inputs:
buildType: current
artifactName: ${{ tests_job.job_name }}_acdb
targetPath: '$(Build.SourcesDirectory)'

- script: vsim -c -do acdb.do
displayName: 'Merge acdb and generate coverage report'
workingDirectory: 'vunit/aes-encryption'

- script: |
acdb2xml -i acdb/results.acdb -o acdb/results.xml
python3 ucdb2cobertura.py --units -i acdb/results.xml -o acdb/cobertura.xml
displayName: 'Convert coverage to cobertura format'
workingDirectory: 'vunit/aes-encryption'

- task: PublishPipelineArtifact@1
inputs:
targetPath: 'vunit/aes-encryption/acdb'
artifactName: Coverage

- task: UseDotNet@2
displayName: 'Use .NET Core sdk 7.0.x'
inputs:
version: 7.0.x

- task: PublishCodeCoverageResults@1
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: 'vunit/aes-encryption/acdb/cobertura.xml'
pathToSources: 'vunit/aes-encryption/vunit_out/test_output'

 

For explanatory purposes, here are only some key functionalities highlighted. Full specification can be found here. Predefined azure-pipelines.yml used in this demo project is composed of following sections:

• trigger - Specifies the branches that trigger the pipeline to run on commit/push.
• parameters - this section allows to create list of variables that can be used to parameterize pipeline execution. In our case there is only one parameter of string type with two possible values. It will be rendered as combo box in "Pipeline Run" page and selected value will be stored as environment variable.
• jobs - collection of steps run by an agent. In this example 3 jobs have been created. 2 of them are based on the template declared in the run_tests_jobs.yml file and are responsible for running specific VUnit test cases in parallel. The third job named "publish_coverage” downloads artifacts from the other 2 jobs to conduct line coverage and publish coverage results in cobertura format. format.
• The pool keyword specifies which pool to use for a job of the pipeline. A pool specification also holds information about the job's strategy for running.
• container - Includes Docker container settings such as image name and service endpoint.
• dependsOn - specifies dependencies between different stages or jobs
• steps - list of predefined tasks or script commands that will be sequentially executed in OS shell.

Running the Pipeline

After creating a pipeline, run it to execute the demo. The integration between Azure and Riviera-PRO allows for generation of coverage metrics and ranking of test plans, with detailed outcomes presented within Riviera-PRO's graphical user interface or a web browser. Figure 5 illustrates an instance of these outcomes. By utilizing a special acdb.do macro file, random VUnit test names are converted to pre-defined names, enabling users to accurately identify the executed tests.

 

Figure 5. The Results of the Test Plan Ranking and Coverage Metrics from VUnit Tests.

 

The workflow can easily scale up to make use of the cloud's computing power. Azure's infrastructure allows for scalability at two levels. Firstly, scalability is achieved at the job level, enabling jobs execution in parallel. Secondly, scalability is attained through the simultaneous execution of VUnit test cases on multiple threads within a single virtual machine. This process reduces testing time and costs. To enable this feature, you must have multiple Riviera-PRO license seats and jobs set up to run on Azure DevOps for parallel testing. When you access the pipeline that has been run at least once, you will find specific tabs displaying the results. The Tests tab shows the outcomes of the VUnit tests. The improved results achieved through the integration of Azure and VUnit are shown in Figure 6. In this particular case, all tests have passed. You can click on the name of each test to view more details.

 

Figure 6. Results of the VUnit Tests

 

Within the Code Coverage tab, users can access coverage data that was previously exported in Cobertura format. This allows for a standardized and visual display of the results. The functionality is made possible by utilizing Aldec’s Riviera-PRO simulator coverage functions and converting them to the Cobertura format. This capability is illustrated in Figure 7. Users have the option to delve into the specifics of each file's code coverage results. In this specific instance, the detailed results reveal line coverage.

 

Figure 7. Coverage Results Shown in Cobertura Format

 

Achievements in This Example

This example demonstrates how the integration of Azure, VUnit, and Riviera-PRO can drive significant improvements in verification workflows, including:

• Unified Coverage: Comprehensive coverage metrics that ensure robust verification visualized in Aldec’s Cobertura format.
• Test Planning with Ranking: Prioritizing and ranking tests based on their impact on code changes available by Riviera-PRO.
• Fast Feedback with VUnit: Rapid execution and results using VUnit’s automation features.
• Scalable Cloud Workflows: Leveraging Azure for parallelism and cost-effective resource management.

 

Known Constraints

While the benefits are great, it’s essential to acknowledge existing constraints:

• Simulator Licensing: Each Azure job requires a dedicated license for Riviera-PRO.
• Job Limits: Azure enforces a limit of 20 parallel jobs per account.

Conclusion

The integration of Azure, VUnit, and Riviera-PRO represents the future of verification workflows, combining the best of cloud computing, open-source innovation, and advanced simulation. This powerful trio enables modern, automated and scalable verification processes, significantly reducing testing times and optimizing resource management.

 

By adopting this setup, teams can improve the quality of their verification workflows while supporting a collaborative and globally accessible environment. Azure provides unparalleled scalability, while VUnit’s open-source framework encourages innovation, automatization and customization. Riviera-PRO complements these tools with advanced simulation and debugging capabilities, ensuring in-depth and reliable verification process.

 

Together, these technologies transform verification from a resource-intensive bottleneck into an agile, efficient, and streamlined process, empowering developers to deliver high-quality designs faster and more reliably.
Aldec is proud to offer not only cutting-edge software solutions, but also a comprehensive range of hardware options to meet all your needs. Our team provides you with a complete technological solution that is adapted to your specific requirements. Don’t hesitate to contact us today to learn more about how we can help you achieve your goals.

 

References

• Introduction to VUnit
• Project Repository on Aldec’s GitHub
• VUnit Official Website