Review Test Report¶
Once you finish your first test run, you may want to review the test reports. Checkbox generates test reports in different formats that can be used to easily share the results of a test session. By default, a text summary and submission files are created. This section will provide you with information on where to locate the report files and their respective contents.
Note
You can tailor desired reports in your launcher file, or define your own exporter to customize the reports. See launcher and exporter for more details.
Text Summary¶
You should see a text summary displayed in the console once all jobs are
completed. The result of each job is presented in the format: outcome:
summary
, and it looks like:
☐ : Resource for NVDIMM detection
☑ : Display USB devices attached to SUT
☒ : Test USB 2.0 or 1.1 ports
☑ : Collect information about supported types of USB
☒ : Test USB 3.0 or 3.1 ports
Checkout this table to map outcome symbols to their corresponding descriptions:
symbol |
outcome |
---|---|
job didn’t run |
|
☑ |
job passed |
☒ |
job failed |
☐ |
job skipped, job cannot be started |
‒ |
job is not implemented |
⁇ |
job needs verification |
⚠ |
job crashed |
As text summary only provides an overview, for more in-depth information, you will need to review submission files.
Submission Files¶
In Checkbox, submission files encompass report files in various formats (HTML, JSON, and JUnit XML) along with attachments like I/O logs and binary files. These files serve the purpose of sharing test results with other users or services, such as Jenkins and Certification Website.
You can find absolute paths of submission files displayed in the console right after the text summary block.
file:///home/user/.local/share/checkbox-ng/submission_2023-07-19T07.36.00.399658.html
file:///home/user/.local/share/checkbox-ng/submission_2023-07-19T07.36.00.399658.junit.xml
file:///home/user/.local/share/checkbox-ng/submission_2023-07-19T07.36.00.399658.tar.xz
Head to the folder ~/.local/share/
, you should find submission files
organized as follows:
├── html
├── junit
└── tar.xz
├── html
├── json
├── junit
└── attachments
Now, let’s explore each of the submission files together!
html
Open HTML files with your preferred web browser, you will see a System Testing Report with the following sections.
System Information
Tests Results
Logs
You can expand entries in Tests Results and Logs sections to access more detailed information. Additionally, you can utilize search bar to filter specific tests for easier navigation. For instance, type “audio” in the search bar to view all test results related to audio testing.
json
JSON files store session exports that are compatible for submission to Certification Website.
junit
JUnit XML files contain test data that can be read by Jenkins.
tar.xz
The xz compressed tarball is a comprehensive archive that includes the aforementioned reports and all associated attachments, such as I/O logs and binary files. You can extract the tarball with
tar -xf sumbission.tar.xz -C /path/to/destination
.Certification Website only accepts submissions tarballs, from which it extracts the
submission.json
file to create a new test report in the database.