Review Test Report¶
Once you finish your first test run, you may want to review the test reports. Checkbox generates test reports in different formats that can be used to easily share the results of a test session. By default, a text summary and submission files are created. This section will provide you with information on where to locate the report files and their respective contents.
For a detailed description of the
submission.json file, see the Submission JSON Schema reference.
You should see a text summary displayed in the console once all jobs are
completed. The result of each job is presented in the format:
summary, and it looks like:
☑ : A job that always passes
☒ : A job that always fails
⚠ : A job that always crashes
☑ : A job that depends on other job that passes
☐ : A job that is skipped because it depends on a job that fails
☑ : A job that generates different resources for tutorial purposes
☑ : A job that requires a resource and it is available
Checkout this table to map outcome symbols to their corresponding descriptions:
job didn’t run
job skipped, job cannot be started
job is not implemented
job needs verification
As text summary only provides an overview, for more in-depth information, you will need to review submission files.
In Checkbox, submission files encompass report files in various formats (HTML, JSON, and JUnit XML) along with attachments like I/O logs and binary files. These files serve the purpose of sharing test results with other users or services, such as Jenkins and Certification Website.
You can find absolute paths of submission files displayed in the console right after the text summary block.
Head to the folder
~/.local/share/, you should find submission files
organized as follows:
Now, let’s explore each of the submission files together!
Open HTML files with your preferred web browser, you will see a System Testing Report with the following sections.
You can expand entries in Tests Results and Logs sections to access more detailed information.
Additionally, if the report is more complex, you can utilize the search bar to filter specific tests for easier navigation. For instance, if you type “audio” in the search bar, you can view all test results related to audio testing.
JSON files store session exports that are generated using the pre-defined Submission JSON Schema.
JUnit XML files contain test data that can be read by Jenkins.
The xz compressed tarball is a comprehensive archive that includes the aforementioned reports and all associated attachments, such as I/O logs and binary files. You can extract the tarball with
tar -xf submission.tar.xz -C /path/to/destination.
Certification Website only accepts submissions tarballs, from which it extracts the
submission.jsonfile to create a new test report in the database.