Test output

When Checkbox runs the tests, it generates a series of outputs, which include the ID and Category of the tests, along with the results of each of them (if a test ID fails, the cause for the failure would be displayed).

Now we will introduce each section of the test output by performing a test for audio device detection in practice.

=========[ Running job 1 / 3. Estimated time left (at least): 0:00:01 ]=========

Progress Indicator shows the current status of the ongoing job, including the total number of jobs and the number completed. It also provides an estimated time left for completion, helping users track progress and estimate when the test will finish.

-----------------------------[ Hardware Manifest ]------------------------------

Summary provides a brief sentence to help users understand the purpose of this job.

In this case, it’s Hardware Manifest, which provides information about various hardware devices used in the tests. When certain tests involve specific hardware devices, Checkbox will prompt users to inform in advance whether the machine includes specific hardware devices required for testing purposes.

ID: com.canonical.plainbox::manifest
Category: com.canonical.plainbox::uncategorised

ID indicates the job IDs run during the test. and it’s organized into different Categories based on their functionalities or characteristics.

... 8< -------------------------------------------------------------------------
ns: com.canonical.certification
name: checkbox-provider-base
has_audio_playback: False
has_audio_capture: True

------------------------------------------------------------------------- >8 ---

This is the job stdout from com.canonical.plainbox::manifest. If a job fails or is skipped, the message will also be shown in this section.

In our case, we are testing audio device detection, and the manifest was prompted earlier to ask for user-provided information, this section displays the collected information from the user.

Outcome: job passed

At the end of each test, the Outcome will be displayed to indicate whether the test has passed.

------------[ Check that at least one audio playback device exits ]-------------
ID: com.canonical.certification::audio/detect-playback-devices
Category: com.canonical.plainbox::audio
Job cannot be started because:
 - resource expression "manifest.has_audio_playback == 'True'" evaluates to false
Outcome: job cannot be started

Oops! Apparently this job was skipped.Based on the stdout, it seems that this issue occurred because when Checkbox asking hardware manifest, we mistakenly set has_audio_playback to False. Consequently, Checkbox determined that the machine lacks the necessary audio devices to support the test.

☑ : Hardware Manifest
☐ : Check that at least one audio playback device exits
☑ : Check that at least one audio capture device exists

After all the jobs are completed, a checklist will summarize the results of each test.

file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.html
file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.junit.xml
file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.tar.xz

Checkbox collect all data related to the test and generates a Test Reports locally.

Do you want to submit 'upload to certification' report?
  y => yes
  n => no

Users would be asked whether they want to upload the results to the Canonical Certification website.

Congrats! Now you have a basic understanding of the test output.