Checkbox by itself doesn’t test anything. It uses unit definitions grouped in providers to actually run or do something.
To better understand how Checkbox works let’s concentrate on the relationship between the following entities:
test command (program that is invoked as a test)
Automated and some interactive tests use external commands to help determine the outcome of the test. For instance the command:
ping 22.214.171.124 -c 1
Will check whether the device can ping a public DNS. The command returns 0 on success (ping came back), and 1 if the ping timed out.
Let’s turn this simple command into a Checkbox test.
The command from the above paragraph can now be used in a Job Unit:
id: ping-public-dns _summary: Ping public DNS plugin: shell command: ping 126.96.36.199 -c 1
Notice how the command field is a straight copy-paste of the Test Command
RELATIONSHIP: Test Command is a part of a Job Unit
Some test are fully manual and don’t run any commands.
When Checkbox is run from the command line without any parameters, i.e.:
It doesn’t present all the tests available in the system. Checkbox asks the user which Test Plan to use.
For how to directly run hand-picked jobs see: Run a particular test plan or a set of jobs.
Test Plans are units for grouping related jobs together. They also provide a mechanism for creating new jobs in runtime. The phase in which this is done is known as bootstrapping. Good example for when the bootstrapping is needed is testing multi-GPU system. No one knows upfront which GPU(s) will be present in the system so bootstrapping phase will instantiate appropriate job units from template units.
RELATIONSHIP: Test Plan includes a job that can be run.
RELATIONSHIP: Test Plan can “generate” a job (through bootstrapping) that can be run.
In order for Checkbox to see any jobs, test plans, and other units, those units need to be written to a .pxu file located in a unit subdirectory of a provider available in the system.
RELATIONSHIP: Units are placed in a Provider
See Checkbox tutorials for a tutorial on how to create a provider from scratch.
Launchers can be used to make it easier to run Checkbox in a preset way.
- Those can for instance preset:
which test plan to use
whether the session should be interactive or automated
which tests to exclude
how and where to submit the results
There is a full launcher reference here: Checkbox launchers.