Contents
You’ve got Checkbox installed, right? Installation
To run command line version of Checkbox, in your terminal run checkbox-cli
.
You should be greeted with test plan selection screen:
With a test plan selected, you can choose the individual tests to run:
When the tests are run, the results are saved to files and the program prompts to submit them to Launchpad.
When checkbox is run without any arguments, i.e.:
$ checkbox-cli
Interactive session is started with the default options.
startprovider
subcommand creates a new provider, e.g.:
$ checkbox-cli startprovider 2016.com.acme:example
The command will also add example units to that provider, to create an empty
provider, use --empty
option, e.g.:
$ checkbox-cli --empty startprovider 2016.com.acme:another-example
list
command prints out all units of the following type.
Currently there are following types you can list:
- job
- test plan
- category
- file
- template
- file
- manifest entry
- packaging meta-data
- exporter
Example:
$ checkbox-cli list job
$ checkbox-cli list "test plan"
Note
For multi-word types like ‘test plan’ remember to escape the spaces in between, or enquote the type name.
launcher
command lets you customize checkbox experience.
See Checkbox launchers tutorial for more details.
Note
launcher
is implied when invoking checkbox-cli with a file as the only
argument. e.g.:
$ checkbox-cli my-launcher
is equivalent to:
$ checkbox-cli launcher my-launcher
run
lets you run particular test plan or a set of jobs.
To just run one test plan, use the test plan’s id as an argument, e.g.:
$ checkbox-cli run 2013.com.canonical.certification::smoke
To run a hand-picked set of jobs, use regex pattern(s) as arguments. Jobs with id matching the expresion will be run, e.g.:
$ checkbox-cli run 2016.com.acme:.*
Note
The command above runs all jobs which id begins with 2016.com.acme:
will be run
You can use multiple patterns to match against, e.g.:
$ checkbox-cli run .*true .*false
Note
The command above runs all jobs which id ends with ‘true’ or ‘false’
First, we installed some “provider” packages. Providers were designed to encapsulate test descriptions and their related tools and data. Providers are shipped in Debian packages, which allows us to express dependencies to ensure required external packages are installed, and we can also separate those dependencies; for instance, the provider used for server testing doesn’t actually contain the server-specific test definitions (we try to keep all the test definitions in the Checkbox provider), but it does depend on all the packages needed for server testing. Most users will want the resource and Checkbox providers which contain many premade tests, but this organization allows shipping the tiny core and a fully customized provider without extraneous dependencies.
A provider is described in a configuration file (stored in
/usr/share/plainbox-providers-1
). This file describes where to find all
the files from the provider. This file is usually managed automatically
(more on this later). A provider can ship jobs, binaries, data and test plans.
A job or test is the smallest unit or description that Checkbox knows about. It describes a single test (historically they’re called jobs). The simplest possible job is:
id: a-job
plugin: manual
_description: Ensure your computer is turned on. Is the computer turned on?
Jobs are shipped in a provider’s jobs directory. This ultra-simple example
has three fields: id
, plugin
, and description
. (A real job
should include a _summary
field, too.) The id
identifies the job
(of course) and the _description
provides a plain-text description of
the job. In the case of this example, the description is shown to the user,
who must respond because the plugin
type is manual
. plugin
types include (but are not limited to):
manual
– A test that requires the user to perform some action and report the results.shell
– An automated test that requires no user interaction; the test is passed or failed on the basis of the return value of the script or command.resource
– Job that identifies the resources that the system has. (e.g. discrete GPU, Wi-Fi module). This information can later be used by other jobs to control other jobs’ execution. (E.g. skip Wi-Fi tests if there’s no Wi-Fi chip).user-interact
– A test that asks the user to perform some action before the test is performed. The test then passes or fails automatically based on the output of the test. An example iskeys/media-control
, which runs a tool to detect keypresses, asks the user to press volume keys, and then exits automatically once the last key has been pressed or the user clicks the skip button in the tool.user-interact-verify
– This type of test is similar to theuser-interact
test, except that the test’s output is displayed for the user, who must then decide whether it has passed or failed. An example of this would be theusb/disk_detect
test, which asks the user to insert a USB key, click thetest
button, and then verify manually that the USB key was detected correctly.user-verify
– A test that the user manually performs or runs automatically and requires the user to verify the result as passed or failed. An example of this is the graphics maximum resolution test which probes the system to determine the maximum supported resolution and then asks the user to confirm that the resolution is correct.qml
– A test with GUI defined in a QML file. See QML-native Jobs Tutorial