A job unit is a smallest unit of testing that can be performed by Checkbox. All jobs have an unique name. There are many types of jobs, some are fully automated others are fully manual. Some jobs are only an implementation detail and a part of the internal architecture of Checkbox.
Jobs are expressed as sections in text files that conform somewhat to the
rfc822
specification format. Our variant of the format is described in
rfc822. Each record defines a single job.
Following fields may be used by the job unit:
id
:name
. That name is now deprecated. For
backwards compatibility it is still recognized and used if id
is
missing.summary
:plugin
:(mandatory) - For historical reasons it’s called “plugin” but it’s better thought of as describing the “type” of job. The allowed types are:
manual: jobs that require the user to perform an action and then decide on the test’s outcome. shell: jobs that run without user intervention and automatically set the test’s outcome. user-interact: jobs that require the user to perform an interaction, after which the outcome is automatically set. user-interact-verify: jobs that require the user to perform an interaction, run a command after which the user is asked to decide on the test’s outcome. This is essentially a manual job with a command. attachment: jobs whose command output will be attached to the test report or submission. resource: A job whose command output results in a set of rfc822 records, containing key/value pairs, and that can be used in other jobs’ requires
expressions.qml: A test with GUI defined in a QML file.
requires
:(optional). If specified, the job will only run if the conditions expressed in this field are met.
Conditions are of the form <resource>.<key> <comparison-operator>
'value' (and|or) ...
. Comparison operators can be ==, != and in
.
Values to compare to can be scalars or (in the case of the in
operator) arrays or tuples. The not in
operator is explicitly
unsupported.
Requirements can be logically chained with or
and
and
operators. They can also be placed in multiple lines,
respecting the rfc822 multi-line syntax, in which case all
requirements must be met for the job to run ( and
ed).
depends
:after
:(optional). If specified, the job will only run if all the listed jobs have run (regardless of the outcome). Multiple job names, separated by spaces, can be specified.
This feature is available since plainbox 0.24.
command
:(optional). A command can be provided, to be executed under specific
circumstances. For manual
, user-interact
and user-verify
jobs, the command will be executed when the user presses a “test”
button present in the user interface. For shell
jobs, the
command will be executed unconditionally as soon as the job is
started. In both cases the exit code from the command (0 for
success, !0 for failure) will be used to set the test’s outcome. For
manual
, user-interact
and user-verify
jobs, the user can
override the command’s outcome. The command will be run using the
default system shell. If a specific shell is needed it should be
instantiated in the command. A multi-line command or shell script
can be used with the usual multi-line syntax.
Note that a shell
job without a command will do nothing.
purpose
:steps
:verification
:user
:environ
:user
attribute.
This key provides a mechanism to account for security policies in
sudo
and pkexec
, which provide a sanitized execution
environment, with the downside that useful configuration specified
in environment variables may be lost in the process.estimated_duration
:(optional) This field contains metadata about how long the job is expected to run for, as a positive float value indicating the estimated job duration in seconds.
Since plainbox version 0.24 this field can be expressed in two formats. The
old format, a floating point number of seconds is somewhat difficult to
read for larger values. To avoid mistakes test designers can use the second
format with separate sections for number of hours, minutes and seconds. The
format, as regular expression, is (\d+h)?[: ]*(\d+m?)[: ]*(\d+s)?
. The
regular expression expresses an optional number of hours, followed by the
h
character, followed by any number of spaces or :
characters,
followed by an optional number of minutes, followed by the m
character,
again followed by any number of spaces or :
characters, followed by the
number of seconds, ultimately followed by the s
character.
The values can no longer be fractional (you cannot say 2.5m
you need to
say 2m 30s
). We feel that sub-second granularity does is too
unpredictable to be useful so that will not be supported in the future.
flags
:(optional) This fields contains list of flags separated by spaces or commas that might induce plainbox to run the job in particular way. Currently, following flags are inspected by plainbox:
preserve-locale
:win32
:noreturn
:
explicit-fail
:- Use this flag to make entering comment mandatory, when the user manually fails the job.
has-leftovers
:- This flag makes plainbox silently ignore (and not log) any files left over by the execution of the command associated with a job. This flag is useful for jobs that don’t bother with maintenance of temporary directories and just want to rely on the one already created by plainbox.
simple
:This flag makes plainbox disable certain validation advice and have some sesible defaults for automated test cases. This simiplification is meant to cut the boiler plate on jobs that are closer to unit tests than elaborate manual interactions.
In practice the following changes are in effect when this flag is set:
- the plugin field defaults to shell
- the description field is entirely optional
- the estimated_duration field is entirely optional
- the preserve-locale flag is entirely optional
A minimal job using the simple flag looks as follows:
id: foo command: echo "Jobs are simple!" flags: simple
preserve-cwd
:- This flag makes plainbox run the job command in the current working directory without creating a temp folder (and running the command from this temp folder). Sometimes needed on snappy (See http://pad.lv/1618197)
Additional flags may be present in job definition; they are ignored.
imports
:(optional) This field lists all the resource jobs that will have to be imported from other namespaces. This enables jobs to use resources from other namespaces. You can use the “as ...” syntax to import jobs that have dashes, slashes or other characters that would make them invalid as identifiers and give them a correct identifier name. E.g.:
imports: from 2013.com.canonical.certification import cpuinfo
requires: 'armhf' in cpuinfo.platform
imports: from 2013.com.canonical.certification import cpu-01-info as cpu01
requires: 'avx2' in cpu01.other
The syntax of each imports line is:
IMPORT_STMT :: "from" <NAMESPACE> "import" <PARTIAL_ID>
| "from" <NAMESPACE> "import" <PARTIAL_ID> AS <IDENTIFIER>