.. _launcher-tutorial: Checkbox launchers tutorial ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Checkbox launchers are INI files that customize checkbox experience. The customization includes: * choosing what jobs will be run * how to handle machine restart * what type of UI to use * how to handle the results Each section in the launcher is optional, when not supplied, the default values will be used. This tutorial describes Launchers version 1. External configuration files ============================ Launcher can specify external file(s) to load values from. ``[config]`` Beginning of the configuration section. ``config_filename`` Name of the configuration file to look for. Default value: ``checkbox.conf`` The directories that will be searched for the file are ``/etc/xdg/`` and ``~/.config/``. Example:: [config] config_filename = testing.conf This will make checkbox look for ``/etc/xdg/testing.conf`` and ``~/config/testing.conf`` files. The ``config_filename`` may be an absolute path, and may use environment variables Example:: [config] config_filename = $MYCONFIGS/testing.conf [config] config_filename = /home/ubuntu/next-testing.conf For more details about value resolution order see :doc:`configs` Launcher meta-information ========================= Launcher meta-information helps to provide consistent checkbox behaviour in the future. ``[launcher]`` Beginning of the launcher meta-information section. ``app_id`` This fields helps to differentiate between checkbox front-ends. This way sessions started with launcher with one ``app_id`` won't interfere with sessions started with a different launcher (provided it has ``app_id`` set to other value). The app_id should be in a IQN form. Default value: ``com.canonical:checkbox-cli`` ``app_version`` This field is purely informational. ``launcher_version`` Version of the launcher language syntax and semantics to use. ``api_flags`` API flags variable determines optional feature set. List of API flags that this launcher requires. Items should be separated by spaces or commas. The default value is an empty list. ``api_version`` API version determines the behaviour of the launcher. Each checkbox feature is added at a specific API version. Default behaviours don't change silently; explicit launcher change is required. Default value: ``0.99`` ``stock_reports`` Stock reports are shortcuts in creating common reports. Instead of having to specify exporter, transport and a report section in a launcher, you can use any number of the stock ones. In launchers version 1 there are 4 stock reports you may use: * ``text`` - print results as text on standard output * ``submission_files`` - write ``html``, ``xlsx``, ``json`` and ``xml`` files to ``$XDG_DATA_HOME`` directory (or to ``~/.local/share/`` if ``$XDG_DATA_HOME`` is not defined. * ``certification`` - send results to certification site * ``certification-staging`` - send results to staging version of certification site This field is a list; use commas or spaces to separate stock reports. The default value: ``text, certification, submission_files``. When using ``certification`` stock report, ``secure_id`` might be overridden by the launcher. To do this define ``secure_id`` in a ``transport:c3`` section (this is the transport that's used by the ``certification`` stock reports). Launcher section example: :: [launcher] app_id = com.foobar:system-testing launcher_version = 1 stock_reports = text Launcher using all defaults with overridden secure_id: :: [transport:c3] secure_id = 001122334455667788 Providers section ================= This section provides control over which providers are used by the launcher. ``[providers]`` Beginning of the providers section. ``use`` A list of globs, from which a provider id must match at least one in order to be used. By default all providers are used. Providers section example: :: [providers] use = provider1, provider2, provider-* Test plan section ================= This section provides control over which test plans are visible in the menus and optionally forces the app to use particular one. ``[test plan]`` Beginning of the test plan section. ``unit`` An ID of a test plan that should be selected by default. By default nothing is selected. ``filter`` Glob that test plan IDs have to match in order to be visible. Default value: ``*`` ``forced`` If set to ``yes``, test plan selection screen will be skipped. Requires ``unit`` field to be set. Default value: ``no``. Test selection section ====================== This section provides lets forcing of test selection. ``[test selection]`` Beginning of the test selection section ``forced`` If set to ``yes``, test selection screen will be skipped and all test specified in the test plan will be selected. Default value: ``no`` User Interface section ====================== This section controls which type of UI to use. ``[ui]`` Beginning of the user interface section ``type`` Type of UI to use. This has to be set to ``interactive``, ``silent``, ``converged``, or ``converged-silent``. Default value: ``interactive``, which runs the Checkbox command line version. Note: the ``converged`` and ``converged-silent`` UI types will launch the QML interface and requires checkbox-converged to be installed on your system. Note: using ``silent`` or ``converged-silent`` UI types requires forcing test selection and test plan selection. ``dont_suppress_output`` .. note:: This field is deprecated, use 'output' to specify which jobs should have their output printed to the screen. Setting this field to ``yes`` disables hiding of command output for jobs of type ``local``, ``resource`` and ``attachment``. Default value: ``no``. ``output`` This setting lets you hide output of commands run by checkbox. It can be set to one of the following values: - ``show`` - output of all jobs will be printed - ``hide-resource-and-attachment`` - output of resource and attachment jobs will be hidden, output of other job types will be printed - ``hide-automated`` - output of shell jobs as well as attachment and resource jobs will be hidden. Only interactive job command's output will be shown - ``hide`` - same as ``hide-automated``. This value is deprecated, use ``hide-automated`` Default value: ``show`` .. note:: Individual jobs can have their output hidden by specifying 'suppress-output' in their definition. ``verbosity`` This setting makes checkbox report more information from checkbox internals. Possible values are: - ``normal`` - report only warnings and errors. - ``verbose`` - report important events that take place during execution (E.g. adding units, starting jobs, changing the state of the session) - ``debug`` - print out everything Default value: ``normal`` .. note:: You can also change this behavior when invoking Checkbox by using ``--verbose`` and ``--debug`` options respectively. ``auto_retry`` If set to ``yes``, failed jobs will automatically be retried at the end of the testing session. In addition, the re-run screen (where user can select failed and skipped jobs to re-run) will not be shown. Default value: ``no``. ``max_attempts`` Defines the maximum number of times a job should be run in auto-retry mode. If the job passes, it won't be retried even if the maximum number of attempts have not been reached. Default value: ``3``. ``delay_before_retry`` The number of seconds to wait before retrying the failed jobs at the end of the testing session. This can be useful when the jobs relying on external factors (e.g. a WiFi access point) and you want to wait before retrying the same job. Default value: ``1``. .. warning:: When ``auto_retry`` is set to ``yes``, **every** failing jobs will be retried. This can be a problem, for instance, for jobs that take a really long time to run. To avoid this, you can use the ``auto-retry=no`` inline override in the test plan to explicitly mark each job you do not wish to see retried. For example:: id: foo-bar-and-froz _name: Tests Foo, Bar and Froz include: foo bar auto-retry=no froz In that case, even if job ``bar`` fails and auto-retry is activated, it will not be retried. Restart section =============== This section enables fine control over how checkbox is restarted. ``[restart]`` Beginning of the restart section ``strategy`` Override the restart strategy that should be used. Currently supported strategies are ``XDG`` and ``Snappy``. By default the best strategy is determined in runtime. Environment section =================== ``[environment]`` Beginning of the environment section Each variable present in the ``environment`` section will be present as environment variable for all jobs run. Example: :: [environment] TESTING_HOST = 192.168.0.100 .. _generating-reports: Generating reports ================== Creation of reports is govern by three sections: ``report``, ``exporter``, and ``transport``. Each of those sections might be specified multiple times to provide more than one report. Exporter -------- ``[exporter:exporter_name]`` Beginning of an exporter declaration. Note that ``exporter_name`` should be replaced with something meaningful, like ``html``. ``unit`` ID of an exporter to use. To get the list of available exporter in your system run ``$ plainbox dev list exporter``. ``options`` A list of options that will be supplied to the exporter. Items should be separated by spaces or commas. Example: :: [exporter:html] unit = com.canonical.plainbox::html Transport --------- ``[transport:transport_name]`` Beginning of a transport declaration. Note that ``transport_name`` should be replaced with something meaningful, like ``standard_out``. ``type`` Type of a transport to use. Allowed values are: ``stream``, ``file``, and ``certification``. Depending on the type of transport there might be additional fields. +-------------------+---------------+----------------+----------------------+ | transport type | variables | meaning | example | +===================+===============+================+======================+ | ``stream`` | ``stream`` | which stream to| ``[transport:out]`` | | | | use ``stdout`` | | | | | or ``stderr`` | ``type = stream`` | | | | | | | | | | ``stream = stdout`` | +-------------------+---------------+----------------+----------------------+ | ``file`` | ``path`` | where to save | ``[transport:f1]`` | | | | the file | | | | | | ``type = file`` | | | | | | | | | | ``path = ~/report`` | +-------------------+---------------+----------------+----------------------+ | ``certification`` | ``secure-id`` | secure-id to | ``[transport:c3]`` | | | | use when | | | | | uploading to | ``secure_id = 01``\ | | | | certification | ``23456789ABCD`` | | | | sites | | | | | | ``staging = yes`` | | | | | | | +---------------+----------------+ | | | ``staging`` | determines if | | | | | staging site | | | | | should be used | | | | | Default: | | | | | ``no`` | | | | | | | | | | | | | | | | | +-------------------+---------------+----------------+----------------------+ Report ------ ``[report:report_name]`` Beginning of a report declaration. Note that ``report_name`` should be replaced with something meaningful, like ``to_screen``. ``exporter`` Name of the exporter to use ``transport`` Name of the transport to use ``forced`` If set to ``yes`` will make checkbox always produce the report (skipping the prompt). Default value: ``no``. Example of all three sections working to produce a report: :: [exporter:text] unit = com.canonical.plainbox::text [transport:out] type = stream stream = stdout [report:screen] exporter = text transport = out forced = yes Launcher examples ================= 1) Fully automatic run of all tests from 'com.canonical.certification::smoke' test plan concluded by producing text report to standard output. :: #!/usr/bin/env checkbox-cli [launcher] launcher_version = 1 app_id = com.canonical.certification:smoke-test stock_reports = text [test plan] unit = com.canonical.certification::smoke forced = yes [test selection] forced = yes [ui] type = silent [transport:out] type = stream stream = stdout [exporter:text] unit = com.canonical.plainbox::text [report:screen] transport = outfile exporter = text 2) Interactive testing of FooBar project. Report should be uploaded to the staging version of certification site and saved to /tmp/submission.xml :: #!/usr/bin/env checkbox-cli [launcher] launcher_version = 1 app_id = com.foobar:system-testing [providers] use = com.megacorp.foo::bar* [test plan] unit = com.megacorp.foo::bar-generic [ui] type = silent output = hide [transport:certification] type = certification secure-id = 00112233445566 staging = yes [transport:local_file] type = file path = /tmp/submission.xml [exporter:xml] unit = com.canonical.plainbox::hexr [report:c3-staging] transport = outfile exporter = xml [report:file] transport = local_file exporter = xml