Launchers¶

Checkbox features and behaviors can be configured and customized using a launcher.

In this section, you will use a launcher to filter the list of test plans displayed when running Checkbox, pre-select a test plan, automatically execute a test plan without manual interaction and configure the environment variables used by some of the test cases. Finally, you will tailor the output generated by Checkbox to your needs.

Filter the list of test plans¶

When you ran Checkbox in the previous section, you were probably overwhelmed with the number of test plans available. Using a launcher, it is possible to filter them to only select the ones that matter to you.

Create a file named mylauncher and add the following information in it:

mylauncher¶

[launcher]
launcher_version = 1

[test plan]
filter = *TODO*

Save it, then launch Checkbox using this launcher:

checkbox.checkbox-cli launcher mylauncher

The test plan selection screen should be much less intimidating now!

Let’s review the content of this launcher.

The [launcher] section contains meta-data related to the launcher itself launcher_version is set to 1 as it is the version of the launcher syntax currently in use.

In the [test plan] section, we set the filter to *TODO*. By doing so, the only test plans that will be displayed are the one whose id contain the word TODO. Note the use of the * glob pattern, similar to Bash for instance.

Try replacing *TODO* with *wireless* and see how it affects the list of test plans displayed in Checkbox.

Select a test plan by default¶

Set the filter back to *TODO* in the [test plan] section and add the following line:

mylauncher¶

[launcher]
launcher_version = 1
app_id = com.canonical.certification:tutorial

[test plan]
filter = *TODO*
unit = com.canonical.certification::smoke

Start Checkbox using the launcher. In addition to filtering the list of test plans, the TODO test plan is now selected by default. You just need to press Enter to go to the test selection screen.

Bypass test plan and test selection screens¶

Now, let’s say that when you run Checkbox, you always want to select the same test plan, and you always want to run all the tests in it. You don’t want to spend time in the test plan selection screen nor the test selection screen. Modify your launcher so it looks like that:

mylauncher¶

[launcher]
launcher_version = 1
app_id = com.canonical.certification:tutorial

[test plan]
unit = com.canonical.certification::TODO
forced = yes

[test selection]
forced = yes

Run Checkbox with this modified version of the launcher:

checkbox.checkbox-cli launcher mylauncher

Notice how none of the initial screens are shown and Checkbox immediately runs the TODO test plan. This is because:

in the [test plan] section, we selected a test plan with unit = TODO and we forced its use with forced = yes, bypassing the test plan selection screen;
in the [test selection] section, we forced the selection of all the tests, bypassing the test selection screen.

Customize test cases with environment variables¶

One of the test cases in the Tutorial test plan uses the value set in an environment variable. Add the following lines in the launcher:

mylauncher¶

[launcher]
launcher_version = 1
app_id = com.canonical.certification:tutorial

[test plan]
unit = com.canonical.certification::TODO
forced = yes

[test selection]
forced = yes

[environment]
TUTO = tutorial

Run Checkbox using your launcher, and observe the output of the TODO test case. The output now shows tutorial.

The [environment] section is often used to provide customized values to test cases. For instance, you may have a generic test case to connect to a WiFi access point, but its SSID and password might change, so you can use an environment variable in the test case definition and set their values in the [environment] section of your launcher.

Tailor Checkbox output¶

At the end of the test session, Checkbox summarizes the test results on the screen, generates test reports and test archive, and asks you whether you want to upload the test results to the Canonical Certification website. Let’s say you don’t need to upload the results there; you are only interested in the text summary and the test reports.

Edit the launcher file:

mylauncher¶

[launcher]
launcher_version = 1
app_id = com.canonical.certification:tutorial
stock_reports = text, submission_files

[test plan]
unit = com.canonical.certification::TODO
forced = yes

[test selection]
forced = yes

[environment]
TUTO = tutorial

Run Checkbox using this launcher and observe that once the test plan is finished running, Checkbox generates a summary on the screen and provides the links to the test reports and test archive, but does not ask if the result should be uploaded to the Canonical Certification website.

This is thanks to the customization of the stock_reports field in the [launcher] section. If not specified in the launcher, its default value is set to text, certification, submission_files.

In Checkbox language, submissions files are the HTML test report as well as an archive containing the test results and additional logs that might have been produced by the test cases.

Create an executable launcher¶

So far, we have called our launcher using the launcher option of the checkbox-cli tool. It is however possible to turn our launcher into a file that can be interpreted, similarly to an executable bash script.

At the top of the launcher file, add this line:

mylauncher¶

#!/usr/bin/env checkbox.checkbox-cli

[launcher]
launcher_version = 1
app_id = com.canonical.certification:tutorial
stock_reports = text, submission_files

[test plan]
unit = com.canonical.certification::TODO
forced = yes

[test selection]
forced = yes

[environment]
TUTO = tutorial

Make the launcher executable:

chmod +x mylauncher

Run it:

./mylauncher

Checkbox runs exactly like before! The line we added is called a shebang and allows us to run checkbox.checkbox-cli using the configuration provided.

Wrapping up¶

In this section, you’ve got more familiar with Checkbox launchers. You created a launcher that modified the behavior of Checkbox by pre-selecting a test plan and executing it, provided environment variables to the test cases and outputted only what you needed: a text summary and the submission files.

There are many more options available in the launchers to customize Checkbox runs. Please check the launchers reference for more information.