How to create and run invariance tests


️ Reach out

Currently, invariance tests are only available for NLP. Let us know if you'd like it to be available for other data types as well!

Creating the test

If you are new to invariance tests, we recommend checking out our tutorial on it. However, this how-to guide is self-contained, so feel free to follow it if you prefer.

To create an invariance test, first, go to the corresponding project page.


Don't have a project yet?

All tests live inside a project. If you haven’t created a project yet, make sure you create it first. In case you missed it, here is a tutorial about it.

On the upper right corner of the Tests block, you will find a Create tests button. Click on Create tests to create a test.

You will be redirected to the test creation page. The first thing you’ll see, at the top of the page is the test category to select from.


️ Reach out

If you would like to use other testing frameworks, feel free to reach out so that we can accommodate your needs!

To create a confidence test, select Invariance. After selecting it, the parameters control panel should appear below it.

Understanding the parameters:

  • Type: the invariance tests at Unbox are based on the CheckList, a testing methodology for NLP models proposed by Ribeiro, Guestrin et al. All the available types with explanations can be checked out on the platform;
  • Sample size: controls the number of tests that will be generated.
    • number of tests: there are two options for this parameter. One possibility is defining an integer, which corresponds to the number of samples that we will draw from our dataset at random to apply the modifications. Alternatively, we can specify a tag at the bottom of the page, directly specifying which samples will be used;
    • number of tests per row: the number of perturbations for each row selected.

Once you’ve defined the test parameters, you can click on Create on the right-hand part of the page.

Running the test

After creating a test, on the test panel on the test page, the newly created test is added to the table. Click on Run to run the test.

After the test finished running, you will be able to see information such as the number of passes and the number of failures.

Click on Open to look at the results in detail.

That's it! You've generated and run an invariance test!

Did this page help you?