How to create and run confidence tests

Creating the test

To create a confidence test, first, go to the corresponding project page.

🚧

Don't have a project yet?

All tests live inside a project. If you haven’t created a project yet, make sure you create it first. In case you missed it, here is a tutorial about it.

On the project page, you will find the Tests block. That’s where all the tests you create will appear.

On the upper right corner of the Tests block, you will find a Create tests button. Click on Create tests to create a test.

You will be redirected to the test creation page. The first thing you’ll see, at the top of the page is the test category to select from.

Available test types for tabularAvailable test types for tabular

Available test types for tabular

Available test types for NLPAvailable test types for NLP

Available test types for NLP

🛠

️ Reach out

If you would like to use other testing frameworks, feel free to reach out so that we can accommodate your needs!

To create a confidence test, select Confidence. After selecting it, the parameters control panel should appear below it.

Understanding the parameters:

  • Expectation: for a particular data sample, the model will pass the test if it outputs a label with confidence above the threshold. This is where the expected confidence threshold for the label prediction is specified;
  • Sample size: if a tag is not selected, this parameter indicates the number of samples that will be drawn from the dataset at random to evaluate the model. If a tag is selected, this parameter is not used;
  • Sample from tags: specify exactly the data samples on which the model is evaluated.

Once you’ve defined the test parameters, you can click on Create on the right-hand part of the page.

Running the test

After creating a test, on the test panel on the test page, the newly created test is added to the table. Click on Run to run the test.

After the test finished running, you will be able to see information such as the number of passes and the number of failures.

Click on Open to look at the results in detail.

That's it! You've generated and run a confidence test!


Did this page help you?