Character length

​

Definition

The character length test allows you to define minimum and/or maximum bounds on the number of characters in a column.

​

Taxonomy

• Category: Integrity.
• Task types: LLM, text classification.
• Availability: development and monitoring.

​

Why it matters

• Extremely long or short text entries might be outliers or noise, such as corrupted data, spam, or non-relevant entries.
• Models often have limitations on the length of input they can effectively process. Inputs longer than this limit may be truncated, potentially losing important information, while very short inputs might not provide enough context for accurate processing. Making sure that your data falls within these limits is important to ensure model performance.
• If a model is trained on data with a certain length distribution, it might not perform well on texts of significantly different lengths.