Bias

Overview

The Bias metric measures bias in AI responses across five protected attributes using a guardian-based detection system. For each interaction, a Guardian evaluates whether the response exhibits bias toward any protected group.

Protected attributes

Gaussia evaluates bias across these attributes by default:

Attribute	Description
`gender`	Gender identity and expression
`race`	Race and ethnic background
`religion`	Religious beliefs and affiliations
`nationality`	National origin and citizenship
`sexual_orientation`	Sexual orientation and romantic attraction

Usage

from gaussia.metrics.bias import Bias
from gaussia.guardians import MyGuardian  # Your Guardian implementation

results = Bias.run(
    MyRetriever,
    guardian=MyGuardian,
)

for r in results:
    for rate in r.attribute_rates:
        print(f"{rate.protected_attribute}: {rate.rate:.3f} ({rate.k_biased}/{rate.n_samples})")

Parameters

Parameter	Type	Default	Description
`retriever`	`type[Retriever]`	required	Retriever class
`guardian`	`type[Guardian]`	required	Guardian class for bias detection
`statistical_mode`	`StatisticalMode`	`FrequentistMode()`	Statistical computation mode

Output schema

BiasMetric

Field	Type	Description
`session_id`	`str`	Session identifier
`assistant_id`	`str`	Assistant identifier
`attribute_rates`	`list[AttributeBiasRate]`	Bias rate per protected attribute
`guardian_interactions`	`dict`	Per-attribute interaction details

AttributeBiasRate

Field	Type	Description
`protected_attribute`	`str`	The attribute being evaluated
`n_samples`	`int`	Total interactions evaluated
`k_biased`	`int`	Number of biased interactions
`rate`	`float`	Bias rate (0–1)
`ci_low`	`float \| None`	Lower CI bound (Bayesian only)
`ci_high`	`float \| None`	Upper CI bound (Bayesian only)

GuardianInteraction

One entry per interaction and protected attribute, under guardian_interactions[attribute].

Field	Type	Description
`qa_id`	`str`	Interaction identifier
`attribute`	`str`	The attribute evaluated
`is_biased`	`bool`	The guardian’s verdict
`certainty`	`float \| None`	P(violation) for this interaction — high means biased, low means not, whichever way `is_biased` went. `None` when the guardian exposed no distribution to read
`method`	`str \| None`	How `certainty` was read: `logprob-last-verdict-token`, `logprob-verdict-softmax`, or `sampled-answer` when the verdict came from the text alone

certainty is absent, not 1.0, whenever the serving provider returns no logprobs — which is a property of the server rather than of the model. Branch on None (or on method) before averaging or thresholding it.

Guardian interface

To use the Bias metric, implement a Guardian subclass:

from gaussia.core.guardian import Guardian

class MyGuardian(Guardian):
    def is_biased(self, question, answer, attribute, context):
        # Return a GuardianBias with is_biased, attribute, certainty and method.
        # Report certainty=None rather than a placeholder when no distribution is available.
        ...

Requires the bias extra: pip install "gaussia[bias]".

Getting started

Core concepts

Metrics

Integrations

Advanced

Overview

Protected attributes

Usage

Parameters

Output schema

BiasMetric

AttributeBiasRate

GuardianInteraction

Guardian interface

​Overview

​Protected attributes

​Usage

​Parameters

​Output schema

​BiasMetric

​AttributeBiasRate

​GuardianInteraction

​Guardian interface

Overview

Protected attributes

Usage

Parameters

Output schema

BiasMetric

AttributeBiasRate

GuardianInteraction

Guardian interface