Skip to main content

Usage

The report command analyzes raw JSON outputs from scenario matrix runs and generates comprehensive performance reports with statistics, parameter comparisons, and trajectory analysis.

Subcommands


generate

Generate a comprehensive report from scenario matrix execution data.

Usage

Arguments

Options

Examples

Report Contents

The generated report includes:

Performance Statistics

  • Execution time metrics (min, max, average)
  • Success/failure rates
  • LLM call counts and token usage

Parameter Analysis

  • Comparison across different parameter combinations
  • Performance impact of each parameter value
  • Optimal parameter identification

Trajectory Analysis

  • Agent decision paths
  • Action sequences
  • State transitions

Matrix Output Structure

The report command expects matrix output in this structure:
  • scenario: Execute scenarios and generate output
  • test: Run tests for projects and plugins