Test execution and SUT presets

This page explains how to configure test execution in Ogoron and how to define SUT presets (system-under-test runbooks).

The configuration file is:

.ogoron/configs/test_execution.yml

When this matters

You typically configure test_execution.yml when:

• you want ogoron run tests to run your test stack consistently (locally and in CI)
• you start using workflows that execute tests as part of a loop (for example, ogoron heal tests)

File shape (high-level)

test_execution.yml has four main parts:

• sut_root: where the project under test lives
• stack: auto or an explicit stack adapter (pytest, rspec, java, dotnet)
• profiles: named execution environments (typically local and ci)
• sut: SUT presets, with up, healthcheck, and down hooks

Start with the smallest possible working file and add details as needed.

Minimal file:

version: 1

sut_root: "{project_root}"
stack: auto

profiles:
  local:
    backend: local_shell
    cwd: "{sut_root}"

sut:
  default_preset: noop
  presets:
    noop:
      up: []
      healthcheck: []
      down: []

Profile selection

Ogoron selects a profile by these rules:

• if --profile is provided, Ogoron uses it
• else, if CI=true and profile ci exists, Ogoron uses ci
• else, if profile local exists, Ogoron uses local
• else, if exactly one profile exists, Ogoron uses it
• else, Ogoron requires --profile

Profiles

A profile selects where and how commands are executed.

Supported profiles.<name>.backend values:

• local_shell — run commands on the host shell
• compose — run commands via Docker Compose (see profiles.<name>.compose)

Common profile fields:

• cwd: working directory for the profile (supports placeholders like {sut_root})
• env: environment variables (merged with step-specific env)
• python_exe: optional Python interpreter path for stacks that need it

Compose profile fields (required when backend: compose):

• project_directory: where the compose project is located (usually {sut_root})
• files: a non-empty list of compose files
• service: the service name used to run commands
• container_workdir: optional workdir inside the container
• artifacts_mount: whether to mount .ogoron/ into the container for reports
• artifacts_container_dir: mount point inside the container (default: /ogoron-artifacts)
• down_on_finish, down_remove_orphans: compose cleanup toggles

If your SUT preset manages docker compose up/down explicitly, set down_on_finish: false in the compose profile and define sut.presets.<name>.down.

Stack selection

stack: auto uses rule-based detection. If auto-detection is not reliable for a repository, set stack explicitly:

stack: pytest

SUT presets

SUT presets are project-defined runbooks describing how to prepare the environment before tests run.

Each preset can define three hooks:

• up: bring dependencies up (containers, services, local dev server)
• healthcheck: verify the environment is ready
• down: cleanup (runs even if tests fail)

Each hook is a list of steps. A step is a mapping with:

• cmd (required): either an argv list (recommended) or a shell string
• name (optional): label used in logs and reports
• backend (optional): local_shell|compose (defaults to the selected profile backend)
• cwd (optional), env (optional), timeout_seconds (optional)

Placeholders supported in strings:

• {project_root}
• {sut_root}

Template: no SUT (unit tests only)

Use a noop preset when tests do not require external services:

sut:
  default_preset: noop
  presets:
    noop:
      up: []
      healthcheck: []
      down: []

Template: shell-based SUT (local services)

This template starts a local dependency and verifies it is reachable.

sut:
  default_preset: local
  presets:
    local:
      up:
        - name: start-deps
          cmd: ["./scripts/dev-up.sh"]
      healthcheck:
        - name: wait-http
          cmd: ["./scripts/wait-http.sh", "http://localhost:8080/health"]
          timeout_seconds: 120
      down:
        - name: stop-deps
          cmd: ["./scripts/dev-down.sh"]

Template: compose-based execution profile

This profile runs commands inside a Compose service.

profiles:
  ci:
    backend: compose
    cwd: "{sut_root}"
    compose:
      project_directory: "{sut_root}"
      files: ["docker-compose.yml"]
      service: app
      artifacts_mount: true
      artifacts_container_dir: "/ogoron-artifacts"
      down_on_finish: false

Template: compose-based SUT preset

This preset brings a compose environment up, checks readiness, and tears it down.

sut:
  default_preset: full
  presets:
    full:
      up:
        - name: compose-up
          backend: local_shell
          cmd: ["bash", "-lc", "docker compose up -d --build"]
      healthcheck:
        - name: healthcheck
          backend: local_shell
          cmd: ["bash", "-lc", "docker compose exec -T app ./scripts/healthcheck.sh"]
          timeout_seconds: 180
      down:
        - name: compose-down
          backend: local_shell
          cmd: ["bash", "-lc", "docker compose down -v --remove-orphans"]

Template: assets prebuild step

Use a dedicated up step for deterministic assets building when the application requires it.

sut:
  presets:
    full:
      up:
        - name: build-assets
          cmd: ["bash", "-lc", "make assets"]
        - name: compose-up
          cmd: ["bash", "-lc", "docker compose up -d --build"]

Selecting a profile and a SUT preset

For ogoron run tests, you can override both:

• --profile <name> selects profiles.<name>
• --sut-preset <name> selects sut.presets.<name>

If --sut-preset is not provided, Ogoron uses sut.default_preset.

An empty --sut-preset "" disables SUT steps for that run.

Common validation errors

Most configuration errors are discovered before tests start.

Common causes:

• missing profiles or an empty profiles list
• backend values other than local_shell or compose
• compose.files is missing or empty
• sut.default_preset points to a missing preset
• a SUT hook contains a non-list value (each hook must be a list of steps)