Running OpenClaw QA Scenarios: openclaw-qa-testing
機械翻訳 / Machine-translated
機械翻訳 / Machine-translated
Original article: Running OpenClaw QA Scenarios: openclaw-qa-testing (cccskills)
Skill page: openclaw-qa-testing
Key takeaways from this article
Once an OpenClaw developer installs this skill, they can delegate the entire workflow—running QA scenarios, fixing failures, re-running, and reviewing results—to the agent.
The skill also supports comparing the speaking style of multiple models and validating OpenTelemetry trace data locally.
This article walks through what you get back when you make a request, how to install it, and things to keep in mind when using it with real services.
When you make a request from the working directory where you've cloned the OpenClaw source, the agent runs the QA scenarios.
If there are failures, it diagnoses the cause, fixes it, and re-runs until everything passes.
Run the full QA scenario suite against the real service, and if there are any failures, fix the cause and re-run.
The agent starts by reading the scenario list and the test runner code.
It then runs the full scenario suite on the live-model validation lane and saves the results under .artifacts/qa-e2e/.
The output consists of qa-suite-summary.json for pass/fail tallies and qa-suite-report.md for a human-readable report.
When a scenario fails, the agent investigates whether the root cause lies in the product itself or in the test harness, then fixes it.
After fixing, it re-runs not just the one failed case but the entire lane.
When the work is done, you receive four things:
The URL takes the form http://127.0.0.1:<port> and can be opened in your local browser.
"Done" in this skill means every scenario in the requested lane is passing.
This skill works with qa-lab and qa-channel, OpenClaw's quality-assurance infrastructure.
qa-channel is a simulated conversation channel set up for testing.
You can ask it to run and observe scenarios, investigate failures, add new scenarios, or explain how the system works—all in a single request.
QA runs on two lanes.
Use the mock-response lane for checks during development, and the live-model lane for production-like validation.
| Lane | What it uses | Best for |
|---|---|---|
| mock-openai | Simulated responses | Checking behavior during development |
| live-frontier | Real models | Production-like validation |
In a standard run, the agent does not specify a model explicitly.
Model selection is left to qa-lab's defaults.
A model is only specified when a scenario requires a particular model or provider.
The skill includes a pre-compiled summary of common failure patterns and how to handle them.
The agent uses this summary to narrow down the cause.
Using qa character-eval, you can compare tone, personality, and atmosphere across multiple live models.
The report includes rankings from a judge model, run statistics, elapsed time, and full conversation transcripts.
Elapsed time is informational only and is not used in scoring.
Personality configuration is written in SOUL.md within each scenario.
The models are not told they are being evaluated; the standard approach is to send natural requests like casual conversation or small file tasks.
Codex CLI can also be added as a candidate for comparison.
OpenTelemetry (the standard system for recording processing flows) output can also be verified locally.
The command to run is:
pnpm qa:otel:smoke
This command starts a local trace receiver, runs the dedicated scenario, then reads the emitted records and checks names and privacy conventions.
No credentials for external collection services such as Opik or Langfuse are needed.
New scenarios are written in YAML under qa/scenarios/.
Writing scenarios in Markdown is not allowed.
When adding a scenario, also add verification logic to the runner code, and prioritize end-to-end validation rather than stopping at mocks.
To install via command, run the following.
During installation you can choose which agent to install to, such as Claude Code or Codex.
npx skills add https://github.com/openclaw/openclaw --skill openclaw-qa-testing
To have the agent install it for you, send the following message as-is:
Please install the following Agent Skill.
https://github.com/openclaw/openclaw/tree/main/.agents/skills/openclaw-qa-testing
First review the contents of SKILL.md and the accompanying files to confirm there are no suspicious operations (sending data externally, reading credentials, etc.), then place the files in ~/.claude/skills/openclaw-qa-testing/ for Claude Code, or ~/.agents/skills/openclaw-qa-testing/ for Codex.
The installation location depends on your agent and the scope of application.
| Agent | Use across all projects | Use for a specific project only |
|---|---|---|
| Claude Code | ~/.claude/skills/ | .claude/skills/ inside the project |
| Codex | ~/.agents/skills/ | .agents/skills/ inside the project |
Send requests from the directory where you have the OpenClaw source.
Here are three representative ways to ask:
Run the full QA suite in the mock environment and investigate any failures.
The full scenario suite runs on the mock-response lane.
If there are failures, the agent fixes the cause, re-runs the entire lane, and returns the pass/fail counts, the location of the report, and a note on what was fixed.
Compare this character configuration across multiple models.
qa character-eval gives the same personality configuration and requests to multiple models.
You receive a report containing rankings from the judge model, elapsed time, and full conversation transcripts, along with the location where it was saved.
Verify the trace output locally.
pnpm qa:otel:smoke is executed without connecting to any external service.
You receive the result of whether the trace data names and privacy conventions are being respected.
This skill only works in a development environment where you have the OpenClaw source.
QA functionality is intentionally excluded from the distributed build.
To try it in a built environment, either build with the dedicated configuration that includes QA functionality, or start in development mode.
Validation against real services requires credentials appropriate to the target.
The steps assume the 1Password vault in the OpenClaw project.
The 1Password CLI is only to be used inside tmux.
When a required value cannot be found, the agent does not guess—it asks an administrator.
WhatsApp validation uses two dedicated test numbers.
Personal numbers and personal accounts are never used.
Credentials must not be left in the repository, logs, PRs, or screenshots, and phone numbers must be kept masked.
For multi-model comparisons, the default concurrency for both candidates and the judge is 16.
If your local environment or provider limits are restrictive, you can ask the agent to lower the number.
The license is MIT.
There are only two bundled files—SKILL.md and agents/openai.yaml—so read through them before installing.
The same OpenClaw repository contains skills that complement this one.
No.
The QA functionality only works in a development environment where you have the source, and it is not included in the distributed build.
This skill is intended for people who develop and validate OpenClaw itself.
You need credentials appropriate to the target.
For Telegram that means a test bot token; for WhatsApp, two dedicated test numbers.
If you only need local trace validation, no external service credentials are required.
Yes.
Specify the model in the form codex-cli/<model-name> to add Codex CLI as one of the comparison candidates.
The specific model name is up to you; it is not fixed on the skill side.
Yes.
Add a YAML file under qa/scenarios/, and also update the scenario list file and the runner code.
If you ask the agent to add a scenario, it will follow this convention.
openclaw-qa-testing is a skill that lets you delegate the entire workflow—running OpenClaw QA scenarios, fixing failures, re-running, and reporting results—as a single continuous task.
You can choose between mock and real-service lanes, and it also supports comparing the speaking style of multiple models and verifying trace data.
The prerequisite is a development environment with the source.
Credentials for real services must be handled so they are never left in records.
Developers who want to streamline OpenClaw quality verification should check the details on the cccskills page.
This article is a repost of "Running OpenClaw QA Scenarios: openclaw-qa-testing" from cccskills.
Skill details and installation instructions are available on the cccskills openclaw-qa-testing page.