Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .changeset/runner-surface-0-27.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
"@qawolf/cli": minor
---

The `qawolf runner` group speaks `@qawolf/api-contracts` 0.27.0, gains four commands, and ships a run's files by walking the flow's imports instead of reading the whole directory.

`qawolf runner run` now sends the flow file, everything it imports, and your `package.json` and `tsconfig.json`. Nothing else travels. Before this change it read every runnable file under the working directory, so on a project of a few thousand files it built a payload the platform refused on size and the run never started; you had to run from a directory holding the flow and little else. Imports are followed the way a run from the QA Wolf app follows them, including its limits: relative paths and `tsconfig.json` aliases, resolving `.ts` and `.js`, and not following `export ... from` or `require()`.

After the first run on a runner, later runs send only the files whose content changed. The baseline lives in `.qawolf/runner-files.json`. A switch to another runner ignores it, a runner that turns out not to hold what was claimed gets the whole set resent, and `--json` reports which happened as `fileSync`.

`qawolf runner run --lines 12-40` runs those lines against the browser as it stands, so you can iterate on one step without paying for the whole flow to reach it again. `--lines-file` says where the range lives when it is not the flow file, and the positional is now `<flowFile>` in the usage line because two file paths with one unlabelled is easy to get backwards. `--env-file` gives the run environment variables from a dotenv file, in the format `qawolf flows pull` writes.

`qawolf runner inspect` reads one thing off the live page and prints it on stdout by itself: `element-html --selector`, `page-html`, or `variable --name`, which is a shorter path to a value than printing it from a snippet and reading it back out of the `console` stream. `qawolf runner stop-run` stops what a runner is executing and leaves the runner up. `qawolf runner import-package` installs a package into a live run so a snippet or a selection can import it without a full run to reinstall dependencies.

Two breaking changes to the `runner` group. `qawolf runner stop` is now `qawolf runner terminate`, which better separates ending a runner from stopping a run on one. And `--name` takes a runner family, `playwright`, `android`, `ios` or `basic`, in place of an image name like `node20WithPlaywright`. Every runner verb also reports a failure as an `outcome` of `failure` with a `failureReason`, in place of one outcome per condition, so `--json` consumers reading `outcome` for a specific condition need to read `failureReason` instead.
2 changes: 1 addition & 1 deletion skills/qawolf-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ current branch.
| `qawolf runner inspect variable` | read | Print a top-level variable's value from the running workflow |
| `qawolf runner keepalive` | read | Reset a runner's inactivity clock, for a caller that pauses between actions |
| `qawolf runner launch` | write | Launch an interactive runner and make it this directory's default |
| `qawolf runner run` | write | Run a flow on an interactive runner, shipping the current directory's files with it |
| `qawolf runner run` | write | Run a flow on an interactive runner, shipping the flow and what it imports |
| `qawolf runner screenshot` | read | Save a JPEG of an interactive runner's screen to a file |
| `qawolf runner stop-run` | write | Stop what a runner is currently executing, leaving the runner up |
| `qawolf runner terminate` | write | End an interactive runner, and the pod it runs on with it |
Expand Down
76 changes: 72 additions & 4 deletions skills/qawolf-cli/references/runner.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,15 @@ None of that is a fault, and none of it clears on its own. **Only
`qawolf runner run <flow>` starts the screen.** A bare navigate does not: it
fails until the first run, however long you wait.

So the first call on a new runner has to be a run. That means a flow file and a
`package.json` on disk, even if all you want is to drive the browser by hand;
there is no "just give me a screen" call. Once one run has happened, the
screenshot-and-act loop below works for the rest of the runner's life.
So the first thing you do to a new runner has to put a browser on it. That means
a flow file and a `package.json` on disk, even if all you want is to drive the
browser by hand; there is no "just give me a screen" call. Once one run has
happened, the screenshot-and-act loop below works for the rest of the runner's
life.

A `--lines` selection is the one call that does not need a run first. Sent to a
runner with no browser, the runner starts one and runs your lines against it,
and says so on stderr. Everything else on this page waits for a run.

Retry on the exit code, not on the message text:

Expand Down Expand Up @@ -128,6 +133,50 @@ has a browser context, so an early empty answer means "not yet", not "broken".
Do not add `--json` here: it wraps each line in an envelope and these field paths
stop matching.

## Reading the page: `inspect`

`qawolf runner inspect` answers one question about the live page and prints the
answer on stdout by itself, so you can redirect or pipe it:

```sh
qawolf runner inspect element-html --selector "#email"
qawolf runner inspect page-html > page.html
qawolf runner inspect page-html --selector "#cart" # just that subtree
qawolf runner inspect variable --name cart | jq .total
```

`page-html` is simplified for a model to read rather than being the browser's
exact markup. `variable` reads a top-level variable of the running workflow and
prints it as JSON, which is how you see what your own code computed rather than
what the page shows.

One failure covers three causes, because a runner cannot tell them apart: no
live page, no element matching the selector, no variable under that name. All
three exit `2` and none clears by waiting, so read the message, which carries
whatever the runner said. An unreachable runner exits `4` and is worth retrying.

Use `inspect` before reaching for `exec`. Reading a value through a snippet
means printing it and then fishing it back out of the `console` stream, which is
two calls and a marker; `inspect variable` is one call and the value.

## Installing a package mid-session

`qawolf runner import-package <name>` installs a package into the runner's live
run, so a snippet or a selection can import it without a whole run to reinstall
dependencies:

```sh
qawolf runner import-package dayjs
qawolf runner import-package dayjs --package-version 1.11.13
```

The version defaults to `latest`, and the flag is `--package-version` because
`--version` belongs to the CLI itself. The install resolves against your
project's own dependencies, read from `package.json`, so it needs a run already
going: there is no live run on a runner that has not run anything. npm's own
refusal comes back verbatim on an exit `2`, which is a name or a version to
correct rather than something to retry.

## Reading the page: `exec`

`qawolf runner exec <file>` evaluates a snippet against whatever the runner's
Expand Down Expand Up @@ -322,6 +371,21 @@ on a timer you forget, and call `qawolf runner terminate` when you are done rath
than leaving a pod to time out. A loop that keeps a runner alive and never stops
it bills until someone notices.

## Stopping a run vs ending a runner

Two different things, and the names are the only warning you get:

- `qawolf runner stop-run` stops what the runner is executing and leaves the
runner up, its browser on whatever page the run reached. The run settles as
stopped rather than passed or failed. Use it to abandon a run and keep the
browser you were working against.
- `qawolf runner terminate` ends the runner and the pod with it. Everything on
it is gone, and the next command under that id launches and bills a new one.

Both succeed when there was nothing to do, and say which: `wasRunning` is
`false` when no run was going, and when no runner was running. Neither is an
error, so a retry needs no special handling.

## End to end

Run from a directory holding a flow and a `package.json`. The run is what starts
Expand All @@ -339,6 +403,10 @@ qawolf runner screenshot --out page.jpg # then read page.jpg yourself
qawolf runner act click --button left --x 480 --y 260
qawolf runner act type --text "someone@example.com"

qawolf runner inspect element-html --selector "#email"
qawolf runner inspect variable --name cart | jq .total

qawolf runner run flows/smoke.flow.ts --lines 12-40 --follow # just those lines
qawolf runner events recorder --tail 5 | jq -r '[.locator] + (.alternates // []) | @tsv'
qawolf runner terminate
```
5 changes: 2 additions & 3 deletions src/commands/__snapshots__/help.test.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,7 @@ Commands:
keepalive [options] Reset a runner's inactivity clock, for a
caller that pauses between actions
run [options] <flowFile> Run a flow on an interactive runner, shipping
the current directory's files with it
the flow and what it imports
events [options] <stream> Print a runner's journal, one entry per line.
QA Wolf writes console, recorder, run-events,
run-logs, run-status
Expand Down Expand Up @@ -342,8 +342,7 @@ Options:
exports[`--help output qawolf runner run 1`] = `
"Usage: qawolf runner run [options] <flowFile>

Run a flow on an interactive runner, shipping the current directory's files with
it
Run a flow on an interactive runner, shipping the flow and what it imports

Options:
--follow Report the run's status until it settles: in progress,
Expand Down
2 changes: 1 addition & 1 deletion src/commands/runner/run.register.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ export function registerRunCommand(
): void {
declareCommandKind(runner.command("run <flowFile>"), "write")
.description(
"Run a flow on an interactive runner, shipping the current directory's files with it",
"Run a flow on an interactive runner, shipping the flow and what it imports",
)
.option(
"--follow",
Expand Down
Loading