Testing QA

Testing / QA

make test-auth (fast, no hardware)

A self-contained authentication regression test: builds timpsd-sim with USE_CONTROL=1, starts it on unprivileged ports against scripts/test_auth.conf, runs scripts/test_auth.sh, then stops it and propagates the exit code. RTSP auth is fully exercised over loopback; the HTTP negative tests are skipped over loopback by design (the HTTP server trusts 127.0.0.0/8) — point scripts/test_auth.sh at a real device's LAN IP to exercise those. See Building.

scripts/timps-qa.sh — full QA harness against a real camera

timps-qa.sh is a ~1200-line bash harness that drives a real timps camera over the network (RTSP + HTTP + ONVIF), optionally with SSH for on-device checks, and produces a PASS/WARN/FAIL summary. It is the tool to run before/after any change that could affect stream stability, timing, or the /control API's live-apply behavior.

Running it

./scripts/timps-qa.sh --cam 192.168.1.100
./scripts/timps-qa.sh --cam 192.168.1.100 --ssh root@192.168.1.100 --profile standard
./scripts/timps-qa.sh --cam 192.168.1.100 --profile quick   # ~3 min smoke test
./scripts/timps-qa.sh --help

Every flag is also settable as an environment variable (CAM=..., LOAD_DUR=..., etc.).

Host-side prerequisites: ffmpeg + ffprobe + curl are required (checked in the Preflight section; a missing ffprobe aborts the stream tests). python3 is optional but needed for JSON parsing in the live-settings round-trip test (falls back to grep otherwise) and for the backchannel test scripts. openssl is used for ONVIF WS-Security digest auth (without it, ONVIF calls go out unauthenticated). --ssh expects key-based auth already configured (it runs with BatchMode=yes, so it fails rather than prompts for a password) — no key-setup step is scripted; that's a prerequisite the operator handles separately.

What it checks, section by section

1. Preflight — required tools present, optional SSH connectivity, camera reachability (ping is warn-only, not fail — it may be firewalled), and that the RTSP/HTTP TCP ports are actually open.
   • 1b Build identity — the version the camera reports in GET /control, next to the local checkout's git describe; and caps.framesource checked against general.fs_keepalive (an explicit 0/1 must resolve to itself, auto -1 to 0 or 1; a timpsd without the key is reported as info).
2. Discovery (ffprobe) — probes both the main (ch0) and sub (ch1) RTSP streams for codec/resolution/fps/audio, recording nominal fps for later comparison.
   • 2b Auth enforcement — both negative (unauthenticated/wrong-password requests must be rejected on /control, /events, /snapshot.jpg, /stream.mp4, /stream.mjpeg, and RTSP DESCRIBE) and positive (correct credentials must be accepted) checks, so "everything is blocked" can't false-pass the test.
   • 2c Backchannel handshake (default-on unless excluded via --only) — a real ONVIF backchannel handshake via scripts/bc-send.py (DESCRIBE+Require → SETUP trackID=2 → PLAY, then streams a short PCMU tone); a disabled/missing backchannel is reported as info, not a failure.
   • 2d Backchannel acoustic loopback — opt-in via --test-backchannel, never run by a profile by default: plays a tone into the speaker over the backchannel while simultaneously recording the camera's own outgoing mic audio over RTSP, then measures the tone-frequency band energy against a quiet control band. A ≥12dB delta passes; 5–12dB warns; below 5dB fails (explicitly flagged as possibly environmental rather than a code defect — mic/ speaker placement, acoustic isolation, etc.).
3. Stream integrity + A/V sync — records each stream for --integ-dur seconds and measures packet counts, real-time playback rate (catches stalls or wrong-clock fast-forward), max inter-packet gap (freeze detection), timestamp monotonicity, A/V drift measured after a warmup period (to exclude the startup transient), audio/video pace ratio (catches sample-rate mismatches), and counts ffmpeg decode/timestamp warnings. Compares measured fps against the nominal fps from Discovery. The main stream is additionally analysed once over UDP transport explicitly (skipped when the run is already --transport udp): the UDP send path has its own batching and its own orphaned-session reaping, and no SO_SNDTIMEO backstop is even possible there, so it needs its own pass rather than inheriting whatever --transport the run used.
4. HTTP fMP4 (/stream.mp4?chn=0) — the same integrity analysis as section 3, over HTTP instead of RTSP.
5. MJPEG (/stream.mjpeg?chn=0) — records and computes actual fps, fails on zero delivered frames.
6. Snapshot (/snapshot.jpg, both channels) — fetches multiple snapshots, validates HTTP 200 + JPEG magic bytes + minimum size, tracks latency.
7. Audio — codec/sample-rate/channel-count probe (with --expect-channels N as a hard assertion), plus a silence-detection scan flagging any gap longer than 1.5s.
8. /control API — basic read + safe write round-trip (reads image.brightness, POSTs the same value back, checks 200).
   • 8b Live settings apply + read-back verify — the most thorough section: for every live-applicable /control key (across image, audio, osd0.0, privacy.0.0, motion, daynight, record, timelapse), POSTs a new value, GETs it back, asserts the change took, restores the original, and confirms the revert — end-to-end proof of JSON-parse → config_apply_kv → HAL live-apply → reported state. Also does a persist-only sanity check on video.0.bitrate (must round-trip through config even though only a restart applies it). Rotation round-trip testing is opt-in via --test-rotation: gated on whatever caps.rotation this SoC's build actually advertises, testing each supported value and, for 90/270, verifying eff_width/eff_height swap correctly.
   • 8f Pixel-verified hflip + forced chn0 relatch — opt-in via --test-flip, needs a static scene: mirrors the hflip=0 snapshot and PSNR-compares it against the hflip=1 snapshot, so the flip is proven to change actual pixels rather than merely round-trip through /control. Then idles the pipeline past MS_IDLE_STOP_US, attaches a fresh client to force a chn0 re-latch, and re-checks that the flip survived. Ambiguous PSNR deltas warn rather than fail.
   • 8g Encoder settings verified after a restart — opt-in via --test-encoder, needs --ssh: first measures what the substream actually delivers under its current config, then cuts the bitrate target to ~0.4x of that measured rate (plus rc_mode), restarts the daemon for real, and measures again. Aiming downward matters: a bitrate target is a ceiling, so raising it proves nothing on a stream that is already quality- or content-limited (min_qp floor, static scene) — a lower ceiling always binds, which is what makes "the encoder followed it" and "the encoder ignored it" distinguishable. This is the only check that can catch a value being accepted, persisted and faithfully echoed while the running encoder ignores or coerces it. Also surfaces the previously-unused IMP_Encoder_Query telemetry (left_pics/work_done/ave_bitrate). Restores and restarts again.
   • 8h Day/night transition — opt-in via --test-daynight: sets mode=time with a window that contains the camera's current time, waits for the state to flip, asserts the board hook actually ran (logread, needs --ssh) and that image.running_mode followed, then inverts the window and asserts it comes back — counting switches in each direction to catch the overnight flap-loop regression. Physically clicks the IR-cut filter twice.
9. /events (SSE) — connects for 8 seconds and checks that some data streamed (an idle camera with no config changes may legitimately send nothing, so this warns rather than fails on silence).
10. ONVIF — skipped entirely if the ONVIF port isn't open. Checks the snapshot CGI proxies, GetSystemDateAndTime liveness, and GetProfiles (cross-checking advertised resolution/codec against the real stream, and flagging template-default frame-rate/bitrate limits that were never updated to match the real encoder).
11. On-demand recording clip — POSTs {"record":{"clip":...,"seconds":4}} and, with --ssh, verifies the resulting file exists and has a plausible size.
12. Reliability — reconnect churn — rapid connect/grab-frames/ disconnect cycles (both TCP and UDP RTSP transport), tracking success rate and time-to-first-frames.
    • 12b Session reaping — opt-in via --test-leak: starts a client, kill -9s it without a TEARDOWN, and requires the daemon's own subscriber count (from the /events stats frame) to return to baseline — within 150s for RTSP-UDP (2× the advertised session timeout) and 90s for HTTP fMP4 and SRT. Regression test for the immortal-session class of bug.
13. Load — concurrent client ramp — spawns increasing numbers of concurrent RTSP pulls of the main stream, tracking per-client and aggregate fps and (with --ssh) timpsd's RSS and system load average before/after, reporting the highest client count with zero failures. Steps above the compiled-in RTSP_MAX_CLIENTS cap (8) are reported as at cap — correctly enforced admission control — rather than as degradation. fd/thread counts are sampled at each step and a monotonic rise across the ramp is failed as a leak.
    • 13b Hostile (stalled) client — opt-in via --test-hostile, needs python3: runs healthy clients alongside one client (scripts/rtsp-stall.py) that completes the RTSP handshake over interleaved TCP and then stops reading — refreshing that session every 12 s, since the server reaps a peer after 15 s of zero write progress and only the first half of the phase would otherwise be hostile. Asserts the healthy clients keep the frame rate they measured in the baseline phase moments earlier — isolation is a differential question, so comparing against nominal fps instead would just re-report section 3's "this SoC does not sustain its configured fps" finding as a bogus isolation failure — that their keyframe rate does not spike (the global IDR-request rate limiter must stop one slow client from forcing keyframes on everybody) and that memory stays bounded.
14. Restart resilience — opt-in via --restart, requires --ssh: restarts the timps service and polls until the stream recovers, failing if it doesn't within 60s.
    • 14c Reboot persistence — opt-in via --test-reboot, needs --ssh: makes one persisted config change, reboots the camera for real, and then verifies the change survived, that /etc/timps.conf's and /usr/bin/timpsd's md5s and the reported version string are what they should be, and that nothing else in the whole /control document silently reset at boot.
15. Soak — runs when a soak duration is configured (auto-enabled by --profile soak, or via --soak-dur): repeatedly records short slices for the full duration, tallying decode/timestamp warnings and (with --ssh) sampling RSS, fd count, thread count and %CPU at each slice. RSS growth is warned on; a monotonic rise in fds or threads is failed (that is the per-session leak signature, and it costs kilobytes rather than megabytes, so an RSS threshold never sees it); a rising CPU trend at constant load is flagged too.
    • 15b Long-session A/V drift — opt-in via --drift-dur (or --profile drift, 2h): holds ONE unbroken RTSP connection open for the whole duration and measures A/V skew at --drift-seg checkpoints (default 5 min) using -reset_timestamps 0 so segment timestamps stay session-relative and comparable. Judged on the trend (first-quarter vs last-quarter average, plus how monotone the movement is), not on any single sample — a drift that creeps to 0.35s over two hours never trips a snapshot threshold but is exactly the failure signature of a stale-timestamp bug. The absolute skew level is reported separately.
16. On-device checks (SSH) — confirms the expected timps version and that timpsd is alive, scans logread for error-ish lines and separately for watchdog escalations (encoder dead, PollingStream idle, forced-recovery, no audio frames received, giving up on this channel — none of which match the generic error grep, and all of which mean the daemon is up and answering while producing nothing), scans dmesg (the kernel/driver log, a different surface from logread) for driver-level trouble — anchored to the sensor bring-up line (<sensor> stream on) so only messages logged after streaming started are scanned, because the boot preamble on these boards unconditionally contains benign lines carrying error/panic/watchdog (the kernel version banner embeds a build-time linker message, panic=2 is in the kernel command line, and the USB-OTG/SDIO probes always complain) — samples idle CPU/fd/thread counts with no clients attached, checks /etc/timps.conf for signs of a config-write bug (glued lines, duplicate keys), stress-tests 20 rapid concurrent /control POSTs toggling audio.agc (this specifically re-exercises a historical live-DSP-toggle use-after-free class that is now avoided by making agc/ns/high_pass restart-only — see Audio), and checks dmesg for segfault signatures.

CLI flags

Flag	Effect	Default
--cam IP	Camera address (required)	—
--profile quick|standard|load|soak	Preset tuning of durations/counts	standard
--rtsp-user/--rtsp-pass	RTSP credentials	thingino/thingino
--http-user/--http-pass	HTTP credentials	thingino/thingino
--expect-channels N	Hard-assert audio channel count	unset (report-only)
--main PATH / --sub PATH	RTSP stream paths	ch0 / ch1
--transport tcp|udp	RTSP transport for the integrity tests	tcp
--ssh TARGET	e.g. root@IP; enables all on-device checks	unset
--integ-dur S	Seconds recorded per integrity test	30 (10 on quick)
--load-dur S / --load-clients "N N ..."	Load-test tuning	30s / "1 2 4 8"
--reconnects N	Reconnect cycles per transport	20
--snaps N	Snapshot request count	30
--soak-dur S	Soak duration in seconds	0 (off unless soak profile)
--restart	Opt-in: exercise a full service restart	off
--drift-dur S / --drift-seg S	Long-session A/V drift window and checkpoint interval (section 15b)	0 (off) / 300
--only LIST	Comma-separated section names/numbers to run	all
--out DIR	Output directory	timps-qa-<timestamp>
--test-backchannel	Opt-in: acoustic loopback test (section 2d)	off, never in a profile
--bc-test-freq HZ / --bc-test-secs S	Backchannel test tone tuning	1500 / 4
--test-rotation	Opt-in: rotation round-trip test (part of 8b)	off, never in a profile
--test-flip	Opt-in: pixel-verified hflip + chn0 relatch (8f)	off, never in a profile
--test-encoder	Opt-in: measured bitrate/rc_mode after a restart (8g), needs --ssh	off, never in a profile
--test-daynight	Opt-in: forced day/night transition + flap check (8h)	off, never in a profile
--test-leak	Opt-in: reaping of SIGKILLed clients (12b)	off, never in a profile
--test-hostile	Opt-in: stalled-client isolation + IDR-storm check (13b)	off, never in a profile
--test-reboot	Opt-in: real reboot, config/binary persistence (14c), needs --ssh	off, never in a profile

Profiles: quick (~3 min smoke test), standard (~15 min, the default), load (~20 min, wider client-count sweep), soak (standard + a 2-hour soak run unless --soak-dur overrides), drift (standard + a 2-hour single-session A/V drift run unless --drift-dur overrides). soak and drift are deliberately separate: both take hours and run serially, so bundling them would silently double a soak run's wall time.

--profile openimp: FrontCrop, encoder ROI and OpenIMP keys

An opt-in section (scripts/timps-qa-openimp.sh, sourced by timps-qa.sh) that checks the picture, not just the POST reply. It never runs in an unfiltered run; select it with --profile openimp (preflight + this section only) or --only openimp (combinable, e.g. --only 1b,openimp):

scripts/timps-qa.sh --cam 192.168.10.163 --profile openimp --ssh root@192.168.10.163
scripts/timps-qa.sh --cam 192.168.10.163 --profile openimp --include-experimental

It reads the tiers from GET /control (caps.roi, caps.fcrop, caps.image, caps.framesource, caps.daynight) and decides per feature: effective = real tests (a missing effect is FAIL), experimental = SKIP unless --include-experimental (then a missing effect is INFO), unsupported = the POST must be refused (422 not_supported_on_soc) and the stream must keep running, sim (host sim) = round-trips only, missing caps (older timps) = SKIP.

Test	Pass condition
FrontCrop, every enabled stream (/snapshot.jpg?chn=S, else an RTSP frame)	crop off → before; centered 50 % window and top-left 50 % window (at least caps.fcrop.min_win) → after. Size unchanged, NCC(after, matching sub-window of before) ≥ 0.85 and ≥ 0.10 above NCC(after, whole before) and NCC(after, the other window). caps.fcrop.zoomable[S]=0 (T20 main) must keep the full picture. After crop off the full picture is back. caps.fcrop.state on/off; the first POST is re-sent once if it is not on, waiting is WARN and SKIPs the picture check. While these run an RTSP consumer is held on every zoomable stream: T10/T20/T21 refuse a window while the stream that would show it is idle. caps.fcrop.live=0 (T23): SKIP; --allow-restart checks the persist only.
Encoder ROI, streams with caps.roi.streams[S]="ok"	the 16-aligned box (~1/3 × 1/3) with the most texture; median Laplacian energy over 3 I-frames, inside/outside, ON vs OFF (rel). qp +D, D = min(20, qp_delta_eff max): rel ≤ 1 − 0.15·D/20; qp −15 (capped likewise): rel ≥ 1.05 (no effect = WARN only). No measurable effect is WARN, not FAIL, when |D| < 16 (the encoder clamps the table, T21 +13/−12) and videoS.rc_mode is not fixqp/vbr (those keep the bits in the region, so T21 under FixQP/VBR is graded), or, for blur, when videoS.rc_mode is cbr and rel is within 0.10 of the bar. An effect in the wrong direction (blur region sharper, rel ≥ 1.05; sharpen region blurrier, rel ≤ 0.85) is FAIL at any delta. A FAIL or a graded "no measurable effect" is measured once more on a fresh RTSP connection (new IDR; the first I-frame of every recording is dropped anyway, it can be the hub's cached IDR from before the POST): the second measurement decides and the line ends with (recheck: first rel X, second rel Y); both are in roi_metrics.txt. A wrong-direction FAIL is not cleared by one PASS (before the IDR the region reads as no effect, not as the wrong sign): FAIL then PASS is WARN (recheck disagrees: first rel X, second rel Y), FAIL twice stays FAIL. A first PASS and the WARN cases are not rechecked. No textured box found (vision helper error): SKIP. GET echo equal, caps.roi.rejected[S] unchanged.
OpenIMP-only image keys (colorfx scene dpc_strength defog_strength drc_strength sinter_strength temper_strength ae_compensation)	in caps.image: POST/GET/restore round-trip; otherwise one POST answers 422. brightness 64→192→64→192 and ae_compensation 40→230→40→230 (each value: 8 s wait for the AE dead time, then up to 40 s settle - the T31 AE needs 25-30 s for 128→255): effect = mean(high, high again) − mean(low, low again) ≥ 8 (WARN if smaller, FAIL if ≤ −8). SKIP (inconclusive) when the two low or the two high readings differ by more than max(8, |high − low|/3) or daynight.mode/night_baseline changed in between (day/night switch, IR probe). With --ssh the section first waits until timpsd has run OI_DN_BOOT_WAIT s (default 75): the daynight boot probe switches the IR off and flips day/night in the first ~45 s. colorfx 1: saturation ≤ 35 % (B/W leaves a few percent of chroma noise; a no-op mode stays near 100 %) (SKIP without colour); sinter/temper: temporal noise as INFO.
Rotation (restart-only, never changed)	if videoN.rotation is 90/270: eff_width/eff_height swapped and the stream size (ffprobe) equals them, or the start-up refusal as INFO.
With --ssh	timpsd --check-libs (libimp flavour), the daynight hr= (AE reserve) values from logread as INFO.

Safety: the touched image.* keys, image.fcrop_* and the ROI slot used are saved first and restored at the end and from the EXIT/INT/TERM trap, then read back and compared (a FAIL if they differ). Nothing restart-only is changed and the script never restarts timpsd. A target listed in QA_PROD_CAMS (space/comma list) or whose hostname (via --ssh) matches QA_PROD_RE (default prod) is refused without --allow-prod.

Limits: the scene must be static, lit and textured (a flat wall or night IR gives INCONCLUSIVE/SKIP, a moving scene spoils the ROI medians); AE is given OI_SETTLE_MAX seconds (default 8) per change (ae_compensation: 8 s + 40 s); the ROI sharpening check needs a base QP well above min_qp. Metrics: scripts/qa/openimp_vision.py (pure Python, no numpy; JPEG via ffmpeg), calibrated by python3 scripts/qa/test_openimp_vision.py (synthetic crops, blur/sharpen, and a real libx264 addroi encode). Against the sim (--only openimp) the round-trips run and the picture checks SKIP; OI_SIM_IMAGES=1 runs them anyway as INFO lines (plumbing only).

Output

Each run creates timps-qa-<YYYYMMDD-HHMMSS>/ (or the directory given by --out) at the repository root, containing summary.txt (the full colorized [PASS]/[WARN]/[FAIL] log) plus raw artifacts: ffprobe JSON dumps, recorded segments + ffmpeg logs for every integrity check, packet-timeline CSVs, /control JSON snapshots from the live-settings round trip, the /events capture log, ONVIF/snapshot JPEGs, per-load-step subdirectories, and reconnect/soak logs. The final summary line reports PASS=X WARN=Y FAIL=Z SKIP=W and an overall RESULT: verdict; the process exit code is 2 on any FAIL, 1 on any WARN with no FAIL, 0 on a clean pass — suitable for wiring into CI-style gating.

Related scripts: scripts/bc-send.py (backchannel tone sender, used by sections 2c/2d), scripts/rtsp-stall.py (the deliberately hostile never-reads RTSP client used by section 13b, also useful by hand), scripts/bc-talk.py (interactive backchannel testing, not invoked by timps-qa.sh itself), and scripts/test_auth.sh + scripts/test_auth.conf (the deeper fail-closed auth audit make test-auth runs, and referenced by section 2b as a companion for transport-level negative testing beyond what timps-qa.sh covers itself).