Skip to content

Add native Test Coverage UI support via Ceedling's gcov plugin - #13

Closed
jake-kim-sndk wants to merge 1 commit into
ThrowTheSwitch:masterfrom
jake-kim-sndk:feat/native-test-coverage
Closed

jake-kim-sndk wants to merge 1 commit into
ThrowTheSwitch:masterfrom
jake-kim-sndk:feat/native-test-coverage

Conversation

@jake-kim-sndk

@jake-kim-sndk jake-kim-sndk commented Sep 29, 2026 •

Copy link
Copy Markdown

Summary

This adds a Run with Coverage profile to the Testing view. It runs the tests through Ceedling's gcov: task and shows the result in VS Code's built-in Test Coverage UI: the Test Coverage panel, per-file percentages, and line and branch hit counts in the editor.

Ceedling's gcov plugin already writes a gcovr Cobertura report with per-line hit counts, branch coverage and, with newer gcovr, per-function entries, but nothing was reading it. This connects that report to the test coverage API that VS Code finalized in 1.88.

Since 2.0 already moved to the native Testing API with createTestController, adding a TestRunProfileKind.Coverage profile felt like the natural next step. I'm happy to change anything so it fits where you want to take the extension.

image

What changed

File What it does
src/gcovrCoberturaReport.ts New. Turns the parsed report into plain data without depending on vscode. It's split out like ceedlingXmlReport.ts for the same reason: vscode can't be loaded under plain mocha, so a parser inside ceedlingEngine.ts couldn't be unit tested.
tests/unit/gcovrCoberturaReport.test.ts New. Runs both fixtures through the same for (const version of ...) loop that ceedlingXmlReport.test.ts uses.
tests/fixtures/gcovr-5.0/, tests/fixtures/gcovr-8.6/ New. Real reports rather than hand-written ones (see Testing), named after the gcovr version the way ceedling-1.0.0/ is.
src/ceedlingEngine.ts Coverage versions of the run entry points, plus reading the report and deleting a stale one.
src/testController.ts A Coverage run profile (not the default one) and loadDetailedCoverage.
package.json The new ceedlingExplorer.coverageCommandArgs setting, which defaults to ["gcov:${TEST_ID}"], and the engine bump.

A few things were deliberate:

  • Coverage has its own profile and isn't the default. A coverage run rebuilds with instrumentation and is noticeably slower (about 20% on my suite, 13.4s vs 11.2s), so it sits behind its own button instead of slowing down every run.
  • coverageCommandArgs is a separate setting rather than a flag on testCommandArgs. The gcov: task isn't a drop-in replacement for test: because it needs the gcov plugin enabled, so if someone has customized one, the other shouldn't quietly change with it.
  • A coverage run deletes the previous report before it starts. If the build fails, gcovr never runs, and the old report would otherwise be shown as if this run had produced it. deleteXmlReport already guards against the same thing for test results. The delete happens once per project rather than once per entry, so a later entry that fails to build doesn't throw away the report an earlier entry of the same run produced.

Why the minimum VS Code version goes from 1.71 to 1.88

TestRunProfileKind.Coverage, FileCoverage, StatementCoverage, BranchCoverage and TestRun.addCoverage were all finalized in 1.88 (March 2024). There's no way to do this on the stable API before that, and there isn't a useful fallback either, since without the API there's nowhere to show the coverage.

As far as I can tell, ^1.71.0 was carried over rather than chosen on purpose. It came in with f94c897 (November 2023, a CI and security fix) and stayed the same through the 2.0 rework. Anyone on a version older than 1.88 keeps getting 2.0.1 from the Marketplace, which doesn't have coverage anyway, so they don't lose anything. That said, it's your call. If you'd rather hold this until a release where you're raising the minimum anyway, just let me know.

Paths

This is the part I'd look at most closely, because any mistake here fails the same way: no coverage shows up and nothing reports an error. There were three things to get right.

  1. gcovr writes <filename> relative to <source> for files inside its root and as an absolute path for files outside it. Ceedling projects often have both.
  2. Before gcovr 8.3, <source> is itself relative. It's just ., which means wherever gcovr ran. gcovr lists this as a known bug (#1037) that was fixed in 8.3.
  3. Uri.file() doesn't resolve relative paths, so Uri.file('src/foo.c') becomes file:///src/foo.c.

So the parser is given the directory Ceedling ran in (absPath, which is the cwd in execCeedling) and uses path.resolve(baseDir, source, filename), which handles every combination. path.join doesn't, because path.join('.', '/abs/foo.c') silently drops the leading slash. I first ran into that in another coverage extension that does exactly this. It's the same absPath resolution that runInternal already does for compiler diagnostics, for the same Uri.file() reason.

Testing

  • npm run build is clean and npm run test:unit passes 69 tests (53 before, 16 new).
  • The fixtures are real reports from Ceedling 1.1.6's gcov plugin. I ran it on a small probe project, using src/Calculator.c from tests/manual plus a lib/Clamp.c one level above the project, once with gcovr 5.0 and once with 8.6. Paths are anonymized to /home/dev/... the same way the ceedling-1.x fixtures are.
  • If I put the old path.join logic back, exactly one test fails: gcovr 5.0 with a file inside the root.
  • With the gcov plugin enabled, tests/manual now resolves to its real src/Calculator.c. Before the fix it came out as /src/Calculator.c.
  • I also installed a packaged VSIX and used it on a real firmware project (gcovr 5.0, with the sources above the Ceedling project). Line and branch coverage show up in the editor and in the Test Coverage panel.

Function rows only show up when gcovr fills in <methods>. gcovr 5.0 leaves it empty while 8.6 fills it in, and the 8.6 fixture covers how those entries are read.

Notes

  • I left docs/ReleaseNotes.md alone because it reads like your own write-up and I don't know which version this would go into. CHANGELOG.md and docs/BreakingChanges.md have [Unreleased] entries you can fold in.
  • src/testController.ts has an existing TS6138 error (isDevelopmentMode is declared but never read). It's on master too, so I didn't touch it here.

The Testing view's Run with Coverage button now runs the suite through
Ceedling's gcov: task and reports the result through VS Code's own
coverage UI - the Test Coverage panel, per-file percentages, and inline
line and branch hit counts.

The data was already there. Ceedling's gcov plugin writes a gcovr
Cobertura report with per-line hit counts, per-branch condition coverage
and, from newer gcovr, per-method entries; nothing consumed it. This
wires that report to the coverage API VS Code finalized in 1.88.

- src/gcovrCoberturaReport.ts: pure, vscode-free normalization of the
  parsed report, unit-tested in plain Node like ceedlingXmlReport.ts.
  Every path comes out absolute. gcovr writes <filename> relative to
  <source> inside its root but absolute outside it, and before gcovr
  8.3 <source> is itself relative to where gcovr ran (gcovr#1037).
  Uri.file() puts a relative path at the filesystem root, so a path
  left unresolved shows no coverage at all.
- ceedlingEngine: a coverage variant of the run entry points, and
  reading the report against the project directory Ceedling runs in.
  A missing or unreadable report is "no coverage", not a failed run.
- testController: a non-default Coverage run profile, since a coverage
  run rebuilds instrumented and is slower. Each project's previous
  report is deleted before the run, so a failed build can't pass the
  last run's coverage off as this one's.

The fixtures are real reports from Ceedling 1.1.6 with gcovr 5.0 and
8.6, of a probe project with one source inside the Ceedling project and
one above it.

Requires raising the minimum VS Code from 1.71.0 to 1.88.0, the release
that finalized the test coverage API.
@mkarlesky

mkarlesky commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

@jake-kim-sndk This is excellent. Thank you so much for your work here.

I have a separate PR #12 that implements some further under-the-hood refactoring with test coverage, integrates support for mixins, and it replaces custom duplicate YAML handling with Ceedling's own. The refactoring is towards some big new features to come. I was planning to finish this up coincident with Ceedling 1.2.0 (coming out very soon).

Anyhow, what I propose is that we merge your PR into #12 with some necessary rework, and bundle this all up to be published when Ceedling 1.2.0 is published. I'd like to also add support for the ReportGenerator option to complement gcovr as it also can produce a Cobertura report (and it's fairly popular in Windows-centric development).

@jake-kim-sndk

Copy link
Copy Markdown
Author

Thanks for your consideration.
I'll retarget destination to #12 and resolve necessary conflicts. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants