Repository navigation
Add native Test Coverage UI support via Ceedling's gcov plugin - #13
jake-kim-sndk wants to merge 1 commit into
Conversation
The Testing view's Run with Coverage button now runs the suite through Ceedling's gcov: task and reports the result through VS Code's own coverage UI - the Test Coverage panel, per-file percentages, and inline line and branch hit counts. The data was already there. Ceedling's gcov plugin writes a gcovr Cobertura report with per-line hit counts, per-branch condition coverage and, from newer gcovr, per-method entries; nothing consumed it. This wires that report to the coverage API VS Code finalized in 1.88. - src/gcovrCoberturaReport.ts: pure, vscode-free normalization of the parsed report, unit-tested in plain Node like ceedlingXmlReport.ts. Every path comes out absolute. gcovr writes <filename> relative to <source> inside its root but absolute outside it, and before gcovr 8.3 <source> is itself relative to where gcovr ran (gcovr#1037). Uri.file() puts a relative path at the filesystem root, so a path left unresolved shows no coverage at all. - ceedlingEngine: a coverage variant of the run entry points, and reading the report against the project directory Ceedling runs in. A missing or unreadable report is "no coverage", not a failed run. - testController: a non-default Coverage run profile, since a coverage run rebuilds instrumented and is slower. Each project's previous report is deleted before the run, so a failed build can't pass the last run's coverage off as this one's. The fixtures are real reports from Ceedling 1.1.6 with gcovr 5.0 and 8.6, of a probe project with one source inside the Ceedling project and one above it. Requires raising the minimum VS Code from 1.71.0 to 1.88.0, the release that finalized the test coverage API.
|
@jake-kim-sndk This is excellent. Thank you so much for your work here. I have a separate PR #12 that implements some further under-the-hood refactoring with test coverage, integrates support for mixins, and it replaces custom duplicate YAML handling with Ceedling's own. The refactoring is towards some big new features to come. I was planning to finish this up coincident with Ceedling 1.2.0 (coming out very soon). Anyhow, what I propose is that we merge your PR into #12 with some necessary rework, and bundle this all up to be published when Ceedling 1.2.0 is published. I'd like to also add support for the |
|
Thanks for your consideration. |
Summary
This adds a Run with Coverage profile to the Testing view. It runs the tests through Ceedling's
gcov:task and shows the result in VS Code's built-in Test Coverage UI: the Test Coverage panel, per-file percentages, and line and branch hit counts in the editor.Ceedling's gcov plugin already writes a gcovr Cobertura report with per-line hit counts, branch coverage and, with newer gcovr, per-function entries, but nothing was reading it. This connects that report to the test coverage API that VS Code finalized in 1.88.
Since 2.0 already moved to the native Testing API with
createTestController, adding aTestRunProfileKind.Coverageprofile felt like the natural next step. I'm happy to change anything so it fits where you want to take the extension.What changed
src/gcovrCoberturaReport.tsvscode. It's split out likeceedlingXmlReport.tsfor the same reason:vscodecan't be loaded under plain mocha, so a parser insideceedlingEngine.tscouldn't be unit tested.tests/unit/gcovrCoberturaReport.test.tsfor (const version of ...)loop thatceedlingXmlReport.test.tsuses.tests/fixtures/gcovr-5.0/,tests/fixtures/gcovr-8.6/ceedling-1.0.0/is.src/ceedlingEngine.tssrc/testController.tsCoveragerun profile (not the default one) andloadDetailedCoverage.package.jsonceedlingExplorer.coverageCommandArgssetting, which defaults to["gcov:${TEST_ID}"], and the engine bump.A few things were deliberate:
coverageCommandArgsis a separate setting rather than a flag ontestCommandArgs. Thegcov:task isn't a drop-in replacement fortest:because it needs the gcov plugin enabled, so if someone has customized one, the other shouldn't quietly change with it.deleteXmlReportalready guards against the same thing for test results. The delete happens once per project rather than once per entry, so a later entry that fails to build doesn't throw away the report an earlier entry of the same run produced.Why the minimum VS Code version goes from 1.71 to 1.88
TestRunProfileKind.Coverage,FileCoverage,StatementCoverage,BranchCoverageandTestRun.addCoveragewere all finalized in 1.88 (March 2024). There's no way to do this on the stable API before that, and there isn't a useful fallback either, since without the API there's nowhere to show the coverage.As far as I can tell,
^1.71.0was carried over rather than chosen on purpose. It came in with f94c897 (November 2023, a CI and security fix) and stayed the same through the 2.0 rework. Anyone on a version older than 1.88 keeps getting 2.0.1 from the Marketplace, which doesn't have coverage anyway, so they don't lose anything. That said, it's your call. If you'd rather hold this until a release where you're raising the minimum anyway, just let me know.Paths
This is the part I'd look at most closely, because any mistake here fails the same way: no coverage shows up and nothing reports an error. There were three things to get right.
<filename>relative to<source>for files inside its root and as an absolute path for files outside it. Ceedling projects often have both.<source>is itself relative. It's just., which means wherever gcovr ran. gcovr lists this as a known bug (#1037) that was fixed in 8.3.Uri.file()doesn't resolve relative paths, soUri.file('src/foo.c')becomesfile:///src/foo.c.So the parser is given the directory Ceedling ran in (
absPath, which is thecwdinexecCeedling) and usespath.resolve(baseDir, source, filename), which handles every combination.path.joindoesn't, becausepath.join('.', '/abs/foo.c')silently drops the leading slash. I first ran into that in another coverage extension that does exactly this. It's the sameabsPathresolution thatrunInternalalready does for compiler diagnostics, for the sameUri.file()reason.Testing
npm run buildis clean andnpm run test:unitpasses 69 tests (53 before, 16 new).src/Calculator.cfromtests/manualplus alib/Clamp.cone level above the project, once with gcovr 5.0 and once with 8.6. Paths are anonymized to/home/dev/...the same way theceedling-1.xfixtures are.path.joinlogic back, exactly one test fails: gcovr 5.0 with a file inside the root.tests/manualnow resolves to its realsrc/Calculator.c. Before the fix it came out as/src/Calculator.c.Function rows only show up when gcovr fills in
<methods>. gcovr 5.0 leaves it empty while 8.6 fills it in, and the 8.6 fixture covers how those entries are read.Notes
docs/ReleaseNotes.mdalone because it reads like your own write-up and I don't know which version this would go into.CHANGELOG.mdanddocs/BreakingChanges.mdhave[Unreleased]entries you can fold in.src/testController.tshas an existing TS6138 error (isDevelopmentModeis declared but never read). It's onmastertoo, so I didn't touch it here.