Skip to content

Regression tests: tags for C++ tests that need particular headers - #9407

Open
tautschnig wants to merge 1 commit into
diffblue:developfrom
tautschnig:pr/cpp-test-tags
Open

tautschnig wants to merge 1 commit into
diffblue:developfrom
tautschnig:pr/cpp-test-tags

Conversation

@tautschnig

Copy link
Copy Markdown
Collaborator

Tests of the C++ front end depend on the standard library headers cbmc preprocesses with (the host gcc's libstdc++, libc++, or the MSVC STL) and on GNU extensions or the Itanium ABI. regression/cbmc-cpp's CMake and Makefile now exclude, according to the host:

  • libcxx (run, as CORE only, in a profile of its own where clang and the libc++ headers are installed), msvc-stl (only with MSVC);
  • libstdcxx, libstdcxx12, libstdcxx13, libstdcxx16 (when the host gcc is not GNU gcc, or older than the version), broken-libstdcxx16 (with GCC 16 or later);
  • gnu-extensions and itanium-abi with MSVC; gcc-only also when the build compiler is not GNU (in regression/cpp, too). CORE runs in four shards with a per-test timeout of 900 s, and without --validate-goto-model with GCC < 13. regression/README.md documents the tags. No test uses them yet: they are for the C++ tests to follow.
  • Each commit message has a non-empty body, explaining why the change was made.
  • n/a Methods or procedures I have added are documented, following the guidelines provided in CODING_STANDARD.md.
  • n/a The feature or user visible behaviour I have added or modified has been documented in the User Guide in doc/cprover-manual/
  • Regression or unit tests are included, or existing tests cover the modified code (in this case I have detailed which ones those are in the commit message).
  • n/a My commit message includes data points confirming performance improvements (if claimed).
  • My PR is restricted to a single feature or bugfix.
  • n/a White-space or formatting changes outside the feature-related changed lines are in commits of their own.

Tests of the C++ front end depend on the standard library headers cbmc
preprocesses with (the host gcc's libstdc++, libc++, or the MSVC STL) and
on GNU extensions or the Itanium ABI.  regression/cbmc-cpp's CMake and
Makefile now exclude, according to the host:
- libcxx (run, as CORE only, in a profile of its own where clang and the
  libc++ headers are installed), msvc-stl (only with MSVC);
- libstdcxx, libstdcxx12, libstdcxx13, libstdcxx16 (when the host gcc is
  not GNU gcc, or older than the version), broken-libstdcxx16 (with
  GCC 16 or later);
- gnu-extensions and itanium-abi with MSVC; gcc-only also when the build
  compiler is not GNU (in regression/cpp, too).
CORE runs in four shards with a per-test timeout of 900 s, and without
--validate-goto-model with GCC < 13.  regression/README.md documents the
tags.  No test uses them yet: they are for the C++ tests to follow.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 11:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

tautschnig added a commit to tautschnig/cbmc that referenced this pull request Nov 10, 2026
Tests of the C++ front end (312 test directories in
regression/cbmc-cpp), as written for the reworked front end.  Each ends
with an assertion that must fail and requires `** 1 of N failed', so
that a test that does not type-check fails.  None of the 340 test
descriptions passes with the current front end, so all are KNOWNBUG;
they become CORE as the changes they test are merged.  Tests that need
particular headers or compilers carry the tags of regression/README.md.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
Depends-on: diffblue#9407
Depends-on: diffblue#9414
tautschnig added a commit to tautschnig/cbmc that referenced this pull request Nov 10, 2026
Tests of the C++ front end (81 test directories in regression/cbmc-cpp),
as written for the reworked front end.  Each ends with an assertion that
must fail and requires `** 1 of N failed', so that a test that does not
type-check fails.  None of the 83 test descriptions passes with the
current front end, so all are KNOWNBUG; they become CORE as the changes
they test are merged.  Tests that need particular headers or compilers
carry the tags of regression/README.md.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
Depends-on: diffblue#9407
Depends-on: diffblue#9414
tautschnig added a commit to tautschnig/cbmc that referenced this pull request Nov 10, 2026
Tests of the C++ front end (961 test directories in
regression/cbmc-cpp), as written for the reworked front end.  Each ends
with an assertion that must fail and requires `** 1 of N failed', so
that a test that does not type-check fails.  Of the 1044 test
descriptions, the 46 that pass with the current front end are CORE and
the other 998 are KNOWNBUG; they become CORE as the changes they test
are merged.  Tests that need particular headers or compilers carry the
tags of regression/README.md.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
Depends-on: diffblue#9407
Depends-on: diffblue#9414
tautschnig added a commit to tautschnig/cbmc that referenced this pull request Nov 10, 2026
Tests of the C++ front end (322 test directories in
regression/cbmc-cpp), as written for the reworked front end.  Each ends
with an assertion that must fail and requires `** 1 of N failed', so
that a test that does not type-check fails.  None of the 353 test
descriptions passes with the current front end, so all are KNOWNBUG;
they become CORE as the changes they test are merged.  Tests that need
particular headers or compilers carry the tags of regression/README.md.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
Depends-on: diffblue#9407
Depends-on: diffblue#9414
tautschnig added a commit to tautschnig/cbmc that referenced this pull request Nov 10, 2026
Tests of the C++ front end (297 test directories in regression/cbmc-cpp
and regression/cpp), as written for the reworked front end.  Each ends
with an assertion that must fail and requires `** 1 of N failed', so
that a test that does not type-check fails.  Of the 319 test
descriptions, the 38 that pass with the current front end are CORE and
the other 281 are KNOWNBUG; they become CORE as the changes they test
are merged.  Tests that need particular headers or compilers carry the
tags of regression/README.md.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
Depends-on: diffblue#9407
Depends-on: diffblue#9414
tautschnig added a commit to tautschnig/cbmc that referenced this pull request Nov 10, 2026
Tests of the C++ front end (78 test directories in regression/cbmc-cpp),
as written for the reworked front end.  Each ends with an assertion that
must fail and requires `** 1 of N failed', so that a test that does not
type-check fails.  None of the 80 test descriptions passes with the
current front end, so all are KNOWNBUG; they become CORE as the changes
they test are merged.  Tests that need particular headers or compilers
carry the tags of regression/README.md.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
Depends-on: diffblue#9407
Depends-on: diffblue#9414
@codecov

codecov Bot commented Nov 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.02%. Comparing base (826c1c7) to head (7afd09c).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9407      +/-   ##
===========================================
- Coverage    81.02%   81.02%   -0.01%     
===========================================
  Files         1723     1723              
  Lines       191314   191314              
  Branches        73       73              
===========================================
- Hits        155020   155019       -1     
- Misses       36294    36295       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

tautschnig added a commit to tautschnig/cbmc that referenced this pull request Nov 10, 2026
Tests of the C++ front end (955 test directories in
regression/cbmc-cpp), as written for the reworked front end.  Each ends
with an assertion that must fail and requires `** 1 of N failed', so
that a test that does not type-check fails.  Of the 1037 test
descriptions, the 39 that pass with the current front end are CORE and
the other 998 are KNOWNBUG; they become CORE as the changes they test
are merged.  Tests that need particular headers or compilers carry the
tags of regression/README.md.  Not included, although written for the
reworked front end too: cpp11_cstring_verify, cpp11_limits_verify,
cpp11_cv_qualified_member_overload_through_base,
cpp11_derived_to_base_class_parameter_is_conversion_rank,
cpp11_integral_promotion_target_of_same_width,
cpp11_reinterpret_cast_ref, as they pass with the current front end on
Linux with GCC 13 but fail on macOS, Windows, 32-bit or with GCC 14;
each comes with the change that makes it pass on every platform.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
Depends-on: diffblue#9407
Depends-on: diffblue#9414
tautschnig added a commit to tautschnig/cbmc that referenced this pull request Nov 10, 2026
Tests of the C++ front end (296 test directories in regression/cbmc-cpp
and regression/cpp), as written for the reworked front end.  Each ends
with an assertion that must fail and requires `** 1 of N failed', so
that a test that does not type-check fails.  Of the 318 test
descriptions, the 37 that pass with the current front end are CORE and
the other 281 are KNOWNBUG; they become CORE as the changes they test
are merged.  Tests that need particular headers or compilers carry the
tags of regression/README.md.  Not included, although written for the
reworked front end too: braced_mem_init1, as it fails with Visual Studio
(goto-cc in cl mode rejects the braced mem-initializer); each comes with
the change that makes it pass on every platform.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
Depends-on: diffblue#9407
Depends-on: diffblue#9414

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants