Skip to content

Add chakra_linter to validate Chakra execution traces against a rule set - #255

Closed
ajbalogh wants to merge 3 commits into
mlcommons:mainfrom
Keysight:feature-linter
Closed

ajbalogh wants to merge 3 commits into
mlcommons:mainfrom
Keysight:feature-linter

Conversation

@ajbalogh

@ajbalogh ajbalogh commented Nov 4, 2026

Copy link
Copy Markdown

Closes #254

Summary

  • Add chakra_linter, a standalone command-line tool that grades Chakra execution traces (.et files) for usability.
  • Grade each trace per consumer: replay, simulator, and analysis.
  • Report a specific reason for every gap, so trace authors and consumers can resolve it.
  • Run last in the toolchain (trace, link, convert, lint) as a quality gate before replay, simulation, or analysis.

Problem

  • Teams share traces without a standard check of what each trace supports.
  • Defects surface late, often as misleading results in a downstream tool.
  • Trace authors receive no structured feedback on what to correct.

Changes

  • Linter engine: src/linter/linter.py implements the check logic and the main entry point.
  • Rule set: src/linter/rules.toml declares each check (ID, name, description, enabled state, report message).
  • Rule schema: src/linter/rules.schema.json validates the syntax of rules.toml.
  • Documentation: src/linter/README.md covers purpose, methodology, rule configuration, and usage.
  • Packaging: pyproject.toml registers the chakra_linter script, the chakra.src.linter package, and the rule files as package data.
  • Dev dependencies: requirements-dev.txt adds jsonschema==4.21.1 and markdown==3.11.2.
  • Tests: four new test modules under tests/linter/ cover converter attributes, CPU replay checks, rule inventory, and rules.toml validation.

Grading Model

  • Tier 0 (integrity): decode failure, empty trace, schema violation, or invalid DAG makes the trace unusable for every consumer.
  • Evidence levels: each check group receives a level; the worst gap sets the level.
  • Tier 1 (consumer usability): gaps reduce usability for the affected consumers and never invalidate a trace.

Usage

chakra_linter traces/ --report-output report.md
  • Accepts .et files or directories.
  • Produces HTML and markdown reports.

Testing

  • ruff format and ruff check pass
  • pytest tests/linter passes
  • Linter runs against a sample trace directory and produces both report formats

Notes for Reviewers

  • linter.py, rules.toml, and rules.schema.json form one contract. Review them together.
  • The change adds no modifications to existing converter, generator, or trace-link behavior.

@ajbalogh ajbalogh self-assigned this Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@ajbalogh
ajbalogh marked this pull request as draft November 5, 2026 13:47
Comment thread src/linter/linter.py
ZERO_PARTIAL_RATIO = 0.05

# Node types whose durations must not all be zero
CHECKED_TYPES = {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about the new data_op and storage_op types? maybe add a note in #222 to ensure updating the linter is an exit criteria as kaitracer is introduced?

Comment thread src/linter/linter.py

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it make sense to break up this single file of thousands of lines into more logical files (e.g. key data classes that may require touch as future extension to schemas in separate files??) and the linter is just the main entry point that is essentially the cli that ties the parts together? otherwise, this file will just keep growing as we go forward.

Comment thread src/linter/README.md

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest in addition to the end user flow in the readme, to also add an additional section for developers who will be touching the code, on what parts they may need to touch (think the use case of future schema extensions)...

@ajbalogh

ajbalogh commented Nov 5, 2026

Copy link
Copy Markdown
Author

Closing this PR without merging.

Reason

  • The refactoring changed the linter structure, including the split into rule-family modules.
  • These changes supersede earlier commits on this branch.
  • Reviews and comments on the earlier commits no longer map to the current code.

Benefit of the refactoring

  • The rule-family structure makes the linter easier for developers to extend.
  • A developer can add or modify rules within one family module without changing the linter core.

Status

  • The branch feature-linter is no longer needed and is retired.
  • No further commits will land on this PR.
  • A new branch will carry the refactored work with a clean commit history.
  • One new PR will follow from that branch and reference this PR.

Reviewer action

  • No further review is required on this PR.
  • Review will resume on the new PR once it opens.

@ajbalogh ajbalogh closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add chakra_linter to validate Chakra execution traces against a rule set

2 participants