[haoxin1998]

Submit evaluation results for TimeRSI

Organization: GaTech & SamayLabs

TimeRSI uses Claude Code / Fable 5 to develop task-specific forecasting programs around frozen TimesFM 3.0, then selects and combines forecasts using historical backtests and feedback available before each forecast cutoff. TimesFM weights are not trained or fine-tuned.

On 100 FEV-Bench tasks (1,308 windows), SQL / MASE / WAPE / WQL Skill scores are 51.243% / 40.579% / 44.788% / 55.759%. The CSV reports research and forecast execution time separately; its end-to-end task-normalized median, including code search and candidate evaluation, is 2,511.713 seconds per 100 forecasts.

For review, we provide three things:

1. The program. The complete search and forecasting implementation is in the pip-installable TimeRSI4FevBench repository, including the research code and FEV task adapter. This PR contains only the thin FEV wrapper, dependency pin, results CSV and model metadata.
2. Execution environment: The public runner uses Docker to isolate code-agent sessions and generated candidates. Implementation.
3. Mini20 trajectories. We release the recorded primary agent trajectories for all 20 official mini tasks for inspection. Our review of the recorded calls found no direct retrieval of current or future evaluation targets or scores.

Installation and evaluation instructions.

[shchur]

Thanks for the contribution, and congrats on the strong results!

I have two concerns:

1. Possible test-set leakage

recipes.json.gz (and specialist_policy.json, group_policy.json, etc.) hard-codes decisions for each evaluation window, matched by exact start/end timestamps: program weights, correction choices, interval scales, and per-series/per-target candidate selection. Many are already non-default at window 0, before any evaluation window has finished (e.g., solar_1W uses program_weight=1.0 from the first window). The agent-written programs were also designed during the search and are applied to every window.

Since we only see the frozen output, we can't rule out that the agent saw test-window errors when designing pipelines or making these choices. For fev, every decision for window k must be based only on data available before that window's cutoff, e.g., backtesting on the historical context.

The runtime is also unclear. The PR description says the reported time includes the agent search, but the CSV shows training_time_s = 0 and an inference_time_s that looks like it only covers running the frozen recipes.

So we can't accept the submission in its current form. To move forward, we'd need:

• the full search code, so that the recipes can be regenerated from scratch;
• evidence that the agent never had access to test data or test scores, only to backtests on data before each cutoff;
• reported training and inference times that include the full search procedure.

2. Code organization

Most of this code shouldn't live in fev. All of the method code should go in https://github.com/haoxin1998/TimeRSI (or another pip-installable package), and the fev submission should only include a thin model.py wrapper that calls it, plus the results CSV and the models.yaml entry.

Happy to take another look once these are addressed.

[haoxin1998]

Thanks @shchur. We have updated the submission to address both concerns.

1. Possible test-set leakage and runtime

• Agent inputs: Agents receive observed history, declared known-future covariates, and historical backtest evidence available before each cutoff; current and future evaluation targets and scores are excluded. Historical backtests can inform a non-default decision even at window 0. Implementation.
• Execution environment: The public runner uses Docker to isolate code-agent sessions and generated candidates. Implementation.
• Inspection: We provide the complete recorded primary agent trajectories for all 20 official mini tasks—including prompts, candidate code, feedback and decisions—for review. Our inspection of the recorded calls found no direct retrieval of current or future evaluation targets or scores. Trajectories.
• Runtime: The CSV includes full search and candidate evaluation in training_time_s and inference_time_s, with an end-to-end median of 2,511.713 s / 100 forecasts. Accounting.

2. Code organization

The full search and forecasting implementation is available at https://github.com/haoxin1998/TimeRSI4FevBench. The FEV PR contains only a thin model.py wrapper, pinned requirements, the results CSV and the models.yaml entry.

We would appreciate another review.

[shchur]

Thank you, this addresses my concerns.

[haoxin1998]

Thanks for maintaining fev-bench @shchur