Clarify how to get a real benchmark run from Benchmark Comparison

[Copilot]

Description

Benchmark Comparison could complete in --dry-run mode without telling users how to trigger a real benchmark. This change adds explicit guidance in both the workflow summary and generated comparison artifact so dry-run output points to the required secrets and rerun path.

Related Issues

Addresses confusion around benchmark runs reporting dry_run without actionable next steps for running the full benchmark workload.

Proposed Changes

• Workflow summary

  • Extend the dry-run branch in benchmark-comparison.yml to state which secrets gate a full benchmark run.
  • Point users to Actions → Benchmark Comparison → Run workflow once those secrets are configured.

• Generated benchmark report

  • Add a dry-run notice to scripts/benchmark_smoke.py when any result has status == "dry_run".
  • Include the exact secrets required for a full run:
    • OPENAI_API_KEY
    • GRAPHRAG_API_BASE
    • AZURE_AI_SEARCH_URL_ENDPOINT
    • AZURE_AI_SEARCH_API_KEY

• Focused coverage

  • Add a unit test that verifies dry-run reports include the new “how to get a real benchmark run” guidance.

> This comparison used `--dry-run`, so it validated commands without executing the real benchmark workload.
> To get a real benchmark run in GitHub Actions, configure these secrets and rerun the `Benchmark Comparison` workflow:
> `OPENAI_API_KEY`, `GRAPHRAG_API_BASE`, `AZURE_AI_SEARCH_URL_ENDPOINT`, and `AZURE_AI_SEARCH_API_KEY`.

Checklist

• I have tested these changes locally.
• I have reviewed the code changes.
• I have updated the documentation (if necessary).
• I have added appropriate unit tests (if applicable).

Additional Notes

This is intentionally a minimal UX/documentation change: benchmark execution behavior is unchanged. The workflow still falls back to --dry-run when secrets are unavailable; it now explains how to get out of that mode.

---

⌨️ Start Copilot coding agent tasks without leaving your editor — available in VS Code, Visual Studio, JetBrains IDEs and Eclipse.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[Copilot]

Pull request overview

This PR improves the UX around Benchmark Comparison runs by making --dry-run output explicitly explain how to trigger a full benchmark run (i.e., which secrets are required and where to rerun the workflow).

Changes:

• Updates the Benchmark Comparison workflow step summary to list required secrets and the manual rerun path when falling back to --dry-run.
• Enhances scripts/benchmark_smoke.py to emit a prominent dry-run notice in the generated markdown report whenever any result is status == "dry_run".
• Adds a unit test ensuring the report includes the new “how to get a real benchmark run” guidance.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
tests/unit/test_benchmark_smoke.py	Adds unit coverage asserting dry-run reports include the new guidance text.
scripts/benchmark_smoke.py	Adds dry-run detection and injects a prominent guidance block into the markdown report.
.github/workflows/benchmark-comparison.yml	Improves workflow summary messaging when secrets are missing and the run is forced into --dry-run.

[Copilot]

has_dry_run_results return statement is likely to fail ruff format --check due to line wrapping (the generator expression exceeds the formatter's line length). Reformat this line (e.g., split the any(...) call across multiple lines) or run ruff format so CI formatting checks pass.

Suggested change

	return any(result.status == "dry_run" for results in result_groups for result in results)
	return any(
	result.status == "dry_run"
	for results in result_groups
	for result in results
	)