fix(megatron_bridge): save TensorBoard logs by default#968
Open
blugassi wants to merge 2 commits into
Open
Conversation
blugassi
requested review from
jeffnvidia,
podkidyshev and
srivatsankrishnan
as code owners
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe changes add a default TensorBoard directory to Megatron Bridge SLURM commands, preserve custom overrides, test generated launcher output, and strengthen ChangesMegatron Bridge TensorBoard configuration
Training parser validation
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
blugassi
force-pushed
the
fix/megatron-bridge-tensorboard-dir
branch
from
3bf8daa to
548fec1
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/cloudai/report_generator/training/parser.py`:
- Around line 79-80: Update the directory validation in can_handle_directory()
to require at least one actual TensorBoard event file matching the
events.out.tfevents* pattern, rather than accepting any non-empty directory.
Preserve the existing warning and skip behavior when no readable event files are
present.
- Around line 82-84: Update can_parse() to validate the config artifact’s
contents, not only config_path.is_file(), before returning True; ensure
malformed or empty JSON/YAML causes the method to return False. Alternatively,
update get_model_config() failures used by parse() to be caught and converted
into a False parseability result, preserving the existing skip behavior for
invalid training reports.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: f1cdc005-6e8b-4dad-8e74-acf4ba06050f
📒 Files selected for processing (4)
src/cloudai/report_generator/training/parser.pysrc/cloudai/workloads/megatron_bridge/slurm_command_gen_strategy.pytests/ref_data/megatron-bridge.sbatchtests/workloads/megatron_bridge/test_command_gen_strategy_slurm.py
blugassi
force-pushed
the
fix/megatron-bridge-tensorboard-dir
branch
from
548fec1 to
3ce59b5
Compare
Set the default log directory to the saved run output so training reports can consume TensorBoard metrics while preserving user overrides. Signed-off-by: Ben Lugassi <blugassi@nvidia.com>
blugassi
force-pushed
the
fix/megatron-bridge-tensorboard-dir
branch
from
3ce59b5 to
4c73044
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/cloudai/report_generator/training/parser.py`:
- Around line 80-100: Optionally optimize can_parse and parse by caching the
validated dictionary returned by get_model_config for JSON/YAML artifacts and
reusing it through _resolve_model_config, avoiding a second disk parse per run
while preserving existing validation behavior.
In `@tests/report_generator/training/test_training_parser.py`:
- Around line 485-522: Update the outer parameter container in the
pytest.mark.parametrize call for test_can_parse_rejects_invalid_file_config from
a tuple to a list, while keeping each individual pytest.param row and test
behavior unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: d7a5889e-6e0f-4a7f-a63f-87b029042d37
📒 Files selected for processing (5)
src/cloudai/report_generator/training/parser.pysrc/cloudai/workloads/megatron_bridge/slurm_command_gen_strategy.pytests/ref_data/megatron-bridge.sbatchtests/report_generator/training/test_training_parser.pytests/workloads/megatron_bridge/test_command_gen_strategy_slurm.py
Reject missing or invalid training artifacts before parsing and log why report generation is skipped. Signed-off-by: Ben Lugassi <blugassi@nvidia.com>
blugassi
force-pushed
the
fix/megatron-bridge-tensorboard-dir
branch
from
4c73044 to
ab02db2
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/report_generator/training/test_training_parser.py`:
- Around line 475-482: Update the test setup around parser.can_parse to place
custom.tfevents.log inside a nested directory under tb_dir, while keeping the
existing non-empty-directory and configuration assertions unchanged. This
ensures recursive TensorBoard event discovery is required.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: d7ca2872-70ba-418d-954d-5e3996d76d51
📒 Files selected for processing (2)
src/cloudai/report_generator/training/parser.pytests/report_generator/training/test_training_parser.py
Contributor
Author
|
@coderabbitai resume |
Contributor
✅ Action performedReviews resumed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Set Megatron-Bridge’s default TensorBoard directory to /nemo_run/tb_logs. Improve training-report checks to reject missing, empty, or malformed artifacts with clear logs. User overrides remain supported.
Test Plan
Tested on Linux with Python 3.12 and pytest 9.1:
python -m pytest tests/report_generator/training/test_training_parser.py
tests/workloads/megatron_bridge/test_command_gen_strategy_slurm.py
tests/workloads/megatron_run/test_megatron_run.py
'tests/test_acceptance.py::test_sbatch_generation[megatron-bridge]'
Result: 79 tests passed.
Additional Notes
Existing logger.tensorboard_dir overrides are unchanged.