Conversation
…calculator inference (ff/calculators.py)
…xes 'got 2 and 3' crash on 3-atom cells, e.g. MATPES stress training)
- Example FF configs -> alignn_atomwise_pure + neighbor_strategy=pure_torch - test_prop/test_eprop/test_alignn_ff -> pure_torch configs - test_force_reduction: rewrite DGL primitives (radius_graph, SumPooling, update_all, reverse) as pure-torch scatter ops; both tests pass on CPU - README: note that DGL is no longer required
Found while training on MPtrj (1.53M structures) on 8 MI250X GCDs. All three are invisible to single-GPU runs. data.py: pass read_existing to the val and test get_torch_dataset calls. It was only forwarded for train, so val/test silently defaulted to False and always took the destructive rebuild path, deleting a prebuilt cache. Under DDP the ranks then race in shutil.rmtree and crash with FileNotFoundError. train.py: in the rank-0-only test evaluation, forward through _unwrap(net) instead of the DDP wrapper. A DDP forward performs a buffer-broadcast collective; running it on rank 0 alone deadlocks that rank until the NCCL watchdog fires at 600s while the other ranks finish and exit. pure_lmdb_dataset.py: open the LMDB environment lazily and drop it in __getstate__. Holding an lmdb.Environment on the dataset made it unpicklable, so any DataLoader(num_workers>0) failed with "cannot pickle 'Environment' object".
A run split across scheduler jobs (e.g. a multi-day run under a 24h walltime cap) previously restarted the LR schedule and optimizer state at every restart: --restart_model_path restores weights only. For a OneCycle schedule that means re-warming and re-annealing each segment, which hurts convergence. - config: resume_checkpoint (bool) opts into restoring optimizer, scheduler and epoch from <output_dir>/current_state.pt. - config: lr_total_epochs decouples the LR-schedule horizon from the per-job epoch count, so OneCycle spans the whole run and stays continuous across segments. None => use epochs (unchanged default). - train.py: write current_state.pt each epoch beside current_model.pt (kept as a pure state_dict for pretrained loaders); on resume restore optimizer/scheduler/best_loss and continue from the saved epoch. Verified on examples/sample_data: a 2-epoch run resumed cleanly to epoch 3 with a continuous schedule and monotonically improving val loss.
Training was limited to the GPUs of a single node: setup() hardcoded MASTER_ADDR="localhost" and world_size came from torch.cuda.device_count(), while mp.spawn cannot cross nodes. Several places also assumed the global rank equals the local GPU index, which breaks as soon as a second node joins. - train_alignn.py: when launched with srun/torchrun as one process per GCD (SLURM_NTASKS > 1), take the global rank from SLURM_PROCID and world size from SLURM_NTASKS instead of spawning. A single-task launch still uses the previous mp.spawn path unchanged. - train_alignn.py: resolve MASTER_ADDR from SLURM_NODELIST (localhost when the allocation is one node) and select the device by local rank. SLURM_LOCALID is only consulted in the multi-node case: under mp.spawn there is one task with LOCALID=0 but 8 processes, so the spawn index is the local one and trusting LOCALID would put every rank on GPU 0. - train.py: derive the training device and DDP device_ids from torch.cuda.current_device() rather than the global rank, which raised "HIP error: invalid device ordinal" on the second node and is also wrong under --gpu-bind=closest, where each rank sees a single GPU. Verified on 2 nodes x 8 MI250X GCDs (16 ranks): all ranks join, training and validation complete over multiple epochs, and single-node runs are unaffected.
…per-model recipes
- Add alignn/examples/recipes/{knn,radius,tensor,spectra,forcefield,atomwise}/,
each with a runnable config_example.json, make_toy_dataset.py (generates a toy
id_prop.json), and a detailed README. All six verified end-to-end on CPU.
- README Examples section now collapsible <details> per model type with quick-run
commands, plus a caution that toy datasets are smoke tests (use real data,
100-300 epochs, batch 32-64 for real models).
- Add 'Reproducing a JARVIS-Leaderboard contribution' section (git clone + install
alignn + jarvis_leaderboard, run a contribution's run.sh).
…ences, Notes; point JARVIS-Leaderboard to atomgptlab.github.io - Performances section now embeds the full 54-task benchmark table (mirrors the ALIGNN 2.0 manuscript) as a collapsible; dropped mlearn per-element rows except Si. - References and Useful-notes consolidated inline as collapsibles (no separate .md hop). - All JARVIS-Leaderboard links -> https://atomgptlab.github.io/jarvis_leaderboard/.
…8 smooth-cutoff, MATPES ep100) as default ALIGNN-FF
- Default ff_model matpes_smooth (was DGL v12.2.2024_dft_3d_307k, crashed dgl-free build)
- Property predictors now load pure-torch ALIGNN 2.0 models (pretrained2) instead of
DGL get_figshare_model; supports scalar, spectra (edos/pdos/ir/raman) and tensor
(elastic/dielectric/piezo/born) outputs
- Add prop_graph config ('radius' default, 'knn' switchable) + robust name resolver
(_prop2_name) handling direct names, {name}_{graph}, and target lookup with graph fallback
- Remove now-dead PROP_ALIASES/_resolve_prop/get_figshare_model import and unused
prop_cutoff/prop_max_neighbors fields
- Verified Si/Cu/NaCl: FF + formation/bandgap + edos(D=300) + elastic(D=36,C11=164) + dielectric(D=9,eps=12.9)
- tests/test_unified_calculator.py: FF+scalar+spectrum(edos D=300)+tensor(elastic D=36), matpes_smooth default, radius/knn name resolution, knn switch, unknown-property validation - docs/pretrained/property-predictor.md: replace dead DGL get_prediction example (crashes in the dgl-free build) with the pure-torch unified calculator / pretrained2 path - README: ASE-calculator caption now notes scalar+spectra+tensor, radius/knn (not scalar-only)
…spectra/tensor property predictors)
Use getattr(config.model, <field>, <default>) for graphwise_weight (1.0), atomwise_weight (0.0), gradwise_weight (0.0), stresswise_weight (0.0) and calculate_gradient (False), so plain property models (ALIGNNConfig, which lacks the FF loss-weight fields) train through the DGL loop without AttributeError. Behavior unchanged when the fields are present.
- Run black -l 79 across ff/, pretrained2.py, train.py, scripts (wrap long code lines) - Fix E231 in ff.py f-string cell indices ([0,0] -> [0, 0]) - CI Lint: ignore E501 in flake8/pycodestyle (black -l 79 is the line-length authority; long URL/description strings in the pretrained2 registry can't be wrapped). This is what made GitHub Actions fail (pycodestyle had no --exit-zero) — pre-existing. - lammps_bridge.py: --model-path now optional, defaults to default_path() (matpes_smooth); drop stray '# noqa' that was inside a LAMMPS command string - scripts/export_torchscript.py: --checkpoint/--config now optional, default to the default_path() model
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.