RFC-0043 Torch Native Runtime - #72
Open
zhxchen17 wants to merge 2 commits into
Open
Conversation
Author
2 tasks
gmagogsfm
reviewed
Apr 24, 2025
|
|
||
| ### “Any performance regression from TorchScript?” | ||
|
|
||
| * Single digit percentage (<10%) improvement over TorchScript interpreter from improvements like static dispatch, memory planning, inter-op parallelism and overhead reduction. |
There was a problem hiding this comment.
Suggested change
| * Single digit percentage (<10%) improvement over TorchScript interpreter from improvements like static dispatch, memory planning, inter-op parallelism and overhead reduction. | |
| No, on the contrary we expect * single digit percentage (<10%) improvement over TorchScript interpreter from improvements like static dispatch, memory planning, inter-op parallelism and overhead reduction. |
yiming0416
added a commit
to yiming0416/pytorch
that referenced
this pull request
Apr 30, 2025
Summary: Pull Request resolved: pytorch#152475 Torch Native Runtime RFC: pytorch/rfcs#72 Thi diff moves `TensorMeta.cpp` and `TensorMeta.h` to PyTorch core. Test Plan: ``` buck2 run mode/dev-nosan caffe2/test/cpp/nativert:nativert ``` Differential Revision: D73820548
yiming0416
added a commit
to yiming0416/pytorch
that referenced
this pull request
May 1, 2025
Summary: Pull Request resolved: pytorch#152475 Torch Native Runtime RFC: pytorch/rfcs#72 Thi diff moves `TensorMeta.cpp` and `TensorMeta.h` to PyTorch core. Test Plan: Internal CI. GitHub CI with newly added test under `test/cpp/nativert/test_tensor_meta.cpp` Differential Revision: D73820548
yiming0416
added a commit
to yiming0416/pytorch
that referenced
this pull request
May 2, 2025
Summary: Pull Request resolved: pytorch#152475 Torch Native Runtime RFC: pytorch/rfcs#72 Thi diff moves `TensorMeta.cpp` and `TensorMeta.h` to PyTorch core. Test Plan: Internal CI. GitHub CI with newly added test under `test/cpp/nativert/test_tensor_meta.cpp` Differential Revision: D73820548
yiming0416
added a commit
to yiming0416/pytorch
that referenced
this pull request
May 2, 2025
Summary: Pull Request resolved: pytorch#152475 Torch Native Runtime RFC: pytorch/rfcs#72 Thi diff moves `TensorMeta.cpp` and `TensorMeta.h` to PyTorch core. Test Plan: Internal CI. GitHub CI with newly added test under `test/cpp/nativert/test_tensor_meta.cpp` Differential Revision: D73820548
yiming0416
added a commit
to yiming0416/pytorch
that referenced
this pull request
May 3, 2025
Summary: Pull Request resolved: pytorch#152475 Torch Native Runtime RFC: pytorch/rfcs#72 Thi diff moves `TensorMeta.cpp` and `TensorMeta.h` to PyTorch core. Test Plan: Internal CI. GitHub CI with newly added test under `test/cpp/nativert/test_tensor_meta.cpp` Differential Revision: D73820548
yiming0416
added a commit
to yiming0416/pytorch
that referenced
this pull request
May 5, 2025
Summary: Pull Request resolved: pytorch#152475 Torch Native Runtime RFC: pytorch/rfcs#72 Thi diff moves `TensorMeta.cpp` and `TensorMeta.h` to PyTorch core. Test Plan: Internal CI. GitHub CI with newly added test under `test/cpp/nativert/test_tensor_meta.cpp` Reviewed By: zhxchen17 Differential Revision: D73820548
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this pull request
May 6, 2025
Summary: Torch Native Runtime RFC: pytorch/rfcs#72 This diff moves `TensorMeta.cpp` and `TensorMeta.h` to PyTorch core under `torch/nativert/graph/` Existing `torch::_export::TensorMeta` in `torch/csrc/utils/generated_serialization_types.h` is auto-generated from the export serde schema and therefore only containing the most basic serializable types. We need the newly added `TensorMeta.cpp` to deserialize the metadata into a in-memory class with c10 types so that it can be consumed by the runtime later. Test Plan: Added test under `test/cpp/nativert/test_tensor_meta.cpp` Differential Revision: D73820548 Pull Request resolved: #152475 Approved by: https://github.com/albanD
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this pull request
May 7, 2025
Summary: Torch Native Runtime RFC: pytorch/rfcs#72 Port string utils functions join and split to c10/util Test Plan: Added tests in `string_util_test.cpp` buck2 run mode/opt caffe2/c10/test:util_base_tests Differential Revision: D74202473 Pull Request resolved: #152873 Approved by: https://github.com/cyyever, https://github.com/Skylion007
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this pull request
May 20, 2025
Summary: Torch Native Runtime RFC: pytorch/rfcs#72 Added an in-memory representation for input and output specs of a graph. The GraphSignature class models the input and output specs of an exported graph produced by torch.export, which holds the graph information deserialized from the pt2 archive package. Runtime relies on the GraphSignature for weight name lookup and weight loading. The serialization schema is defined in torch/_export/serde/schema.py See more at: https://docs.pytorch.org/docs/stable/export.html#torch.export.ExportGraphSignature Test Plan: Added tests under `test/cpp/nativert/test_graph_signature.cpp` Differential Revision: D73895378 Pull Request resolved: #152969 Approved by: https://github.com/swolchok
yiming0416
added a commit
to pytorch/pytorch
that referenced
this pull request
May 28, 2025
We split the large PR for adding Graph.h and Graph.cpp to `nativert` into 3 smaller PRs: 1. **Add header file** 2. Add source file 3. Add test and build rules. Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75491860](https://our.internmc.facebook.com/intern/diff/D75491860/) [ghstack-poisoned]
yiming0416
added a commit
to pytorch/pytorch
that referenced
this pull request
Jun 3, 2025
We split the large PR for adding Graph.h and Graph.cpp to nativert into 3 smaller PRs: 1. Add header file 2. **Add source file** 3. Add test and build rules. Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75492405](https://our.internmc.facebook.com/intern/diff/D75492405/) [ghstack-poisoned]
yiming0416
added a commit
to pytorch/pytorch
that referenced
this pull request
Jun 3, 2025
… nativert" We split the large PR for adding Graph.h and Graph.cpp to nativert into 3 smaller PRs: 1. Add header file 2. Add source file 3. **Add test and build rules** Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75495273](https://our.internmc.facebook.com/intern/diff/D75495273/) [ghstack-poisoned]
yiming0416
added a commit
to pytorch/pytorch
that referenced
this pull request
Jun 3, 2025
We split the large PR for adding Graph.h and Graph.cpp to nativert into 3 smaller PRs: 1. Add header file 2. Add source file 3. **Add test and build rules** Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75495273](https://our.internmc.facebook.com/intern/diff/D75495273/) [ghstack-poisoned]
yiming0416
added a commit
to pytorch/pytorch
that referenced
this pull request
Jun 3, 2025
We split the large PR for adding Graph.h and Graph.cpp to `nativert` into 3 smaller PRs: 1. **Add header file** 2. Add source file 3. Add test and build rules. Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75491860](https://our.internmc.facebook.com/intern/diff/D75491860/) [ghstack-poisoned]
yiming0416
added a commit
to pytorch/pytorch
that referenced
this pull request
Jun 3, 2025
We split the large PR for adding Graph.h and Graph.cpp to nativert into 3 smaller PRs: 1. Add header file 2. **Add source file** 3. Add test and build rules. Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75492405](https://our.internmc.facebook.com/intern/diff/D75492405/) [ghstack-poisoned]
yiming0416
added a commit
to pytorch/pytorch
that referenced
this pull request
Jun 3, 2025
We split the large PR for adding Graph.h and Graph.cpp to `nativert` into 3 smaller PRs: 1. **Add header file** 2. Add source file 3. Add test and build rules. Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75491860](https://our.internmc.facebook.com/intern/diff/D75491860/) [ghstack-poisoned]
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this pull request
Jun 3, 2025
We split the large PR for adding Graph.h and Graph.cpp to `nativert` into 3 smaller PRs: 1. **Add header file** 2. Add source file 3. Add test and build rules. Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75491860](https://our.internmc.facebook.com/intern/diff/D75491860/) Pull Request resolved: #154530 Approved by: https://github.com/SherlockNoMad
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this pull request
Jun 3, 2025
We split the large PR for adding Graph.h and Graph.cpp to nativert into 3 smaller PRs: 1. Add header file 2. **Add source file** 3. Add test and build rules. Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75492405](https://our.internmc.facebook.com/intern/diff/D75492405/) Pull Request resolved: #154531 Approved by: https://github.com/SherlockNoMad ghstack dependencies: #154530
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this pull request
Jun 3, 2025
We split the large PR for adding Graph.h and Graph.cpp to nativert into 3 smaller PRs: 1. Add header file 2. Add source file 3. **Add test and build rules** Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75495273](https://our.internmc.facebook.com/intern/diff/D75495273/) Pull Request resolved: #154532 Approved by: https://github.com/SherlockNoMad ghstack dependencies: #154530, #154531
iupaikov-amd
pushed a commit
to ROCm/pytorch
that referenced
this pull request
Jun 4, 2025
We split the large PR for adding Graph.h and Graph.cpp to `nativert` into 3 smaller PRs: 1. **Add header file** 2. Add source file 3. Add test and build rules. Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75491860](https://our.internmc.facebook.com/intern/diff/D75491860/) Pull Request resolved: pytorch#154530 Approved by: https://github.com/SherlockNoMad
iupaikov-amd
pushed a commit
to ROCm/pytorch
that referenced
this pull request
Jun 4, 2025
We split the large PR for adding Graph.h and Graph.cpp to nativert into 3 smaller PRs: 1. Add header file 2. **Add source file** 3. Add test and build rules. Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75492405](https://our.internmc.facebook.com/intern/diff/D75492405/) Pull Request resolved: pytorch#154531 Approved by: https://github.com/SherlockNoMad ghstack dependencies: pytorch#154530
iupaikov-amd
pushed a commit
to ROCm/pytorch
that referenced
this pull request
Jun 4, 2025
We split the large PR for adding Graph.h and Graph.cpp to nativert into 3 smaller PRs: 1. Add header file 2. Add source file 3. **Add test and build rules** Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75495273](https://our.internmc.facebook.com/intern/diff/D75495273/) Pull Request resolved: pytorch#154532 Approved by: https://github.com/SherlockNoMad ghstack dependencies: pytorch#154530, pytorch#154531
angelayi
pushed a commit
to angelayi/pytorch
that referenced
this pull request
Jun 5, 2025
We split the large PR for adding Graph.h and Graph.cpp to `nativert` into 3 smaller PRs: 1. **Add header file** 2. Add source file 3. Add test and build rules. Torch Native Runtime RFC: pytorch/rfcs#72 4 classes have been introduced: `Graph`, `Node`, `Value`, `Type` - `Type` represents the kind of a `Value` - `Value` represents a single symbolic value, it could be any kind that exists in `Type`. Values are inputs and outputs of a `Node`. - `Node` represents a single unit of execution, typically a PyTorch op. - `Graph` represents a model's computation graph, which is designed to facilitate transformation/analysis. Differential Revision: [D75491860](https://our.internmc.facebook.com/intern/diff/D75491860/) Pull Request resolved: pytorch#154530 Approved by: https://github.com/SherlockNoMad
This was referenced Jun 10, 2025
This was referenced Jun 17, 2025
|
Hi, thanks for this RFC! We're currently evaluating NativeRT for our production inference pipeline, especially the inter-op parallelism support via I've opened an issue here with some detailed questions: Would really appreciate any guidance. Thanks! |
|
Hi @zhxchen17! Thank you for your pull request. We require contributors to sign our Contributor License Agreement, and yours needs attention. You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.