Skip to content

Commit

Permalink
Apply suggestions from code review
Browse files Browse the repository at this point in the history
Co-authored-by: Dalton Bohning <[email protected]>
Signed-off-by: enakta <[email protected]>
  • Loading branch information
enakta and daltonbohning authored Jan 7, 2025
1 parent e72b63d commit f3eafd1
Show file tree
Hide file tree
Showing 3 changed files with 8 additions and 8 deletions.
10 changes: 5 additions & 5 deletions docs/user/pytorch.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@

PyTorch is fully featured framework for building deep learning models and training them.
It is widely used in the research community and in the industry.
PyTroch allows loading data from various sources and DAOS can be used as a storage backend for training data and model's checkpoints.
PyTorch allows loading data from various sources and DAOS can be used as a storage backend for training data and models' checkpoints.

[DFS plugin](https://github.com/daos-stack/daos/tree/master/src/client/pydaos/torch) implements PyTroch interfaces for loading data from DAOS: Map and Iterable style datasets.
This allows to use all features of `torch.utils.data.DataLoader` to load data from DAOS POSIX container, including parallel data loading, batching, shuffling, etc.
[DFS plugin](https://github.com/daos-stack/daos/tree/master/src/client/pydaos/torch) implements PyTorch interfaces for loading data from DAOS: Map and Iterable style datasets.
This allows to use all features of `torch.utils.data.DataLoader` to load data from DAOS POSIX containers, including parallel data loading, batching, shuffling, etc.

## Installation

To install the plugin, you need to have PyTorch installed, please follow official [PyTorch installation guide](https://pytorch.org/get-started/).
`pydoas.torch` module comes with DAOS client package, please refer to DAOS installation guide for your distribution.
To install the plugin, you need to have PyTorch installed. Please follow the official [PyTorch installation guide](https://pytorch.org/get-started/).
`pydoas.torch` module comes with DAOS client package. Please refer to DAOS installation guide for your distribution.


## Usage
Expand Down
2 changes: 1 addition & 1 deletion src/client/pydaos/torch/Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ plt.show()
Torch framework provides a way to save and load model's checkpoints: `torch.save` and `torch.load` functions are used to save and load the model state dictionary.
The `torch.save` function expects a state dictionary object and a file like object `Union[str, PathLike, BinaryIO, IO[bytes]]`.
To implement such interface, `pydaos.torch.WriteBuffer` class is introduced, which is a wrapper around `io.BufferedIOBase` object, behaving like a writable stream.
It accomulates the data in the buffer and writes it to the DAOS container when the close method is called.
It accumulates the data in the buffer and writes it to the DAOS container when the close method is called.
Implementation of the loader is pretty straightforward - it reads the data from the file with existing API and returns it as a buffer.

For convenience, the `pydoas.torch.Checkpoint` class is provided that manages the DAOS connections and provides `reader` and `writer` methods.
Expand Down
4 changes: 2 additions & 2 deletions src/tests/ftest/pytorch/map_dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ def test_map_style_dataset(self):
:avocado: tags=all,full_regression
:avocado: tags=vm
:avocado: tags=pytorch
:avocado: tags=dfuse,pytorch
:avocado: tags=PytorchMapStyleDatasetTest,test_map_style_dataset
"""
pool = self.get_pool()
Expand Down Expand Up @@ -83,7 +83,7 @@ def test_dataloader(self):
:avocado: tags=all,full_regression
:avocado: tags=vm
:avocado: tags=pytorch
:avocado: tags=dfuse,pytorch
:avocado: tags=PytorchMapStyleDatasetTest,test_dataloader
"""
pool = self.get_pool()
Expand Down

0 comments on commit f3eafd1

Please sign in to comment.