Skip to content

Commit

Permalink
added comment
Browse files Browse the repository at this point in the history
  • Loading branch information
Sagi Polaczek committed Sep 18, 2024
1 parent c60edb3 commit 7ccb6d0
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions fuse/data/tokenizers/modular_tokenizer/op.py
Original file line number Diff line number Diff line change
Expand Up @@ -524,6 +524,12 @@ def from_pretrained(
local_files_only: bool = False,
revision: Optional[str] = None,
) -> "ModularTokenizerOp":
"""Load pre-trained tokenizer from HF repo_id or a local dirpath.
Args:
identifier (str): repo_id or local dirpath.
* For other args see `snapshot_download()`
"""
if not os.path.isdir(identifier):
# Try to download from hub
try:
Expand Down

0 comments on commit 7ccb6d0

Please sign in to comment.