Code

Tokenization at scale#287

Code

Draft

le1nux wants to merge 10 commits intomainModalities/modalities:mainfrom tokenization_at_scaleModalities/modalities:tokenization_at_scaleCopy head branch name to clipboard

+1,329-642

Commits

Commits on Jan 2, 2025

• feat: added enhanced logging

  le1nux

  committed
  bbee65c

• refactor: scaled the index creation and tokenization by supporting multprocessing

  le1nux

  committed
  90f5d5a

Commits on Jan 5, 2025

• refactor: finalized processing strategies

  le1nux

  committed
  e876c97

• feat: introduced process controller

  le1nux

  committed
  3fe7690

• feat: added custom process implementation

  le1nux

  committed
  3d79cc4

• feat: added interface for the processing strategies

  le1nux

  committed
  8f41a74

• refactor: bug fixing in TokenizationInstantiationModel

  le1nux

  committed
  a05bd9f

• refactor: API uses now process controller

  le1nux

  committed
  ac8f74e

Commits on Jan 8, 2025

• feat: added multiple queue destinations for strategy

  le1nux

  committed
  95f30d4

• feat: temp env var decorator

  le1nux

  committed
  d5d59a7