Code

Integrate the audio modality in CoCa#94

Code

Draft

manasMauryax wants to merge 129 commits intomainModalities/modalities:mainfrom feat/audio_cocaModalities/modalities:feat/audio_cocaCopy head branch name to clipboard

+4,076-228

Commits

Commits on May 7, 2024

• feat: add basic webdataset

  spravil

  committed
  d909ae0

• fix: dim of cls token

  spravil

  committed
  e233676

• feat: simple console logging

  spravil

  committed
  9986691

• fix: add attention mask to cross entropy loss

  spravil

  committed
  c47b6c1

• feat: allow multiple loss functions

  spravil

  committed
  70823e1

• fix: register nce loss

  spravil

  committed
  0c87d91

• feat: add dataloader for webdataset

  spravil

  committed
  b652a7d

• chore: add config

  spravil

  committed
  b0e933a

• feat: add nicer logging to wandb

  spravil

  committed
  d8d5a5f

• fix: hardcoded batches in web loader

  spravil

  committed
  4ea65c8

• chore: update coca config

  spravil

  committed
  dfe88c9

• fix: rebase

  spravil

  committed
  5a3e844

Commits on May 9, 2024

• fix: print only on main rank in component factory

  spravil

  committed
  043384d

• fix: total loss average logging

  spravil

  committed
  3cd9244

• fix: cuda env and run script

  spravil

  committed
  b09d20e

• chore: update coca config

  spravil

  committed
  e09745d

• fix: print parameters and done only on main rank

  spravil

  committed
  5a74dee

• chore: update coca wds config

  spravil

  committed
  f7b725c

• fix: tokenizer config of coca

  spravil

  committed
  c7308e2

• fix: add multinode splitter to webdataset

  spravil

  committed
  32d0b19

• fix: webdataset slow loading

  spravil

  committed
  55c039f

• fix: add batching

  spravil

  committed
  966d237

• fix: add more options to webloader

  spravil

  committed
  dacf639

• fix: webloader

  spravil

  committed
  b40ecd5

• fix: dataset factory

  spravil

  committed
  a9ce132

• fix: webdataset

  spravil

  committed
  63ef47c

Commits on May 10, 2024

• fix: loss accumulation

  spravil

  committed
  d6d84dc

• fix: loss average for eval

  spravil

  committed
  3d04f78

• refactor: remove unused code from coca collator

  spravil

  committed
  e65a3cd

• fix: coca collator

  spravil

  committed
  622570d

• fix: loss normalization

  spravil

  committed
  efedc77

• feat: add clip loss

  spravil

  committed
  eba23a9

• fix: use clip loss in coca

  spravil

  committed
  9bdc830

• chore: update coca webdataset config

  spravil

  committed
  b3aff91

• chore: update coca webdataset config

  spravil

  committed
  dffe644

Commits on May 13, 2024

• fix: gradient accumulation

  spravil

  committed
  d8bdebc

• fix: normalize cls token of coca

  spravil

  committed
  244fac9

• feat: add weight option to loss config

  spravil

  committed
  0524acd

• fix: log weighted loss

  spravil

  committed
  df48b62

• feat: add cosine scheduler with warmup

  spravil

  committed
  2945a03

• fix: loss logging

  spravil

  committed
  2d0e3f3

• feat: add local clip loss

  spravil

  committed
  1c0993f

• fix: clip loss

  spravil

  committed
  0e3b239

• fix: add barrier to eval

  spravil

  committed
  7459ee0

• feat: print global batch size

  spravil

  committed
  1c0f456

• fix: force integer for rank env variable

  spravil

  committed
  a04bf97

• fix: validation set loading

  spravil

  committed
  9086af6

• fix: webdataset splitter

  spravil

  committed
  351990a

• feat: add drop_last to webloader

  spravil

  committed
  c90df9d

• fix: val dataset with webdataset

  spravil

  committed
  860b8fd

• refactor: evaluator

  spravil

  committed
  962c4ca

• fix: nodesplitter in webdataset

  spravil

  committed
  2d1ea92

• feat: add wandb grouping

  spravil

  committed
  1fb4fba

• chore: update coca config

  spravil

  committed
  22c77cd

• fix: coca config

  spravil

  committed
  6d980e7

Commits on Jun 10, 2024

• feat: add universal multimodal dataset

  spravil

  committed
  dfb7884

• feat: add flatten_dict function

  spravil

  committed
  da27f0d

• fix: webdataset

  spravil

  committed
  016bed3

Commits on Jun 11, 2024

• feat: add conformer audio encoder

  

  manasMauryax

  authored and

  mmaurya

  committed
  587c52b

• feat: make CoCa audio compatible

  

  manasMauryax

  authored and

  mmaurya

  committed
  1ba15bc

• test: change config and dummy dataset for E2E CoCa test

  

  manasMauryax

  authored and

  mmaurya

  committed
  d71cbdb

• feat: add new Dataset class for Audio/Vision

  

  mmaurya

  committed
  e824400

• fix: incorrect variable usage and audio input shape

  

  mmaurya

  committed
  fcaccea

• fix: to avoid torch.tensor(tensor)

  

  mmaurya

  committed
  5dd535e

• fix: add argument to ignore padding indices

  

  mmaurya

  committed
  302d396

• test: uptate tests to comply with changes

  

  mmaurya

  committed
  94c85ce

• feat: add entrypoint for audio coca tokenizer

  

  mmaurya

  committed
  ceaa897

• chore: add configs

  Show description for b3b7f77
  

  mmaurya

  committed
  b3b7f77

• chore: update pyproject.toml

  

  mmaurya

  committed
  85c0c69

• feat: allow masking of "pad" keys

  

  mmaurya

  committed
  e4c956c

• feat: implement Conformer from scratch

  

  mmaurya

  committed
  94e042b

• test: fix to comply to changes

  

  mmaurya

  committed
  c6c6f30

• test: remove deprecated test

  

  mmaurya

  committed
  03db6ec

• feat: improve arrow dataset

  

  mmaurya

  committed
  9704c09

• feat: add audio-only and vision-only arrow dataset

  

  mmaurya

  committed
  7c5e063

• chore: fix configs to comply to changes

  

  mmaurya

  committed
  a554b82

• chore: add script which generates tokenizer

  Show description for 75a0ebe
  

  mmaurya

  committed
  75a0ebe

• fix: accelerate import

  

  thomaschhh

  authored and

  mmaurya

  committed
  50add61

• refactor: introduce global constants

  

  thomaschhh

  authored and

  mmaurya

  committed
  433ac9d

• fix: constant renaming

  

  thomaschhh

  authored and

  mmaurya

  committed
  aa50864

• fix: web dataset integration

  spravil

  committed
  faeec41

• chore: add todo statement

  spravil

  committed
  1c6fcff

• chore: merge remote-tracking branch 'origin/main' into exp/vision_language_coca

  spravil

  committed
  e90a8d5

• chore: update start script

  spravil

  committed
  42f469b

• chore: merge remote-tracking branch 'origin/main' into exp/vision_language_coca

  spravil

  committed
  03082b8

• fix: coca webdataset config

  spravil

  committed
  78efe14

• feat: extend vision transformer model to video data

  

  SogolHaghighat

  authored and

  spravil

  committed
  deb0788

• test: extend vision transformer test to video data

  

  SogolHaghighat

  authored and

  spravil

  committed
  4a9695d

• test: add and update config for testing vision transformer with image and video

  

  SogolHaghighat

  authored and

  spravil

  committed
  f04243a

• feat: add video transforms

  spravil

  committed
  cad8d11

• chore: add video config

  spravil

  committed
  3653922

• fix: video coca

  spravil

  committed
  3b72cd2

• chore: add audio config

  thomaschhh

  committed
  357c76e

• feat: add audio transform

  thomaschhh

  committed
  2a4f45d

• chore: remove get_web_dataset

  thomaschhh

  committed
  ae71bb7

• fix: collator

  thomaschhh

  committed
  b04652a

• feat: add multimodal transforms

  thomaschhh

  committed
  137d4f2

• chore: adjust forward call

  thomaschhh

  committed
  d93e6e5

• fix: get feat_lens for webdataset

  thomaschhh

  committed
  9528bb6

Commits on Jun 12, 2024

• feat: add conformer audio encoder

  

  manasMauryax

  authored and

  mmaurya

  committed
  0c0317a

• feat: make CoCa audio compatible

  

  manasMauryax

  authored and

  mmaurya

  committed
  5f63246

• test: change config and dummy dataset for E2E CoCa test

  

  manasMauryax

  authored and

  mmaurya

  committed
  857867e

• feat: add new Dataset class for Audio/Vision

  

  mmaurya

  committed
  c9ad825

• fix: incorrect variable usage and audio input shape

  

  mmaurya

  committed
  97ecb20

• fix: to avoid torch.tensor(tensor)

  

  mmaurya

  committed
  4463ae7

• fix: add argument to ignore padding indices

  

  mmaurya

  committed
  b410073

• test: uptate tests to comply with changes

  

  mmaurya

  committed
  6c0ced6

• feat: add entrypoint for audio coca tokenizer

  

  mmaurya

  committed
  e0577b7

• chore: add configs

  Show description for f40f7f6
  

  mmaurya

  committed
  f40f7f6

• chore: update pyproject.toml

  

  mmaurya

  committed
  8bed2c2

• feat: allow masking of "pad" keys

  

  mmaurya

  committed
  743d999

• feat: implement Conformer from scratch

  

  mmaurya

  committed
  b2cb27c

• test: fix to comply to changes

  

  mmaurya

  committed
  7a23ced

• test: remove deprecated test

  

  mmaurya

  committed
  363701c

• feat: improve arrow dataset

  

  mmaurya

  committed
  d403c62

• feat: add audio-only and vision-only arrow dataset

  

  mmaurya

  committed
  bc0d9f0

• chore: fix configs to comply to changes

  

  mmaurya

  committed
  bae2f0f

• chore: add script which generates tokenizer

  Show description for 0cf2290
  

  mmaurya

  committed
  0cf2290

• fix: accelerate import

  

  thomaschhh

  authored and

  mmaurya

  committed
  f871403

• refactor: introduce global constants

  

  thomaschhh

  authored and

  mmaurya

  committed
  c9f58b9

• fix: constant renaming

  

  thomaschhh

  authored and

  mmaurya

  committed
  1806062

• fix: disable mamba imports

  

  mmaurya

  committed
  f8181d0

• chore: update audio coca arrow dataset config

  

  mmaurya

  committed
  25803ca

Commits on Jun 13, 2024

• chore: Merge branch 'feat/audio_coca' of https://github.com/Modalities/modalities into feat/audio_coca

  thomaschhh

  committed
  b27bf78

• feat: add audio transform

  thomaschhh

  committed
  cca934e

• fix: cross entropy loss ignore index

  thomaschhh

  committed
  bcc7e9d

• fix: prepare_sample

  thomaschhh

  committed
  e78b2ea

• chore: apply changes from origin/feat/audio_coca

  thomaschhh

  committed
  8d5f459

• chore: fix configs to comply with changes

  thomaschhh

  committed
  162c965