Model Configurations

overview

	flamingo 3B	flamingo-mini (ours)	flamingo-tiny (ours)
language model	chinchilla	OPT-350m	OPT-125m
# params	1.4B
# layers	24
# heads	16
embedding size	2048
number of tokens	32000	50256	50256
vision encoder	NFNet-F6	ViT-L	ViT-L
output shape	---	257 x 1024	257 x 1024
resampler
# params
# heads	16
# layers	6	6	6
hidden size	1536
KV size	128
# latents	64	64	64
activation function	sq. ReLU	sq. ReLU	sq. ReLU
xattn dense
# params
# heads	16
# layers	24
hidden size	2048
KV size	128
activation function	sq. ReLU	sq. ReLU	sq. ReLU

flamingo-mini

TODO

flamingo-tiny

TODO