TFG: Language Models on Question Answering problems

Author: Enrique Saiz Oubiña
Advisor: Manuel Antonio Sánchez-Montañés Isla
Date: June 2024

---

• Repositories
• Resources
  • Courses
  • Documentation
  • Utils
  • Tokenizers
  • Training and hyperparameters
  • Evaluation
  • More

---

All the notebooks in this repository can be run in Google Colab session with T4 GPU.

Repositories

• TFG HuggingFace Collection (datasets and models)
• Github Repository (training and evaluation code)

Resources

Collection of the resources i found useful during the elaboration of my work. These are different from the main document bibliography. There is something to learn from each; i recommend to take a look.

Courses

• HuggingFace NLP
• Large Language Model Course

Documentation

• Question Answering
• PEFT ParameterEfficient FineTuning
• Intro to BitsAndBytes and Quantization
• BitsAndBytes: Quantization and matrix multiplication
• Phi-2 finetuning. Adapters
• Generation with LLMs: strategies and pitfalls

Utils

• QA utils and scripts
• Model memory estimations
• MRQA 2019 Shared Task
• Models Database

Tokenizers

• Left padding in autoregressive models
• Phi-2 finetuning and EOS token

Training and hyperparameters

• Performance and scalability: gradient acc, FP16, ...
• About training in half precision (BF16 & FP16)
• Half-precision in T5
• About batch sizes (Nvidia)
• About learning rate schedulers
• Methods and tools for efficient training on a single GPU
• Finetuning on consumer hardware (PyTorch)
• Large research of training parameters (StableDiffusion)

Evaluation

• Evaluation considerations
• The compute_metrics function
• Further evaluation: BERT Score
• Further evaluation: Semantic Answer Similarity
• Further evaluation: Eval Harness

More

• The illustrated transformer
• Some motivation
• SQuAD 2 and the Null Response
• CO2 Impact calculator
• AI yearly report

---