Simple Implementation for Transformer with C and Python

Implement Tranformer with C and Python for educational purpose. Code is written for readability.

Build

cd lib/sml
make
cd ..

cd src/c
make

Run

C version

cd src/c
./transformer

Python version

cd src/python

python ./experiments/classify.py  --random-seed=1234 --num-epochs=1 --tiny

TODO:

• main Makefile for build library and c executable
• add config load
• add trained weight load
• add python code to generate test vector, and use that to test the C code

Reference:

• SML: small math library, http://www.bios.unc.edu/distrib/bios235/sml/
• Transformer tutorial: http://jalammar.github.io/illustrated-transformer/
• Python Transformer implementation: http://peterbloem.nl/blog/transformers
• tranformer implementation: https://github.com/tunz/transformer-pytorch