Skip to content
IanCal edited this page Sep 13, 2010 · 6 revisions

Info:

The project is an implementation of Restricted Boltzmann Machines on the GPU, specifically in CUDA.

Acronym-tastic.

It started as a research project, to see if it was a decent idea and to explore the various variables. Turns out, yeah, it is. On an 8600GT I was getting training rates of about 1k images per second on the MNIST dataset and a network size of 784-512-512-2048 including all file reading and transfer times, which is 1 minute per epoch. Recent papers put the training time as several hours per layer for a standard implementation. As work progresses onto larger networks (for such things as fMRI, or perhaps multi-scale/attribute images) it seems pretty useful, the main limitation being the amount of ram on the card.

Current Status: Basic functionality. File readers and parameter updaters working. Learning rates for each layer and weight decays are working. Now uses on-gpu random number generator (a simple LCG). Current speed >1k/s mnist images (including loading from a file & all transfer times) on an 8600GT.

Uses waf and nvcc to compile. Not the best solution, but it’s a single line command. Compiles to a shared library.

You will be able to:

- Train multiple layers at once. Simply set the learning rates (individual ones for each layer), the system will only process the data as far as it needs to, and will update any weights when the learning rate is not 0. – DONE

- Learn as many labels as you like. Multiple layers labeled? No problem. [Done] Some softmax, some not? Why not! Want to change from one to the other mid-learning? Go for it.

- Modify learning rates, momentums and weight decays in any way, at every iteration. [Done, but not momentum yet]

- Read and write matlab files for easy experiment running, using this implementation for the fast greedy training, then using the weights in a matlab program for whatever further work you wish.

- Have enormous files. This one is really quite simple actually, just using mmap. If there’s a better way, then just implement your own file reader, it’s really quite simple.

If you have any questions, or want to try and compile it, then feel free to drop me an email and I’ll help out best I can.

Clone this wiki locally