New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Donating System Memory? #586

Open

NavodPeiris opened this issue Jun 22, 2024 · 1 comment

NavodPeiris commented Jun 22, 2024

is there a way to donate System memory instead of GPU VRAM?

this may become more economical

KMouratidis commented Jul 4, 2024

I think if you don't have a GPU it will default to CPU, but you need to set --num_blocks manually:

petals/src/petals/server/server.py

Lines 275 to 279 in 6858586

    
           def _choose_num_blocks(self) -> int: 
        
               assert self.device.type in ("cuda", "mps"), ( 
        
                   "GPU is not available. If you want to run a CPU-only server, please specify --num_blocks. " 
        
                   "CPU-only servers in the public swarm are discouraged since they are much slower" 
        
               )

Might need more configuration though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment