Welcome to Incremental Social! Learn more about this project here! Check out lemmyverse to find more communities to join from here!
TheChurn , 4 months ago Every billion parameters needs about 2 GB of VRAM - if using bfloat16 representation. 16 bits per parameter, 8 bits per byte -> 2 bytes per parameter. 1 billion parameters ~ 2 Billion bytes ~ 2 GB. From the name, this model has 72 Billion parameters, so ~144 GB of VRAM
Every billion parameters needs about 2 GB of VRAM - if using bfloat16 representation. 16 bits per parameter, 8 bits per byte -> 2 bytes per parameter.
1 billion parameters ~ 2 Billion bytes ~ 2 GB.
From the name, this model has 72 Billion parameters, so ~144 GB of VRAM