Sizing an LLM: Parameters
 Posted on May 01, 2024
 cloud, kubernetes, engineering, openshift
 By Carlos Camacho
Large language models (LLMs) like GPT3 and GPT4 have become a cornerstone of modern natural language processing (NLP) applications, driving advancements in machine translation, text generation, question answering, and more. These models are powered by neural networks with billions of parameters, making them incredibly powerful but also resourceintensive.
Understanding Large Language Models: Parameters, Calculations, and VRAM Requirements
In this blog post, we’ll explore what parameters in large language models are, how they are calculated, and the implications of these calculations on the space required to load a model into VRAM (Video RAM).
What Are Large Language Models?
Large language models are deep learning models designed to understand and generate human language. They are typically based on transformer architectures, which allow the model to handle complex language tasks by capturing contextual information from input text sequences.
Parameters: The Building Blocks of LLMs
In the context of neural networks, parameters refer to the weights and biases within the model. These are the elements that the model adjusts during training to minimize the difference between its predictions and the actual outcomes (i.e., to reduce the loss function). The parameters are what the model “learns” and are essential for the model’s ability to generalize from training data to unseen text.
The number of parameters in a model is a key factor in its complexity and capacity. For example, GPT3 has 175 billion parameters, while earlier versions like GPT2 had 1.5 billion parameters. The sheer scale of these models means they can capture a wide range of linguistic patterns, making them versatile across different NLP tasks.
How Are Parameters Calculated?
Calculating the parameters of a language model involves understanding the architecture of the model. For transformerbased models, the parameters are primarily located in:

Embedding Layers: These layers convert input tokens (words, subwords, or characters) into vectors of continuous numbers. The parameters here are the weights used to create these embeddings.

Attention Mechanisms: Transformers rely heavily on attention mechanisms, particularly selfattention, to determine the importance of each word in a sequence relative to others. The parameters in these mechanisms include weights that govern the interactions between different tokens.

Feedforward Layers: After the attention mechanism processes the input, the output is passed through feedforward neural networks, which consist of fully connected layers. Each of these layers has its own set of parameters (weights and biases).

Output Layers: Finally, the processed data is passed through the output layers, which convert the model’s internal representation back into a format that can be interpreted (e.g., predicting the next word in a sequence).
Example Calculation
Let’s consider a simplified transformer model with the following characteristics:
 Embedding Size: 512
 Number of Layers: 12
 Attention Heads: 8
 Vocabulary Size: 50,000
The total number of parameters can be broken down as follows:
 Embedding Layer:
 The embedding layer would have
Vocabulary Size * Embedding Size
parameters.  Example: (50,000 \times 512 = 25,600,000) parameters.
 The embedding layer would have
 Attention Mechanism:
 Each attention head has
Embedding Size / Attention Heads
parameters for queries, keys, and values.  Example: If we have 8 heads, each head would have (512 / 8 = 64) parameters per token.
 For the entire model: (12 \times 8 \times (Embedding Size \times Embedding Size / Attention Heads) = 12 \times 8 \times (512 \times 64) = 3,932,160) parameters.
 Each attention head has
 Feedforward Layers:
 Each layer typically has
2 * Embedding Size^2
parameters.  Example: (12 \times 2 \times 512^2 = 6,291,456) parameters.
 Each layer typically has
 Output Layers:
 The output layer usually has
Embedding Size * Vocabulary Size
parameters.  Example: (512 \times 50,000 = 25,600,000) parameters.
 The output layer usually has
Adding these up, the total number of parameters for this simplified model would be approximately:
[ 25,600,000 + 3,932,160 + 6,291,456 + 25,600,000 = 61,423,616 \text{ parameters} ]
VRAM Requirements: How Much Space Do LLMs Need?
The enormous number of parameters in large language models poses significant challenges for deployment, especially concerning memory requirements. VRAM, which is specialized memory used by GPUs (Graphics Processing Units), plays a critical role in running these models efficiently.
Parameter Storage
Each parameter in a model is typically stored as a floatingpoint number, usually in 16bit (FP16), 32bit (FP32), or in some cases, 8bit (INT8) precision. The choice of precision affects both the VRAM requirements and the performance of the model.
 FP32 (32bit precision): Each parameter requires 4 bytes.
 FP16 (16bit precision): Each parameter requires 2 bytes.
 INT8 (8bit precision): Each parameter requires 1 byte.
Using our previous example of a model with approximately 61 million parameters:
 FP32: (61,423,616 \times 4 = 245,694,464) bytes or roughly 234 MB of VRAM.
 FP16: (61,423,616 \times 2 = 122,847,232) bytes or roughly 117 MB of VRAM.
 INT8: (61,423,616 \times 1 = 61,423,616) bytes or roughly 58.6 MB of VRAM.
Additional Memory Requirements
Besides storing the parameters, VRAM is also needed for:

Activations: Intermediate activations during the forward and backward passes require additional memory. The size of activations can be substantial, especially for deep networks with many layers.

Optimizer States: When training a model, optimizers like Adam maintain additional states for each parameter, such as momentums and velocities, which also consume VRAM.

Batch Size: The size of the input batch processed by the model in one go also affects VRAM usage. Larger batch sizes require more memory for both the input data and the activations.
Estimating VRAM Requirements
To estimate the total VRAM required, you can use the following rule of thumb:
[ \text{Total VRAM} \approx \text{Model Parameters} + \text{Activations} + \text{Optimizer States} + \text{Input Data} ]
For a model with 61 million parameters using FP32 precision, a small batch size, and moderate complexity, you might need:
 Model Parameters: 234 MB (from above)
 Activations: 200 MB (estimated)
 Optimizer States: 468 MB (2 times model parameters, typical for Adam)
 Input Data: 50 MB (depends on input size and batch size)
Total estimated VRAM requirement: (234 + 200 + 468 + 50 = 952) MB.
Conclusion
Large language models are incredibly powerful tools that rely on a vast number of parameters to perform complex language tasks. Understanding how these parameters are calculated and their impact on VRAM usage is crucial for effectively deploying these models in realworld applications. While the resource demands are significant, advances in model optimization, mixed precision training, and hardware capabilities are making it increasingly feasible to leverage these models in various settings. As technology continues to evolve, the balance between model size, performance, and resource efficiency will remain a critical area of focus in the development of large language models.
Responses
Want to leave a comment? Visit this post's issue page on GitHub (you'll need a GitHub account. What? Like you already don't have one?!).