The Log Trick

Logarithms appear throughout machine learning, from pretraining to reinforcement learning. In this article, I will explain what they mean, the computational problems they help solve, and why they are useful when training models. No prior knowledge of logarithms is assumed; we will build the ideas through small examples and PyTorch code.

What Are Logarithms?

A logarithm tells us the exponent to which we must raise a base to obtain a number. For example, because $2^4 = 16$, the logarithm of $16$ in base $2$ is:

$$\log_2 \ 16 = 4$$

For this whole-number power, we can also picture the logarithm as the number of divisions by $2$ needed to reach $1$. Let's look at this in action.

$$\begin{aligned} (1.)& \ 16 \ /\ 2 = 8 \\ (2.)& \ \ 8 \ /\ 2 = 4 \\ (3.)& \ \ 4 \ / \ 2 = 2 \\ (4.)& \ \ 2 \ / \ 2 = 1 \end{aligned}$$

As you can see above, starting from $16$ , it took $4$ steps of dividing by $2$ to get to $1$ . That matches the exponent: $2^4 = 16$.

The exponent definition also works when the answer is negative or fractional. For example, $2^{-3}=1/8$, so $\log_2(1/8)=-3$. And because $2^{1/2}=\sqrt{2}$, we have $\log_2(\sqrt{2})=1/2$.

In general,

$$\log_b N=x \quad\Longleftrightarrow\quad b^x=N,$$

where $N>0$, $b>0$, and $b\ne1$

Let's move on and talk about exponentiation, the inverse of taking a logarithm. Exponentiation goes in the other direction: given the base and exponent, what number do we obtain? For a positive integer exponent, this means multiplying that many copies of the base: $2^4 = 2\times2\times2\times2 = 16$.

Logarithms and exponentiation with the same base invert each other. therefore;

$$b ^ {\log_b \ N} = N$$

The above may look complicated, so let's break it down. Assuming we have

$$2 ^ {\log_2 \ 16} = 16$$

Consider $\log_2 \ 16 = 4$ , therefore $2^{\log_2 \ 16} = 2^4 = 16$

What this means is, if you take the log of a number using a base $b$ and you then compute the exponent of the result with same base $b$ you get back the number. We will exploit this very fact later.

The Curse of Small Numbers

Now that we know what logarithms do, let's look at a numerical problem in ML. Computers represent numbers using a finite range and precision. Multiplying many probabilities can produce a positive result so small that the chosen floating-point format rounds it to zero. This is called underflow. Let's see it in code.

The example outputs below were checked with PyTorch 2.8.0 on CPU. Exact last digits can vary across environments.

import torch
torch.manual_seed(0)

def multiply_small_nums(nums: torch.Tensor):
	result = nums.prod(dim=-1)
	print(f"Product: {result.item()}")
	
small_nums = torch.rand(20, dtype=torch.float32)

print(f"Numbers: {small_nums[:5]}")
multiply_small_nums(small_nums)

In the function above, we define a set of small numbers with 32-bit precision and we define a function that multiplies them together and prints their result. When you run this, you get something like this

Numbers: tensor([0.4963, 0.7682, 0.0885, 0.1320, 0.3074])
Product: 2.2445269254323108e-10

What you have above is a 1D tensor of small numbers and when you multiply them all together, you get a very small number as the product. The problem is, multiplying many small numbers will give you a much smaller number and this gets worse as the size of that tensor increases. Below if you change from 20 to something like 200, at float32, the product will vanish to zero.

small_nums = torch.rand(200, dtype=torch.float32)
multiply_small_nums(small_nums)

When you run the above, you get

Product: 0.0

To avoid forming such a tiny product, we can do the operation in log space. A tiny positive number can have a logarithm with a much more manageable magnitude.

For example, the natural logarithm uses base $e\approx2.71828$. For the small number $2\times10^{-4}$, we have $\log(2\times10^{-4})\approx-8.5172$. From here on, $\log$ means the natural logarithm.

The log value is easier to represent in this example. Log space greatly extends the range of products we can work with. The laws of logarithms let us replace multiplication of positive numbers with addition of their logarithms.

The numbers we are operating on must be strictly positive for finite real logarithms. At zero, the logarithm tends to negative infinity and negative inputs have no real logarithm.

For positive $a$ and $b$, the product rule states:

$$\begin{aligned} \log& \ (a * b) = \log \ a + \log \ b \end{aligned}$$

this can be generalised to any arbitrary sequence of multiplications as

$$\log\left(\prod_i x_i\right) = \ \sum_i \log \ {x_i}$$

The above basically states that the log of the product of an array of numbers is equal to the sum of the logs of the individual elements. Notably, you can apply exponent to the sum of the logs to get the actual product.

If we start from this equation

$$\log\left(\prod_i x_i\right) = \ \sum_i \log \ {x_i}$$

Let's take the exponent of both sides.

$$\exp\left(\log\left(\prod_i x_i\right)\right) = \exp\left(\ \sum_i \log \ {x_i}\right)$$

On the left side, the exp and the log cancels out, leaving us with

$$\prod_i x_i = \exp\left(\ \sum_i \log \ {x_i}\right)$$

In exact arithmetic, the product of an array of positive numbers equals the exponential of the sum of their natural logarithms. In floating-point arithmetic, the two routes can differ because of rounding. We can see this play out in code.

import torch

nums = torch.tensor([2.0, 4.0, 5.0], dtype=torch.float32)
direct_product = nums.prod(dim=-1)
print(f"Direct Product: {direct_product}")

nums_log = nums.log()
print(f"Logs: {nums_log}")

nums_log_sum = nums_log.sum(dim=-1)
print(f"Logs sum: {nums_log_sum}")

product_via_log = nums_log_sum.exp()
print(f"Product via Log: {product_via_log}")

Above, we first compute the product directly, then we compute the log of the elements, sum their log and then take the exponent to recover the actual product, when you run this, the direct product and the product via log will be same.

Direct Product: 40.0
Logs: tensor([0.6931, 1.3863, 1.6094])
Logs sum: 3.6888794898986816
Product via Log: 40.0

This example returns the same result by both routes. Other inputs can show numerical differences, even when the inputs are not particularly small.

Now, let's test this with a lot of smaller numbers

import torch
torch.manual_seed(0)

nums = torch.rand(100, dtype=torch.float32)

direct_product = nums.prod(dim=-1)
print(f"Direct Product: {direct_product}")

nums_log = nums.log()

nums_log_sum = nums_log.sum(dim=-1)
print(f"Log sum: {nums_log_sum}")

product_via_log = nums_log_sum.exp()
print(f"Product via Log: {product_via_log}")

In this code, we create 100 small numbers in float32 and calculate the product directly and through logarithms. running this on my machine, gives;

Direct Product: 0.0
Log sum: -107.00592041015625
Product via Log: 0.0

The direct product underflows to zero. The log sum remains representable, but exponentiating it in float32 underflows too: the reconstructed product is still too small for that format. Keeping the log sum is sufficient when the next calculation can also operate in log space.

If we need the ordinary product, we can convert the single log sum to float64 before exponentiating:

product_via_log = nums_log_sum.to(torch.float64).exp()
print(f"Product via Log: {product_via_log}")

Product via Log: 3.3722458966360343e-47

This prevents underflow during reconstruction, but does not recover accuracy lost when computing the float32 logarithms and sum. For comparison, multiplying the same input values in float64 gives approximately $3.37225098837134\times10^{-47}$. The reconstructed result differs by about $0.000151\%$.

Logarithms Enable LLM Pretraining

So far, we have seen how log space helps us work with very small products. This becomes especially useful in LLM pretraining, where sequence probabilities are products of many conditional token probabilities. Let's take a step back from thinking about logarithms for a moment. In pretraining, you are trying to teach a very large transformer model how to understand and respond in discrete tokens that could represent text or some other modalities such as audio and vision. Given some input text, "London is", you want the model to learn a completion such as "the capital of the United Kingdom". Let's imagine we have such a text in our pretraining data, the text says. London is the capital of the United Kingdom. We want to train our model to assign a high probability to this sequence. Each token contributes through the probability the model assigns to it given the tokens before it. Let's call that full sequence $y$ and every single token in that sequence as $y_t$ . In this case, we will make the assumption that every single word + empty space is a token. For simplicity, let's have the following tokens.

y = {
"y_0": "London ",
"y_1": "is ",
"y_2": "the ",
"y_3": "capital ",
"y_4": "of ",
"y_5": "the ",
"y_6": "United ",
"y_7": "Kingdom"
}

We can therefore define $y$ as follows

$$y = (y_0, y_1, y_2, y_3, y_4, y_5, y_6, y_7)$$

Each $y_t$ is a token, and its conditional probability with respect to the prior tokens can be expressed as $p \ (y_t \ | \ y_{<t})$ , for example, the probability of $y_3$ is conditional on $y_0, \ y_1 \ \text{and} \ y_2$ , this can be written as, $p \ (y_3 \ | \ y_0, \ y_1, \ y_2)$ .

If this confuses you, what we basically mean by $p \ (y_3 \ | \ y_0, \ y_1, \ y_2)$ is the probability of token $y_3$ given the context $y_0, \ y_1, \ y_2$ .

Example:

$$p_\theta(\text{capital}\mid\text{London is the}) = 0.7$$

This value says that the model assigns probability $0.7$ to capital after the context London is the.

With that explained, the probability of a token sequence $y$ of length $T$ factorizes into a product of conditional token probabilities:

$$p(y) = \prod_{t=0}^{T-1} p (y_t | y_{<t})$$

Our goal in pretraining is to maximize $p(y)$ and the equation above provides us the mechanism to do just that by converting the above product into a log sum. So let's break this down.

Rather than directly trying to maximize the probability of the sequence, we can maximize its log. As the probability gets larger, its log gets larger too. So, whichever model settings give us the highest probability also give us the highest log probability. This allows us to maximize the probability of the model generating the sequence by optimizing the parameters of the model in a direction that maximizes the sum of the conditional log probabilities of the individual tokens. This brings it to the following transformation

$$\log\left(\prod_{t=0}^{T-1} \ p(y_t | y_{<t}) \right) = \sum_{t=0}^{T-1} \log \left( p \ (y_t \ | \ y_{<t} \ ) \right)$$

In the above, we simply used the laws of logarithms to convert the log of the product of the probabilities into the sum of the log of the probabilities of each token.

Maximizing this sum is equivalent to maximizing the probability of the sequence $y$. As we typically minimize a loss function in gradient descent, we can convert this maximization problem into a minimization problem by optimizing the negative.

You might wonder why this is a good thing. If you have an algorithm that only knows how to minimize a value, but you want to maximize a number, by flipping the sign, your maximum becomes the minimum. E.g, while $7$ is greater than $3$, once you flip the sign, $-7$ becomes less than $-3$ .

Therefore, we will flip the sign on our objective and our pretraining objective becomes.

$$J(\theta) = -\sum_{t=0}^{T-1} \left( \log \ p_\theta \ (y_t \ | \ y_{<t}) \right)$$

This is the negative log-likelihood (NLL) loss for one sequence. The subscript $\theta$ denotes that the probabilities depend on the model's trainable parameters. For the first token, $y_{<0}$ is an empty prefix, or the model's designated beginning-of-sequence context. If a prompt is given, we condition on that prompt too.

Training aggregates these losses across many sequences, often averaging over valid target tokens and excluding padding. In next-token training, each prediction is paired with the token that follows its input context.

Let's explain this with an example. Suppose a three-token sequence has conditional probabilities $0.5$, $0.2$, and $0.1$. Its probability is

$$0.5\times0.2\times0.1=0.01.$$

Its log probability is

$$\log(0.5)+\log(0.2)+\log(0.1) \approx-0.693147-1.609438-2.302585=-4.605170,$$

so its NLL is approximately $4.6052$. If the model raises the probability of the third target token from $0.1$ to $0.2$, with the other two probabilities unchanged, the sequence probability doubles to $0.02$ and its NLL falls to approximately $3.9120$. A lower loss means the model assigns a higher probability to this training sequence.

In practice, we obtain log probabilities directly from logits using a numerically stable log_softmax operation. Computing probabilities first and then taking their logs can lose information if those probabilities have already rounded to zero, so we typically stay in log-space as long as we can. Cross-entropy loss typically implements the log-softmax and NLL calculation together. We do not need to exponentiate the sequence log probability to train the model.

Let's see this in PyTorch. Suppose we have three target tokens and a vocabulary of four possible tokens. Each row below contains the model's raw scores, called logits, for predicting one token given its preceding context. The target IDs tell us which token actually occurred at each position.

import torch
import torch.nn.functional as F

# Shape: [3 token positions, 4 vocabulary entries].
# These are example raw scores from which we compute probabilities
logits = torch.tensor([
    [2.0, 1.0, 0.0, -1.0],
    [0.0, 2.0, 1.0, -1.0],
    [1.0, 0.0, -1.0, 2.0],
])

targets = torch.tensor([0, 2, 1], dtype=torch.long)

# First, calculate the negative log-likelihood explicitly.
log_probs = F.log_softmax(logits, dim=-1)
target_log_probs = log_probs.gather(
    dim=-1, index=targets.unsqueeze(-1)
).squeeze(-1)
manual_nll = -target_log_probs.sum()

# Cross-entropy performs the same calculation directly from logits.
loss = F.cross_entropy(logits, targets, reduction="sum")

print(f"Manual NLL: {manual_nll.item():.5f}")
print(f"Cross-entropy (sum): {loss.item():.5f}")

Manual NLL: 4.32057
Cross-entropy (sum): 4.32057

The gather operation selects the log probability of the target token from each row. Negating and adding those three values gives the sequence NLL. With reduction="sum", cross-entropy gives the same result, up to floating-point rounding.

Notice that we pass the raw logits to cross_entropy; we do not apply softmax first. What we basically have here is two different ways to compute the NLL loss, manually or by using pytorch's cross entropy function, both operate directly on the logits producing identical results.

Logarithms in Reinforcement Learning

Logarithms also appear in reinforcement learning, including probability ratios and KL divergences. Here, we will focus on the probability ratio used by PPO.

Suppose a token had probability $0.2$ under the policy that generated the data, and now has probability $0.3$ under the policy we are updating. The ratio is $0.3/0.2=1.5$: the new policy assigns that token 1.5 times its previous probability. A ratio of one means no change for that token and context.

For a token $y_t$, let $\pi_{\mathrm{old}}$ be the data-generating policy, $\pi_\theta$ the policy being updated, and $A_t$ the estimated advantage. The advantage indicates whether the sampled action was better or worse than the baseline used by the algorithm. The per-token PPO-Clip policy loss is:

$$\begin{aligned} r_t &= \frac{\pi_\theta(y_t\mid y_{<t})} {\pi_{\mathrm{old}}(y_t\mid y_{<t})} \\ u_t &= r_t A_t \\ c_t &= \operatorname{clip}(r_t,1-\epsilon,1+\epsilon)A_t \\ \ell_t &= -\min(u_t,c_t). \end{aligned}$$

The small positive hyperparameter $\epsilon$ controls the clipping interval; it is different from the exponential base $e$. We clip the ratio first, then multiply by the advantage. The policy loss is averaged over the relevant training samples.

We will focus on how to calculate $r_t$ in a stable manner. Probabilities can be small, and if we directly materialize the probabilities of the tokens from the policies, they can get rounded down to zero if they are too small to be represented in the precision we are operating in, creating underflow problems, therefore, we will make the above equation work in log-space instead.

Unlike the product examples, we use the division rule of logarithms. For positive $a$ and $b$,

$$\log\left(\frac{a}{b}\right)=\log a-\log b.$$

Exponentiating both sides gives

$$\frac{a}{b}=\exp(\log a-\log b).$$

For our example,

$$\exp(\log 0.3-\log 0.2) =\exp(\log 1.5)=1.5.$$

Applying this to the PPO ratio,

$$r_t=\exp\left( \log\pi_\theta(y_t\mid y_{<t}) -\log\pi_{\mathrm{old}}(y_t\mid y_{<t}) \right).$$

Here is an example of this in action.

import torch

torch.manual_seed(0)
batch_size = 2
seq_len = 8
vocab_size = 20

# Each position represents an already-aligned next-token prediction.
old_logits = torch.randn(batch_size, seq_len, vocab_size)
new_logits = torch.randn(batch_size, seq_len, vocab_size)

# Normalize across vocabulary entries, producing [batch, time, vocabulary].
old_log_probs = torch.log_softmax(old_logits, dim=-1)
new_log_probs = torch.log_softmax(new_logits, dim=-1)

labels = torch.randint(
    low=0, high=vocab_size, size=(batch_size, seq_len)
)

# Select the same token under each policy: [batch, time].
old_token_log_probs = old_log_probs.gather(
    dim=-1, index=labels.unsqueeze(-1)
).squeeze(-1).detach()

new_token_log_probs = new_log_probs.gather(
    dim=-1, index=labels.unsqueeze(-1)
).squeeze(-1)

ppo_ratio = torch.exp(new_token_log_probs - old_token_log_probs)
print(f"Ratio shape: {tuple(ppo_ratio.shape)}")

Ratio shape: (2, 8)

There is one ratio for each selected token, not one ratio for every vocabulary entry. In actual PPO, the selected tokens are actions sampled from the old policy, and their recorded old log probabilities stay fixed while the new policy is updated. Both policies must score the same tokens under the same contexts. The subtraction gives us the log ratio directly. We then exponentiate because the PPO objective uses the ordinary ratio. Extreme log ratios can still overflow or underflow during exponentiation. For this reason, you want to perform these operations in float32 rather than float16 to reduce the chances of overflow or underflow, although this does not eliminate the risk completely.

Summary

Logarithms turn products into sums and quotients into differences. This lets us work with log probabilities even when the corresponding ordinary probabilities are too small to represent reliably.

In pretraining, we minimize a sum of negative conditional log probabilities. In PPO, we subtract the old token log probability from the new one and exponentiate to obtain a probability ratio. The same elementary log identities support both calculations.

The practical lesson is to compute and retain log probabilities when possible, using stable operations such as log_softmax. If we convert back to ordinary probabilities, the result must still fit the output format, and earlier rounding error remains. I hope these examples make the logarithms in ML equations easier to recognize and understand.

References

• PyTorch: log-softmax and numerical stability
• PyTorch: cross-entropy loss
• PyTorch: uniform random values and their range
• OpenAI Spinning Up: PPO-Clip equations