Chapter 30

Contrastive & Metric Learning

Contrastive, triplet, InfoNCE, CLIP, and SigLIP losses, and embedding retrieval.

Contrastive learning turns related objects into nearby vectors and unrelated objects into separated vectors. In LLM systems, those vectors power retrieval, ranking, deduplication, and image-text alignment before a generator ever sees a token. The core pattern is simple: score all pairs in a batch, make matched pairs win, and backpropagate through the scores.

30.1 Embeddings and cosine scores

An embedding model maps an input to a vector $\vz \in \R^d$ . We compare two rows by the cosine of the angle between them, the same geometry as Chapter 2:

s(\vz_i, \vw_j) = \frac{\vz_i^\T \vw_j}{\|\vz_i\|\,\|\vw_j\|} .\tag{30.1}

Normalizing rows first makes every dot product a cosine. That matters because the model cannot win by increasing vector norms; it must rotate matched examples together and mismatched examples apart.

Listing 30.1 Row normalization and cosine similarity

def normalize_rows(x, eps=1e-12):
    """Return rows scaled to unit length."""
    x = np.asarray(x)
    norm = np.linalg.norm(x, axis=1, keepdims=True)
    return x / np.maximum(norm, eps)


def cosine_similarity(a, b):
    """All pairwise cosine similarities between row embeddings."""
    return normalize_rows(a) @ normalize_rows(b).T

A retrieval system builds a matrix $S$ of query-document similarities and sorts each row. The diagonal is the target in paired data: image $i$ matches caption $i$ , question $i$ matches passage $i$ , and so on. The rest of this chapter changes only the loss placed on that matrix.

Cosine similarity also separates training from serving. During training the model sees a whole batch and can compare every item with every other item. During serving a single query is embedded once, then compared against stored normalized vectors by dot product. The loss must therefore teach the geometry directly: nearby means interchangeable for the task, not merely large in norm or convenient for a classifier head.

30.2 Margins: pairs and triplets

A margin loss states how far is far enough. Let $d_{ij}=1-s_{ij}$ be cosine distance and $y=1$ for a matching pair, $y=0$ otherwise. The pairwise contrastive loss is

L = y d^2 + (1-y)\max(0, m-d)^2 .\tag{30.2}

The positive term pulls matches to distance zero. The negative term pushes only impostors that are inside the margin $m$ ; once $d \ge m$ , they stop contributing. A triplet loss compares one anchor, one positive, and one negative:

L = \max(0, d(a,p) - d(a,n) + m) .\tag{30.3}

The useful negative is rarely a random one. Hard-negative mining picks the nonmatching item with the largest similarity in the current batch, so the model spends updates on mistakes it actually makes.

Margins make these losses local. A positive pair keeps pulling until it is nearly identical under the chosen distance, but a negative pair is forgotten once it crosses the margin. This is efficient when labels say only "same" or "different"; it is also brittle when two negatives are semantically close, because the loss has no way to say "farther than the positive, but not too far." Triplets fix part of that by comparing a negative to the positive for the same anchor.

Listing 30.2 Pair and triplet losses with hard negatives

def pairwise_contrastive_loss(a, b, same, margin=0.5):
    """Mean y d^2 + (1-y) max(0, margin-d)^2 for cosine distance d."""
    same = np.asarray(same, dtype=np.float64)
    d = cosine_distance(a, b)
    pull = same * d**2
    push = (1.0 - same) * np.maximum(0.0, margin - d) ** 2
    return float(np.mean(pull + push))


def hard_negative_indices(anchor, candidates):
    """For paired rows, choose the nonmatching candidate with largest cosine."""
    scores = cosine_similarity(anchor, candidates)
    scores = scores.copy()
    np.fill_diagonal(scores, -np.inf)
    return np.argmax(scores, axis=1)


def triplet_loss_hard_negative(anchor, positive, margin=0.2):
    """Mean max(0, d(a,p)-d(a,n)+margin), mining n from the batch."""
    negatives = positive[hard_negative_indices(anchor, positive)]
    d_pos = cosine_distance(anchor, positive)
    d_neg = cosine_distance(anchor, negatives)
    return float(np.mean(np.maximum(0.0, d_pos - d_neg + margin)))

30.3 InfoNCE and its gradient

InfoNCE turns each row of similarities into a classification problem. With temperature $\tau$ , logits are $S_{ij}/\tau$ , and the correct class for row $i$ is column $i$ :

L_i = -\log \frac{\exp(S_{ii}/\tau)}{\sum_j \exp(S_{ij}/\tau)} .\tag{30.4}

Let $P_{ij}=\softmax(S_i/\tau)_j$ . Differentiate the log-softmax: the derivative of $-S_{ii}/\tau$ contributes $-1/\tau$ at the target, and the derivative of $\log\sum_j \exp(S_{ij}/\tau)$ contributes $P_{ij}/\tau$ . Averaging over $N$ rows gives

\frac{\partial L}{\partial S_{ij}} = \frac{P_{ij} - \one[i=j]}{N\tau} .\tag{30.5}

A smaller $\tau$ sharpens the softmax and also scales the gradient. The tests check this backward pass against central finite differences.

The sign is the whole algorithm. For the diagonal, $P_{ii}-1$ is negative unless the model is already certain, so gradient descent raises the matched score. For an off-diagonal entry, $P_{ij}$ is positive, so gradient descent lowers that impostor score. Hard negatives arise automatically: an off-diagonal with high probability gets the largest push down.

Listing 30.3 InfoNCE as row-wise cross-entropy

def info_nce_loss_and_grad(similarity, temperature=0.1):
    """Cross-entropy over rows, with correct pair on the diagonal."""
    n = similarity.shape[0]
    probabilities = softmax_rows(similarity / temperature)
    loss = -np.log(np.diag(probabilities)).mean()
    grad = probabilities.copy()
    grad[np.arange(n), np.arange(n)] -= 1.0
    grad /= n * temperature
    return float(loss), grad

30.4 CLIP and SigLIP losses

CLIP uses both retrieval directions. If $L_{\mathrm{row}}(S)$ predicts the matching text from an image and $L_{\mathrm{col}}(S)$ predicts the matching image from text, then

L_{\mathrm{CLIP}} = \tfrac12 L_{\mathrm{row}}(S) + \tfrac12 L_{\mathrm{col}}(S) .\tag{30.6}

The symmetric loss makes every image compete for texts and every text compete for images [radford2021learning]. SigLIP removes the row softmax. It labels diagonal pairs $Y_{ij}=1$ and off-diagonal pairs $Y_{ij}=-1$ , then applies a binary logistic loss with a learnable bias $b$ :

L = \frac{1}{N^2}\sum_{i,j}\log\big(1 + \exp(-Y_{ij}(S_{ij}+b))\big) .\tag{30.7}

The bias shifts the threshold between positive and negative pairs, which is useful because a batch has many more negatives than positives.

The two objectives disagree about how examples compete. In CLIP, each row and column is a probability distribution; making one negative lower raises the relative probability of all the others. In SigLIP, every pair is judged independently by a sigmoid, so the loss can be evaluated or sharded without a global softmax over the whole batch. The learnable bias absorbs the class imbalance created by many off-diagonal negatives.

Listing 30.4 Symmetric CLIP and pairwise SigLIP losses

def clip_loss_and_grad(similarity, temperature=0.1):
    """Symmetric CLIP loss: row retrieval plus column retrieval."""
    row_loss, row_grad = info_nce_loss_and_grad(similarity, temperature)
    col_loss, col_grad_t = info_nce_loss_and_grad(similarity.T, temperature)
    return 0.5 * (row_loss + col_loss), 0.5 * (row_grad + col_grad_t.T)


def siglip_loss_and_grad(similarity, bias=0.0):
    """Pairwise sigmoid loss with +1 labels on the diagonal and -1 elsewhere."""
    n = similarity.shape[0]
    labels = -np.ones_like(similarity)
    labels[np.arange(n), np.arange(n)] = 1.0
    logits = labels * (similarity + bias)
    loss = np.logaddexp(0.0, -logits).mean()
    grad_logits = -labels / (1.0 + np.exp(logits)) / similarity.size
    grad_bias = float(np.sum(grad_logits))
    return float(loss), grad_logits, grad_bias

30.5 Retrieval on synthetic pairs

A minimal two-tower retriever has one encoder for each side. Here both encoders are only linear maps, trained by the symmetric CLIP loss on synthetic paired observations. The evaluation is recall@k: the fraction of queries whose true partner appears among the top $k$ scores.

Listing 30.5 Synthetic paired data and recall@k

def make_synthetic_pairs(n=48, latent_dim=4, image_dim=7, text_dim=6, seed=0):
    rng = np.random.default_rng(seed)
    z = rng.standard_normal((n, latent_dim)).astype(np.float32)
    image_map = rng.standard_normal((latent_dim, image_dim)).astype(np.float32)
    text_map = rng.standard_normal((latent_dim, text_dim)).astype(np.float32)
    images = z @ image_map + 0.05 * rng.standard_normal((n, image_dim))
    texts = z @ text_map + 0.05 * rng.standard_normal((n, text_dim))
    return images.astype(np.float32), texts.astype(np.float32)


def retrieval_recall_at_k(image_embeddings, text_embeddings, k=1):
    scores = cosine_similarity(image_embeddings, text_embeddings)
    topk = np.argsort(-scores, axis=1)[:, :k]
    target = np.arange(scores.shape[0])[:, None]
    return float(np.mean(np.any(topk == target, axis=1)))

Listing 30.6 Tiny two-tower training loop

def train_linear_pair(images, texts, embed_dim=4, steps=250, lr=0.8, seed=1):
    rng = np.random.default_rng(seed)
    w_image = (0.1 * rng.standard_normal((images.shape[1], embed_dim)))
    w_text = (0.1 * rng.standard_normal((texts.shape[1], embed_dim)))
    w_image, w_text = w_image.astype(np.float32), w_text.astype(np.float32)
    for _ in range(steps):
        _, grad_image, grad_text = linear_pair_loss_and_grad(images, texts,
                                                             w_image, w_text)
        w_image -= lr * grad_image.astype(np.float32)
        w_text -= lr * grad_text.astype(np.float32)
    return images @ w_image, texts @ w_text

The point is not the architecture; it is the pressure from the loss. A batch supplies positives on the diagonal and negatives everywhere else. After training, recall@1 on the seeded synthetic problem is high, while an untrained slice of the raw features is near chance.

This tiny example uses full-batch training only to keep the code readable. Real systems sample many batches, refresh mined negatives, and evaluate both directions of retrieval. The same recall@k calculation still applies: a query succeeds if its paired item appears before enough wrong items in the sorted similarity row.

In practice

Modern embedding systems use these losses at very large batch sizes because every extra paired example adds many in-batch negatives. CLIP-style image-text pretraining remains the standard way to align vision and language encoders [radford2021learning]. CPC introduced InfoNCE as a contrastive predictive objective [oord2018], FaceNet popularized triplet loss for metric embeddings [schroff2015facenet], and SigLIP replaces the softmax competition with pairwise sigmoid terms [zhai2023sigmoid]. Production retrievers usually add mined negatives from an index, not only from the current minibatch.

Key equations

s(\vz, \vw) = \frac{\vz^\T\vw}{\|\vz\|\,\|\vw\|}, \qquad d = 1 - s

L_{\mathrm{pair}} = y d^2 + (1-y)\max(0, m-d)^2

L_{\mathrm{triplet}} = \max(0, d(a,p)-d(a,n)+m)

L_i = -\log\softmax(S_i/\tau)_i, \qquad \bar{S}_{ij} = \frac{P_{ij}-\one[i=j]}{N\tau}

L_{\mathrm{SigLIP}} = N^{-2}\sum_{i,j}\log(1+e^{-Y_{ij}(S_{ij}+b)})

30.6 Teach it

The one-sentence version. Contrastive learning trains embeddings by making the correct pair score higher than the alternatives.

An analogy. Arrange name tags and faces on a table: pull the matching tag toward each face, and push away the most confusing wrong tag.

At the board.

Draw two normalized vectors and write cosine as a dot product on the unit sphere.
For a positive/negative pair, draw the margin: negatives outside it are ignored.
Turn a row of pair scores into a softmax classifier; the target is the diagonal.
Add the column direction to get CLIP, then replace the softmax with independent sigmoid decisions to get SigLIP.

Misconceptions to address.

"Only positives matter." The negatives define what counts as close.
"Hard negatives are always mislabeled." They are often valid confusions; use labels and filters when mining them.
"Temperature is just a constant." It changes both probabilities and gradient scale.

Check for understanding. If a batch doubles in size, how many off-diagonal image-text comparisons does the CLIP similarity matrix contain, and why does that help retrieval training?

30.7 Exercises

Exercise 30.1 ★ Temperature and confidence

For two logits whose unscaled gap is $0.4$ , compute the target softmax probability at $\tau=0.1$ and $\tau=0.5$ . What changes besides the probability?

Solution in Appendix D

Exercise 30.2 ★★ Derive the InfoNCE gradient

Starting from (30.4), derive (30.5). Identify the two terms that come from the numerator and the log-normalizer.

Solution in Appendix D

Exercise 30.3 ★★ SigLIP’s bias gradient

For $N=3$ and all similarities and bias equal to zero, compute the derivative of (30.7) with respect to $b$ . Explain the sign.

Solution in Appendix D

Exercise 30.4 ★★★ Build a tiny retriever

Use make_synthetic_pairs, train_linear_pair, and retrieval_recall_at_k to train a small linear image-text retriever. Report recall@1 and recall@5 before and after training.

Solution in Appendix D

References

[oord2018] A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. 2018. arXiv:1807.03748
[radford2021learning] A. Radford et al. Learning Transferable Visual Models From Natural Language Supervision. 2021. arXiv:2103.00020
[schroff2015facenet] F. Schroff, D. Kalenichenko, and J. Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. 2015. arXiv:1503.03832
[zhai2023sigmoid] X. Zhai et al. Sigmoid Loss for Language Image Pre-Training. 2023. arXiv:2303.15343