Transformer’s Self-Attention

Why Is Attention All You Need?

In 2017, Vaswani et al. published a paper titled Attention Is All You Need for the NeurIPS conference. The transformer architecture does not use recurrence or convolution. It solely relies on attention mechanisms.

In this article, we discuss the attention mechanisms in the transformer:

• Dot-Product And Word Embedding
• Scaled Dot-Product Attention
• Multi-Head Attention
• Self-Attention

1 Dot-Product And Word Embedding

The dot-product takes two equal-length vectors and returns a single number.

$$\begin{array}{rl}\mathit{a}& =[a_1,a_2,\dots ,a_n]\\ \mathit{b}& =[b_1,b_2,\dots ,b_n]\\ \\ \mathit{a}\cdot \mathit{b}& =[a_1{b}_1+a_2{b}_2+\cdots +a_n{b}_n]\\ & =\sum _{i=1}^n{a}_i{b}_i\end{array}$$

We use the dot operator to express the dot-product operation. We also call it the inner product as we calculate element-wise multiplication (that is, “inner product”) and sum those products together.

The geometric definition of the dot product is as follows:

$$\mathit{a}\cdot \mathit{b}=\parallel \mathit{a}\parallel \parallel \mathit{b}\parallel \cos \theta$$

$\parallel \mathit{a}\parallel$ is the magnitude of vector $\mathit{a}$, and $\parallel \mathit{b}\parallel$ is the magnitude of vector $\mathit{a}$. $\theta$ is the angle between vector $\mathit{a}$ and vector $\mathit{b}$.

We can also project vector $\mathit{a}$ onto vector $\mathit{b}$ and visualize the product of $\parallel \mathit{a}\parallel cos\theta$ and $\parallel \mathit{b}\parallel$. The calculated value is the same either way.

The dot-product $\parallel \mathit{a}\parallel \parallel \mathit{b}\parallel \cos \theta$ is at maximum when $\theta$ is 0 ($\cos 0=1$), and at minimum when is $\theta$ is $2\pi$ ($\cos 2\pi =-1$). In other words, the dot-product is bigger when two vectors have similar directions than otherwise.

Before discussing dot-product attention, we should talk about word embeddings as it gives us some intuition on how dot-product attention works.

Word embeddings use distributed representations of words (tokens), which is more efficient than one-hot vector representation. The word2vec and GloVe (Global Vectors for Word Representation) are well-known pre-trained word embeddings. Those word embedding vectors allow us to decompose a word into analogies.

The following diagram shows how we can decompose king and queen using word embedding vectors.

Figure 1 of Towards Understanding Linear Word Analogies

In other words, a word vector may contain multiple semantics. Hypothetically, we could query if a word vector has “royal” by projecting the vector with some matrix operation.

The question is: can a language model automatically learn to query different meanings and functions of each word in a sentence?

The transformer learns word embeddings from scratch, learns matrix weights for word vector projections, and calculates the dot-products for attention mechanisms. In the next section, we discuss how those vector mathematics work together.

2 Scaled Dot-Product Attention

Say we have a model that translates an English sentence $X$ into a French sentence $Y$. In the output sentence, the first-word vector would be a BOS (beginning-of-sentence) marker which is just another word vector (albeit fixed), and we call it vector ${\mathit{y}}_1$. The model needs to extract a context from the input sentence $X$ for the first-word vector ${\mathit{y}}_1$ in the output sentence $Y$.

For simplicity, we express a word vector as a row vector with four dimensions.

We use a 4x3 matrix $W^Q$ to project word vector ${\mathit{y}}_1$:

It’s a simple matrix multiplication on the word vector ${\mathit{y}}_1$.

$${\mathit{q}}_1={\mathit{y}}_1{W}^Q$$

As you can see, the dimension of ${\mathit{q}}_1$ does not have to be the same as that of ${\mathit{y}}_1$.

In this example, the dimension of the resulting vector became three.

We also project word vectors in $X$ using another matrix $W^K$. For example, we project word vector ${\mathit{x}}_1$ in $X$ as follows:

$${\mathit{k}}_1={\mathit{x}}_1{W}^K$$

Again, the dimension of ${\mathit{k}}_1$ does not have to be the same as the dimension of ${\mathit{x}}_1$.

However, the dimension of ${\mathit{k}}_1$ must be the same as the dimension of ${\mathit{q}}_1$. Only then can we apply the dot-product to them.

$$\dim ({\mathit{q}}_1)=\dim ({\mathit{k}}_1)$$

We apply the dot-product to see how strong these projected features (we call them query ${\mathit{q}}_1$ and key ${\mathit{k}}_1$) relate to each other:

$$s_{11}={\mathit{q}}_1\cdot {\mathit{k}}_1$$

We can calculate the dot-product by transposing vector ${\mathit{k}}_1$ and performing matrix multiplication:

The result is a scalar value. Let’s call it “score”. As noted above, we also call vector ${\mathit{q}}_1$ “query” and vector ${\mathit{k}}_1$ “key”. In other words, we extract a query from vector ${\mathit{y}}_1$, extract a key from vector ${\mathit{x}}_1$, and check how well the query matches with the key by the score $s_{11}$. So, we are finding out how two-word vectors relate in terms of extracted features (the query and the key).

Since we are using a linear transformation, we can calculate scores between multiple word-vectors by a simple matrix operation.

Sentence $X$ consists of $n$ word vectors (in reality, an input sentence has a pre-defined number of tokens, and unused word positions filled with a filler are effectively ignored):

$$X=\left[\begin{array}{c}{\mathit{x}}_1\\ {\mathit{x}}_2\\ ⋮\\ {\mathit{x}}_n\end{array}\right]$$

We can apply the following matrix operation to extract keys from sentence $X$:

$$X{W}^K=\left[\begin{array}{c}{\mathit{x}}_1\\ {\mathit{x}}_2\\ ⋮\\ {\mathit{x}}_n\end{array}\right]W^K=\left[\begin{array}{c}{\mathit{x}}_1{W}^K\\ {\mathit{x}}_2{W}^K\\ ⋮\\ {\mathit{x}}_n{W}^K\end{array}\right]=\left[\begin{array}{c}{\mathit{k}}_1\\ {\mathit{k}}_2\\ ⋮\\ {\mathit{k}}_n\end{array}\right]=K$$

We can apply the dot-product between the query ${\mathit{q}}_1$ and all of the keys in $K$:

$$\begin{array}{rl}{\mathit{q}}_1{K}^{\mathrm{\top }}& ={\mathit{q}}_1{\left[\begin{array}{c}{\mathit{k}}_1\\ {\mathit{k}}_2\\ ⋮\\ {\mathit{k}}_n\end{array}\right]}^{\mathrm{\top }}\\ & ={\mathit{q}}_1\left[\begin{array}{cccc}{\mathit{k}}_1^{\mathrm{\top }}& {\mathit{k}}_2^{\mathrm{\top }}& \dots & {\mathit{k}}_n^{\mathrm{\top }}\end{array}\right]\\ & =\left[\begin{array}{cccc}{\mathit{q}}_1{\mathit{k}}_1^{\mathrm{\top }}& {\mathit{q}}_1{\mathit{k}}_2^{\mathrm{\top }}& \dots & {\mathit{q}}_1{\mathit{k}}_n^{\mathrm{\top }}\end{array}\right]\\ & =\left[\begin{array}{cccc}{s}_{11}& s_{12}& \dots & s_{1n}\end{array}\right]\end{array}$$

We now have a list of scores that tells us how the query ${\mathit{q}}_1$ relates to each key in sentence $X$. We can apply the softmax function to convert the scores into weights:

$$\begin{array}{r}\\ \text{softmax}({\mathit{q}}_1{K}^{\mathrm{\top }})& =\text{softmax}(\left[\begin{array}{cccc}{s}_{11}& s_{12}& \dots & s_{1n}\end{array}\right])\\ & =\left[\begin{array}{cccc}{w}_{11}& w_{12}& \dots & w_{1n}\end{array}\right]\end{array}$$

The weights indicate how much the model should pay attention to each word in sentence $X$ regarding the query ${\mathit{q}}_1$. So, let’s call the weights “attention weights”.

We use the attention weights to extract a context from sentence $X$, but we need to handle one more step. Since we are dealing with a specific query, we should also extract particular values from sentence $X$. We use another matrix $W^V$ to project sentence $X$ to extract values.

$$X{W}^V=\left[\begin{array}{c}{\mathit{x}}_1\\ {\mathit{x}}_2\\ ⋮\\ {\mathit{x}}_n\end{array}\right]W^V=\left[\begin{array}{c}{\mathit{x}}_1{W}^V\\ {\mathit{x}}_2{W}^V\\ ⋮\\ {\mathit{x}}_n{W}^V\end{array}\right]=\left[\begin{array}{c}{\mathit{v}}_1\\ {\mathit{v}}_2\\ ⋮\\ {\mathit{v}}_n\end{array}\right]=V$$

The dimension of vectors in $V$ may be different from the dimension of vectors in $X$.

We calculate the weighted sum of value vectors using the attention weights:

$$\begin{array}{r}\\ \text{softmax}({\mathit{q}}_1^{\mathrm{\top }}K)V& =\left[\begin{array}{cccc}{w}_{11}& w_{12}& \dots & w_{1n}\end{array}\right]V\\ & =\left[\begin{array}{cccc}{w}_{11}& w_{12}& \dots & w_{1n}\end{array}\right]\left[\begin{array}{c}{\mathit{v}}_1\\ {\mathit{v}}_2\\ ⋮\\ {\mathit{v}}_n\end{array}\right]\\ & =\sum _{i=1}^n{w}_{1i}{\mathit{v}}_i\end{array}$$

The weighted sum of value vectors represents a context from sentence $X$ regarding the query ${\mathit{q}}_1$ from word vector ${\mathit{y}}_1$.

We can extend the logic to multiple words $({\mathit{y}}_1,{\mathit{y}}_2,\dots ,{\mathit{y}}_m)$ in sentence $Y$ (in reality, an output sentence has a pre-defined number of tokens and unpredicted word positions are masked and won’t affect the attention mechanism):

$$Y=\left[\begin{array}{c}{\mathit{y}}_1\\ {\mathit{y}}_2\\ ⋮\\ {\mathit{y}}_m\end{array}\right]$$

We represent all queries in a matrix:

$$Q=\left[\begin{array}{c}{\mathit{q}}_1\\ {\mathit{q}}_2\\ ⋮\\ {\mathit{q}}_n\end{array}\right]=\left[\begin{array}{c}{\mathit{y}}_1{W}^Q\\ {\mathit{y}}_2{W}^Q\\ ⋮\\ {\mathit{y}}_n{W}^Q\end{array}\right]=\left[\begin{array}{c}{\mathit{y}}_1\\ {\mathit{y}}_2\\ ⋮\\ {\mathit{y}}_n\end{array}\right]W^Q=Y{W}^Q$$

So, we can calculate the attention vectors for all tokens in sentence $Y$:

$$\begin{array}{r}\\ \text{Attention}(Q,K,V)& =\text{softmax}\left({\displaystyle \frac{Q{K}^{\mathrm{\top }}}{\sqrt{d_k}}}\right)V\\ \\ \text{where }Q=Y{W}^Q,& K=X{W}^K,V=X{W}^V\end{array}$$

The scaling $\frac{1}{\sqrt{d_k}}$ deserves some explanation. $d_k$ is the dimension of query and key vectors. When $d_k$ is large, the dot-product tends to be large in magnitude.

The paper explains the reason, assuming a scenario where the elements of the query and key vectors are independent random variables with mean 0 and variance 1. The dot-product of vectors $\mathit{q}$ and $\mathit{k}$ has mean 0 and variance $d_k$ due to the distribution of the product of two random variables:

$$\begin{array}{rl}\mathit{q}\cdot \mathit{k}& =\sum _{i=1}^{d_k}{q}_i{k}_i\\ \\ \mathbb{E}[\mathit{q}\cdot \mathit{k}]& =\mathbb{E}\left[\sum _{i=1}^{d_k}{q}_i{k}_i\right]\\ & =\sum _{i=1}^{d_k}\mathbb{E}[q_i{k}_i]\\ & =\sum _{i=1}^{d_k}\mathbb{E}[q_i]\mathbb{E}[k_i]\\ & =0\\ \\ \text{Var}(\mathit{q}\cdot \mathit{k})& =\text{Var}\left(\sum _{i=1}^{d_k}{q}_i{k}_i\right)\\ & =\sum _{i=1}^{d_k}\text{Var}(q_i{k}_i)\\ & =\sum _{i=1}^{d_k}({\sigma }_{q_i}^2+{\mu }_{q_i}^2)({\sigma }_{k_i}^2+{\mu }_{k_i}^2)-{\mu }_{q_i}^2{\mu }_{k_i}^2\\ & =\sum _{i=1}^{d_k}(1+0)(1+0)-0\cdot 0\\ & =d_k\end{array}$$

In short, more dimensions tend to produce larger scores, which is problematic because the softmax function uses the exponential function, pushing large values even larger.

The following example Python script should clarify the reason:

import numpy as np

def softmax(x):
    return np.exp(x)/np.exp(x).sum()

s1 = np.array([1.0, 2.0, 3.0, 4.0])
s2 = s1 * 10

print(f’s1: {softmax(s1)}’)
print(f’s2: {softmax(s2)}’)

The output is as follows:

s1: [0.0320586 0.08714432 0.23688282 0.64391426]
s2: [9.35719813e-14 2.06106005e-09 4.53978686e-05 9.99954600e-01]

The weights in s2 except the last element are almost zero, which makes the gradients very small.

Let me explain the reason why the gradients become small.

Suppose the softmax probability of class $i\in \{1,2,\dots ,N\}$ be:

$$p_i=\text{softmax}(z_i)=\frac{e^{z_i}}{\sum _{k=1}^N{e}^{z_k}}$$

Then, the partial derivatives of the softmax with respect to the variable $z_j$ are as follows:

$$\frac{\partial p_i}{\partial z_j}=p_i({\delta }_{ij}–p_j),{\delta }_{ij}=\{\begin{array}{ll}1& \text{if i=j}\\ 0& \text{otherwise}\end{array}$$

So, it’s like $p_i$ and $p_j$ are close to 0 or 1. Therefore, any of the partial derivatives will be almost 0.

As such, Vaswani et al. introduced the scaling factor, and they call the attention mechanism “scaled dot-product attention”.

3 Multi-Head Attention

A word could have a different meaning or function depending on the context. So, we should use multiple queries per word rather than just one.

In the paper, they use eight parallel attention calculations. They call each attention function “head”. In other words, they used eight heads (h=8).

The base transformer uses 512-dimensional word vectors projected into eight vectors of 64 (=512/8) dimensions — yielding eight representation subspaces. The scaled product-dot attention processes each of the eight representations using a different set of projection matrices:

$$\begin{array}{rl}{\text{head}}_i& =\text{Attention}(Q_i,K_i,V_i)(i=1\dots h)\\ \\ & \text{where }{Q}_i=Y{W}_i^Q,K_i=X{W}_i^K,V_i=X{W}_i^V\end{array}$$

Even though we have eight sets of matrix operations, we can perform them in parallel. So, it’s very fast.

The next step of the multi-head attention is to concatenate all eight heads and apply one more matrix operation $W^O$.

$$\begin{array}{r}\\ \text{MultiHead}(Y,X)=\text{Concat}({\text{head}}_1,\dots ,{\text{head}}_h)W^O\end{array}$$

Note: I’m using a slightly different notation than the paper to keep similar usage of Q, K, and V from the scaled dot-product attention section.

Since we use the same number of dimensions (=64) for value vectors, the concatenated vectors restore the original dimension (=64x8=512). But they could also use a different number of dimensions for value vectors because _W_O can adjust the final vector dimension.

The transformer uses multi-head attention in multiple ways. One is for encoder-decoder (source-target) attention where Y and X are different language sentences. Another use of multi-head attention is for self-attention, where Y and X are the same sentences.

4 Self-Attention

A word can have a different meaning or function depending on the word itself and the words around it.

For example, in the below two sentences, the word “second” has different meanings:

Give me a second, please.

I came second in the exam.

But there is only one-word embedding for the word “second”. However, the meaning of “second” depends on its context. So, we must treat the word “second” with its context.

Let’s look at another example.

My dog chases after the thief.

The above sentence shows a functional relationship between “dog” and “chases”. We also see another relationship between “after” and “thief”.

In summary, we need to extract word contexts and relationships within a sentence, where self-attention comes into play.

With self-attention, all queries, keys, and values originate from the same sentence. So, we use multi-head attention like MultiHead(X, X) to extract contexts for each word in a sentence.

Self-attention handles long-range dependencies well compared with RNN and CNN. For RNN, we have diminishing gradients that even LSTMs can not eliminate. CNN can only associate data within the kernel size. Self-attention has none of those issues. Moreover, self-attention operations can run parallel and much faster than sequential processing like RNN cells.

Also, we can visually inspect self-attention, making it more interpretable. The paper has a few visualizations of the attention mechanism. For example, the following is a self-attention visualization for the word “making” in layer 5 of the encoder.

Figure 3 of the paper

There are eight different colors with various intensities, representing the eight attention heads.

It is clear there is a strong relationship between “making” and “more difficult”.

The below image shows the word “its” and the referred words in layer 5 of the encoder (isolating only attention head 5 and attention head 6).

Figure 4 (bottom) of the paper

The word “its” has strong attention to “Law” and “application”.

The below shows two heads separately (green and red). Each head seems to have learned to perform different tasks based on the structure of the sentence.

Figure 5 of the paper

5 References

• Attention Is All You Need
  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
• The Annotated Transformer
  Harvard NLP
• The Illustrated Transformer
  Jay Alammar