Introduction:
H.264, the latest video compression standard, uses CAVLC for encoding the coefficients after quantization. CAVLC encodes the coefficients (the coefficients may be positive or negative) into binary bit stream.
Original
Residue

DCT
Quantization
CAVLC
Predicted
NALU
In H.264 Encoder, the predicted submacroblock (of size 4x4) is subtracted from the original submacroblock (of size 4x4), resulting the residue submacroblock (of size 4x4). This residue is transformed by modified Discrete Cosine Transform. The transformed residue is also of size 4x4. Then this transformed residue is quantized (dividing by some defined number, QStep). This quantizedtransformed residue is encoded as follows. Encoding process:
There will be 16 coefficients in a 4x4 submacroblock. The total number of coefficients is tot_coef = 16.
(0,0)
(1,0)
(2,0)
(3,0)
(0,1)
(1,1)
(2,1)
(3,1)
(0,2)
(1,2)
(2,2)
(3,2)
(0,3)
(1,3)
(2,3)
(3,3)
The 16 coefficients are taken in a zigzag manner as shown in the above figure, and written in an array.
(0,0) – (0,1) – (1,0) – (2,0) – (1,1) – (0,2) – (0,3) – (1,2) – (2,1) – (3,0) – (3,1) – (2,2) – (1,3) – (2,3) – (3,2) – (3,3)
Note: For Chroma coding, two different blocks will come for coding. One is Chroma DC of 2x2 size (in case of 4:2:0), or 4x2 size (in case 4:2:2) or 4x4 size (in case of 4:4:4). Here, tot_coef = 4.
Another one is Chroma AC of 4x4 size. But in this Chroma AC, the DC Coefficient is 0 at (0,0). Here, tot_coef = 15.
The important parameters to be encoded.
1. Number of nonzero coefficients (numCoef) and Trailing Ones (T1) 2. The pattern of Trailing Ones (T1)
3. The nonzero coefficients (Levels)
4. Number of zeros embedded in the nonzero coefficients (Total_zeros) 5. The location of those embedded zeros (run_before)
1. Number of nonzero coefficients (numCoef) and Trailing Ones (T1)
CAVLC Encoder
2
Calculate number of nonzero coefficients (numCoeff). For this array (or sub macroblock), allot nCPresent = numCoeff. [0 ≤ numCoeff ≤ tot_coef] Remove the zero coefficients from the last, one – by – one, till the nonzero coefficient is reached. This reduced array may have zeros in between the first and last nonzero coefficients.
If the last nonzero coefficient is not equal to ‘±1’, then Trailing ones (T1) = 0. If the last nonzero coefficients is equal to ‘±1’, then calculate the number of continuous ones (+1 or 1) from last nonzero coefficient ‘±1’ towards first nonzero coefficient. T1 includes the last nonzero coefficient also, if the last nonzero coefficient is ‘±1’. Restrict the value of T1 to 3. Maximum value of T1 = 3 i.e., consider only three ‘±1’s only from the last nonzero coefficient inclusive. If there is any other ‘±1’ still, consider those ‘±1’ as normal numbers, not as T1s. So, now T1 is ready. i.e., [0 ≤ T1 ≤ 3]. For Luma, Chroma DC (4:4:4) and Chroma AC
(T1, numCoeff) will be coded based on nCContext. nCContext = (nCA + nCB + 1) >> 1. nCA is the number of nonzero coefficients available in the left 4x4 submacroblock and nCB is the number of nonzero coefficients available in the up 4x4 submacroblock. If any of nCA or nCB is not available, then nCContext = (nCA + nCB). Note that nCPresent has no role here. Based on nCContext, the corresponding column will be selected in the Table 1. For (T1, numCoeff), the code will be selected. If nCContext < 8, the codes will be variable length codes. If nCContext > 7, the codes will be fixed length codes (6bits). The first four bits of fixed length code is (numCoeff – 1) and the last two bits of fixed length code is T1. But for T1 = 0 and numCoeff = 0, the code is different.
CAVLC Encoder
T1 numCoeff
0 nC < 2
0
0
1
0
1
000101
0
2
00000111
0
3
000000111
0
4
0000000111
0
5
00000000111
0
6
0000000001111
0
7
0000000001011
0
8
0000000001000
0
9
00000000001111
0
10
00000000001011
0
11...