Mr. Integral
Embedded notes/Moving and Processing Data/Analog Signals and Conversion

UNIT 11 · LESSON 2 OF 6

Resolution, Quantization, and Accuracy

Is the converter broken? What does “12 bits” actually promise?

INTERACTIVEFrom voltage to code
The staircase transfer function of an ADC and the code for one input voltage703.3 V1 LSB = 3.3 V / 8 = 413 mV; 1.9 V reads code 4code 4 means anything from 1.65 V to 2.06 V; using the step centre 1.86 V, the erroris -43.7 mV
The staircase transfer function of an ADC and the code for one input voltage703.3 V1 LSB = 3.3 V / 8 = 413 mV; 1.9 V reads code 4code 4 means anything from 1.65 V to 2.06 V; usingthe step centre 1.86 V, the error is -43.7 mV

Try this

Resolution
Code 4 of 7; 1 LSB = 413 mV.

An N-bit ADC divides its reference range into 2^N steps of one LSB = Vref / 2^N and outputs the number of the step the input falls in. Every code stands for a whole band of voltages, so even a perfect converter is uncertain by up to one LSB (±½ LSB if the code is read as the centre of its step). Inputs at or above the reference read full scale.

What you will be able to do
  • Compute the LSB size and the code for an input voltage for an N-bit ADC.
  • Convert a code back to a voltage and state the quantization uncertainty.
  • Distinguish resolution from accuracy, and name offset, gain, nonlinearity and noise errors.
  • Interpret an effective-number-of-bits (ENOB) specification.
  • Build a simple error budget for a measurement in volts and in LSBs.
Before you start
  • Sampling and aliasing (lesson 1).
  • Binary numbers and integer types (unit 2, lessons 1 and 2).
Steps in this lesson
  1. The staircase
  2. Quantization error
  3. Resolution is not accuracy
  4. Effective number of bits
  5. Worked example: the 2.500 V reading
  6. Common misconceptions

The puzzle

A 12-bit ADC on a 3.3 V reference should resolve 0.8 mV, yet the reading of a steady 2.500 V wanders over a dozen codes and sits 30 mV high. Is the converter broken? What does “12 bits” actually promise?

STEP 1

The staircase

An N-bit ADC divides its input range, 0 to Vref, into 2^N equal steps. One step is the least significant bit:

1 LSB=Vref2N,1\ \text{LSB} = \frac{V_{\text{ref}}}{2^{N}}, code=⌊VinVref 2N⌋\text{code} = \left\lfloor \frac{V_{\text{in}}}{V_{\text{ref}}}\, 2^{N} \right\rfloor

limited to 0 … 2^N − 1. With 12 bits and 3.3 V, 1 LSB = 3.3 / 4096 ≈ 0.806 mV. Real converters place their transition points slightly differently (some centre the first step on 0 V), and the datasheet’s transfer diagram says which; the difference is half an LSB.

↑ This step uses the figure at the top of the page.

STEP 2

Quantization error

Every code stands for a band one LSB wide. Converting a code back to volts with code × LSB gives the bottom of the band; adding half an LSB gives its centre, with an error of at most ±½ LSB for an ideal converter. The pico-sdk examples convert with 3.3 / (1 << 12), that is, divide by 2^N, consistent with the staircase above. Dividing by 2^N − 1 instead is a common habit that stretches the scale by one part in 4095, a gain error of its own.

STEP 3

Resolution is not accuracy

Resolution is the step size; accuracy is how close the reading is to the truth. A real ADC adds:

  • offset error: the whole staircase shifted by a fixed number of LSBs;
  • gain error: the slope wrong, so the error grows with the input;
  • nonlinearity: steps of unequal width (differential nonlinearity) and a curved overall line (integral nonlinearity);
  • noise: the code changes from one conversion to the next even for a perfectly steady input;
  • the reference itself: every error in Vref is a gain error in every reading (lesson 3).
INTERACTIVEResolution is not accuracy
Ideal and actual ADC transfer lines with offset and gain error+1000−100error in LSB (grey band: ±3σ noise)2.5 V: ideal code 3103, actual 3140: error +37 LSB = 29.8 mV8.7 ENOB implies about 2.8 LSB rms (2.29 mV) of noise and distortion, a spread ofroughly 17 LSB peak to peak12 bits resolve 0.806 mV steps; the accuracy here is set by offset and gain, whichcalibration can remove.
Ideal and actual ADC transfer lines with offset and gain error+1000−100error in LSB (grey band: ±3σ noise)2.5 V: ideal code 3103, actual 3140: error +37 LSB =29.8 mV8.7 ENOB implies about 2.8 LSB rms (2.29 mV) ofnoise and distortion, a spread of roughly 17 LSBpeak to peak12 bits resolve 0.806 mV steps; the accuracy here isset by offset and gain, which calibration canremove.
Error +37 LSB at 2.5 V.

Real converters add offset (the whole line shifted), gain error (the slope wrong) and noise on top of quantisation. The plot shows the resulting error across the input range: offset moves it up or down, gain error tilts it. The RP2040’s ADC is documented as 12 bits with 8.7 effective bits (ENOB): its noise and nonlinearity are as large as those of an ideal 8.7-bit converter. Offset and gain are steady and can be calibrated; random noise can only be averaged.

STEP 4

Effective number of bits

An ideal N-bit converter driven by a full-scale sine has a signal-to-noise ratio of

SNR≈6.02 N+1.76 dB\text{SNR} \approx 6.02\,N + 1.76\ \text{dB}

The effective number of bits turns a measured SNR (including distortion) back into the resolution of an ideal converter with the same performance. The RP2040’s ADC is specified as 12 bits with an ENOB of 8.7: its noise and distortion are like those of an ideal 8.7-bit converter, and the bottom three or so bits of a single reading are not dependable. Averaging several readings (lesson 5) reduces random noise; it cannot remove offset, gain or nonlinearity.

STEP 5

Worked example: the 2.500 V reading

On a 12-bit, 3.3 V ADC, 2.500 V should read ⌊2.5 / 3.3 × 4096⌋ = 3103. Suppose the channel has an offset of +6 LSB and a gain error of +1 %:

code≈3103.0×1.01+6≈3140\text{code} \approx 3103.0 \times 1.01 + 6 \approx 3140

37 LSB high, about 30 mV. Noise adds a spread: an ideal 8.7-bit converter’s quantization noise is its step, 2^(12 − 8.7) ≈ 10 LSB of the 12-bit scale, divided by √12, so about 2.8 LSB rms (2.3 mV), a spread of roughly 15–20 LSB peak to peak. (ENOB also counts distortion, so not all of that is random noise.) So the “0.8 mV” resolution is real, but the accuracy of one uncalibrated reading is tens of millivolts: calibrate the offset and gain (lesson 5) and average to reduce the noise.

MYTHS AND FACTS

Common misconceptions

12 bits means 0.8 mV accuracy

It means 0.8 mV steps; offset, gain, nonlinearity, noise and the reference decide the accuracy.

Divide by 4095 to get volts

The staircase has 4096 steps of Vref / 4096; follow the converter’s transfer diagram.

More bits always help

Bits below the noise floor add numbers, not information.

Averaging fixes everything

It reduces random noise only; systematic errors remain.

Check yourself

Answer in your head, then open the card.

A 10-bit ADC has a 2.5 V reference. What is 1 LSB, and what code does 1.000 V give?

2.5 / 1024 ≈ 2.44 mV; ⌊1.0 / 2.5 × 1024⌋ = 409.

The 12-bit RP2040 ADC reads code 2048. What voltage does that represent, and with what quantization uncertainty?

2048 × 3.3 / 4096 = 1.650 V at the bottom of the step; the true input lies between 1.6500 and 1.6508 V (the centre is 1.6504 V, ±0.4 mV), before any other error.

What SNR does an ideal 8.7-bit converter have?

6.02 × 8.7 + 1.76 ≈ 54.1 dB.

A reading has +10 LSB offset and −0.5 % gain error. At what input do the two errors cancel on a 12-bit converter?

When 0.005 × code = 10, code ≈ 2000, about 1.61 V; below that the reading is high, above it low.

Sources (3)
  1. Raspberry Pi Ltd, pico-sdk 1.5.1, hardware_adc/adc.h — “12 bit (8.7 ENOB)”; adc_fifo_setup err_in_fifo: “bit 15 of the FIFO contains error flag for each sample”; byte_shift reduces results to 8 bits for byte DMA
  2. Raspberry Pi Ltd, pico-examples, adc/hello_adc/hello_adc.c — “12-bit conversion, assume max value == ADC_VREF == 3.3 V”; conversion_factor = 3.3f / (1 << 12); prints result * conversion_factor
  3. STMicroelectronics, stm32f4xx-hal-driver, Inc/stm32f4xx_hal_adc.h — selectable resolutions 12, 10, 8 and 6 bits; “processing time (12 ADC clock cycles at ADC resolution 12 bits, 11 cycles at 10 bits, 9 cycles at 8 bits, 7 cycles at 6 bits)”