Diffusion Models: From Noise to Images

A diffusion model learns to generate images by first studying a much easier operation: destroying them. We will take one clean training image, add a little random noise, repeat that operation many times, and then ask what a neural network would need to learn in order to reverse it. Here is our clean data example. I am using a simple landscape so that its large shapes and small details are easy to track. In a real training set, this position would hold an ordinary photograph represented by a tensor of pixel values. Call that image x zero. The subscript does not name a pixel. It names a time in a corruption process, with time zero meaning that no artificial noise has yet been added. Now take one small step. Add an independent random value to every pixel, while retaining almost all of the image that was already there. One step would be difficult to notice, so this first comparison shows the effect after one hundred such steps. The house, mountains, horizon, and sun are still recognizable. But local pixel values have begun to wander. The corruption is not a blur that only removes high frequencies. It is random noise entering throughout the image. Continue to step four hundred. More of the observed tensor now comes from noise, and less comes from the original image. The broad composition survives, while windows, roof edges, and smaller color boundaries become unreliable. Push farther. By step seven hundred, the random component dominates most local evidence. You may still infer that something large sits near the center, but the image itself no longer supports confident object recognition. Near the end, even those weak global clues disappear. The process has not replaced the image with one fixed gray card. Every run ends in a different random tensor whose values follow a simple Gaussian distribution. That final simplicity is deliberate. Natural images occupy a complicated and highly structured part of pixel space. Standard Gaussian noise is a distribution we can sample immediately, without knowing anything about cats, houses, faces, or landscapes. So the forward process builds a bridge between two distributions. At one end are real images from the training data. At the other end is nearly pure Gaussian noise, which is easy to manufacture whenever we want a new sample. Write the bridge as a chain. We begin at x zero, take one random transition to x one, another to x two, and continue until x T. This is called the forward process. Forward refers to the direction from data toward noise. It is not the direction in which images are generated, and it does not require a neural network. We choose the corruption rule ourselves. That means we know exactly how much signal and how much random disturbance appear at every step. The entire forward chain can be simulated on demand from any training image. At the chosen final time, x T is approximately a standard Gaussian tensor. The approximation matters: schedules are designed so that almost no usable information about x zero remains, while the endpoint is numerically well behaved. One subtlety is worth fixing now. We do not repeatedly add raw noise without restraint, because the variance would grow without bound. Each step slightly reduces the existing signal while adding a calibrated amount of fresh noise. That calibration gives us clean algebra, stable scales, and a known target at every noise level. Those three facts will turn image generation into a supervised learning problem rather than a demand that a network invent an entire image in one unexplained leap.

Now let us make one forward step precise. Choose a small positive number beta t. This is the noise schedule at step t, and it controls how much new uncertainty enters during that transition. Define alpha t as one minus beta t. When beta is small, alpha is close to one, which is the algebraic statement that a single step should preserve most of its input. The next image has two ingredients. Multiply the current image by square root alpha t, then add an independent standard Gaussian tensor multiplied by square root one minus alpha t. Those square roots are not decorative. Variances scale with the square of a multiplier, so this choice keeps the total variance controlled as signal is exchanged for noise. The same statement can be written as a conditional distribution. Given x t minus one, x t is Gaussian. Its mean is the slightly reduced image, and its covariance is one minus alpha t times the identity. Every pixel channel receives independent Gaussian disturbance in this basic formulation. The image structure comes from the mean, while the covariance tells us how widely the next sample may wander around that mean. A remarkable simplification appears when we compose many of these Gaussian steps. Begin with the one-step rule we just wrote. Substitute the rule for x t minus one. The surviving signal multipliers multiply, while the independent Gaussian terms combine into another Gaussian term. Repeat all the way back to x zero. Define alpha bar t as the product of every alpha up to time t. The result is the closed form. At any chosen time t, x t equals square root alpha bar t times the clean image, plus square root one minus alpha bar t times one standard Gaussian tensor. This means training never needs to simulate all earlier corruption steps. Sample a time t, sample one epsilon, and construct x t directly from the original image. The picture on the right is a toy two-dimensional version of that mixture. Blue is a fixed signal vector, red is a fixed noise vector, and yellow is the noisy observation formed from both. At a low noise fraction, yellow stays close to the blue signal. The noise changes it, but signal still controls the result. Increase the noise fraction. The blue contribution shrinks, the red contribution gains weight, and the observed vector turns away from its original direction. Near the end, almost all reliable information comes from epsilon. The original signal has not been abruptly deleted. Its coefficient has been driven close to zero by the accumulated schedule. A useful summary is the signal-to-noise ratio. Signal power is alpha bar t, noise power is one minus alpha bar t, so their ratio falls as t increases. Keep three meanings attached to the notation. Beta t controls the fresh noise in one transition. Alpha bar t records the signal surviving from the original image. And the closed form lets one epsilon produce any desired training noise level. We chose every part of this corruption process. Therefore, for each noisy image, we know the clean source, the time, and the exact random tensor that was mixed in. That is the opening a supervised learner needs.

The forward process gives us noisy images, but training still needs a target. What should the neural network predict when it receives a noisy image x t and the time t? One tempting answer is the clean image x zero. Some diffusion parameterizations do predict clean data or a related quantity. But the most common first formulation asks for something even more directly known: the noise epsilon that we sampled ourselves. Here is the complete learning problem as a pipeline. Begin with a clean training image x zero. Independently sample a Gaussian noise tensor epsilon. It has exactly the same shape as the image tensor, with one random value for every channel and spatial location. Choose a time t, read the schedule coefficients at that time, and combine the clean image with epsilon to construct x t. We now possess both the input and the label. Feed x t and t to a neural network. The time input matters because the same visible pattern means something different at a low noise level and at a high one. The network returns a tensor with the image's shape. Call it epsilon hat, the network's estimate of the particular noise tensor used to create this example. Compare epsilon hat with the known epsilon using squared error. Pixel by pixel and channel by channel, the loss penalizes disagreement between the generated label and the prediction. Then backpropagate through the denoising network exactly as in ordinary supervised learning. The parameters move in the direction that reduces expected noise-prediction error. There is no human annotation step. Labels are free because the forward process manufactured them. Yet the statistical structure is completely supervised: each input is paired with a definite target tensor. Let us run one training example in the order an implementation uses. First, select a clean image from the data set. Second, choose a time uniformly from the allowed diffusion steps. Across training, this makes the same network practise very light denoising, medium denoising, and recovery from extremely noisy inputs. Third, draw a fresh Gaussian epsilon. Because it is newly sampled, the same clean image can generate unlimited distinct supervised examples. Fourth, use the closed-form forward equation to construct x t directly. There is no need to execute every earlier noise step. Fifth, ask the network for epsilon and compare it with the sampled target. Repeat with new images, times, and noise tensors until the predictor works across the entire schedule. The equations say the same thing compactly. Sample t uniformly from one through T. Sample epsilon from a standard Gaussian. This epsilon is randomness, but once sampled it becomes an exact observed label. Mix x zero and epsilon using the schedule at t. The result x t is the network input. The network receives x t and t, and predicts epsilon hat. In image models, this predictor is often a convolutional network or transformer with multiscale spatial features, but its architecture does not change the learning target. Finally minimize expected squared error. Expectations mean that we average over clean images, sampled times, and sampled noise. Standard minibatches approximate that average. Now compare this with demanding a finished image directly. A random seed does not come with one uniquely correct photograph. Many outputs could be plausible, so ordinary squared error has no single natural target. Noise prediction is different. We chose the clean image, chose t, sampled epsilon, and computed x t. The exact epsilon is therefore a valid label for that exact input. The network is still learning something profound. To identify which fluctuations are noise, it must learn what natural image structure looks like at every scale. But the optimization interface remains an ordinary supervised regression problem. Once that predictor is accurate, we can present a noisy tensor, estimate the disturbance hidden inside it, and take a carefully calibrated step toward a cleaner tensor. Repeating that operation is the generative process.

Generation begins at the endpoint of the forward process. Sample x T directly from standard Gaussian noise. There is no hidden photograph underneath this particular tensor, because we did not obtain it by corrupting a data image. Ask the trained network which part of this tensor looks like noise at time T. Then use the reverse transition to produce x T minus one, a sample expected to contain very slightly more image structure. One step is not supposed to reveal a finished picture. Training taught the network a family of modest corrections, one for every noise level. Image generation comes from composing those corrections. After many early reverse steps, broad statistical organization appears. Large regions begin to behave like sky and ground. A horizon and mountain mass can emerge even though individual pixels remain highly uncertain. Continue through medium noise levels. The process commits to object-scale structure: a bright region in the sky, a building-sized block, and a roof shape. These are global decisions about composition rather than finished texture. This ordering is typical, not an absolute law. High-noise inputs preserve little local evidence, so early reverse steps mainly settle low-frequency, large-scale relationships. Fine edges become recoverable only after the broader image has stabilized. At lower noise, edges sharpen. The network can now distinguish a wall from its windows, a roof boundary from the sky, and one contour from a nearby contour. The last steps make smaller corrections. They adjust local contrast, texture, color consistency, and tiny geometric details while preserving the composition chosen earlier. The completed image is not retrieved from a training record. It is one sample reached by starting at a newly drawn noise tensor and following the learned reverse dynamics. A second initial noise tensor would usually produce a different landscape, arrangement, or object identity. Randomness at the start supplies diversity, while the learned denoiser keeps the trajectory near the distribution of natural images. Now inspect one reverse step. First, the neural network predicts epsilon hat t from the current noisy image and the current time. The reverse mean uses that prediction with coefficients derived from the same forward schedule. The long expression is calibrated subtraction, not simply x t minus epsilon hat. Why the coefficients? The forward process scaled both signal and noise at every transition. A valid reverse step must undo that scaling while estimating which component belongs to the clean-image direction. For a standard stochastic diffusion sampler, add a calibrated Gaussian term sigma t z. This does not carelessly put back the noise we removed. It represents uncertainty in which slightly cleaner image could have produced the current one. That uncertainty can also be handled by deterministic trajectories or by samplers using fewer, larger numerical steps. These are alternative ways to traverse a learned denoising field. The central model is still the time-dependent predictor trained on noisy images. The full sampling algorithm is now short enough to read as a loop. Start by drawing x T from a standard Gaussian. At the current time, run the network once to predict the noise component. Use that prediction and the schedule to sample a slightly cleaner state. The state remains noisy unless t is already near zero. Decrease t and repeat. The same network is reused at every step, with its time input telling it which denoising regime is required. At zero, the state is interpreted as an image sample. Pure noise supplied the initial randomness; the learned sequence of local reverse decisions supplied recognizable structure. We can now generate plausible images, but we have not told the model which plausible image we want. The final ingredient is conditioning, where a text prompt steers the same denoising process toward a chosen description.

To request a particular image, add a condition c. Let the prompt be: a red sailboat at sunset. The reverse process remains a sequence of denoising steps, but every prediction may now depend on that description. A text encoder converts the prompt into a sequence of learned vectors. These vectors represent tokens and their relationships, rather than a single hand-written list of visual rules. The noisy image x t enters the denoiser as before. Time t still tells the network how much corruption to expect. Text features also enter the denoising network. In common architectures, cross-attention lets spatial image features query the token sequence. A region beginning to resemble a boat can therefore respond strongly to the boat token. The output is still a noise estimate, not a finished image. Conditioning changes which noise the model predicts should be removed from a sample compatible with the prompt. This distinction keeps the generative mechanism unified. An unconditional model and a text-conditioned model both perform learned reverse diffusion. The condition changes the direction of each local step. At an early, very noisy step, prompt information mainly influences global choices. The words sailboat and sunset can affect composition, object category, and broad color relationships before any rope or edge exists. At later steps, the same conditioning can influence finer details: the outline of a sail, reflections on water, and local texture. The condition is available throughout the trajectory, but what it can usefully control changes with the noise level. During training, the model sees image and text pairs. The noise target is still known exactly, so conditioning does not destroy the supervised objective. It adds information the predictor may use to estimate epsilon. A plain conditional prediction may follow the text, but practitioners often want a stronger response. Classifier-free guidance supplies a controllable way to amplify the prompt's influence without training a separate image classifier. The method asks the same network for two predictions. First, run it with an empty condition. This estimates how the sample should denoise without knowing the prompt. Second, run it with condition c. This prediction includes both general image knowledge and the influence of the text. Subtract the unconditional prediction from the conditional one. The difference points toward a denoising decision that makes the current sample more compatible with the prompt. Here is the guidance formula. Start with the unconditional prediction. Then multiply the conditional direction by a guidance strength w and add it. At w equal to zero, the conditional direction contributes nothing. At w equal to one, the formula returns the ordinary conditional prediction. Values above one extrapolate beyond it. The three pictures are a schematic comparison from the same prompt. With zero guidance, the model can produce a plausible image while largely ignoring the requested sailboat. At one, the ordinary conditional prediction introduces the requested boat and sunset while retaining the model's natural range of possible compositions. Turn guidance much higher and prompt features are pushed harder. The boat may become more unmistakable, colors more forceful, and composition more literal. But stronger is not simply better. Extrapolation can move the trajectory away from the distribution on which the denoiser was trained. Excessive guidance may create harsh contrast, repeated edges, distorted geometry, or artificial texture. It also reduces diversity. If the conditional direction is amplified on every sample, different initial noise tensors are pulled toward a narrower set of prompt-compatible solutions. So guidance strength controls a tradeoff. Turning it up usually improves literal prompt adherence, but tends to sacrifice variation and can eventually sacrifice naturalness and image quality. There is no universal best number. The useful range depends on the model, its training, the sampler, the prompt, and the desired balance between faithfulness and variety. Now the complete diffusion story fits into four steps. First, define a forward process that turns real images into known Gaussian noise. Second, create supervised examples by sampling a clean image, a time, and epsilon. Train a neural network to predict that exact epsilon from the noisy image and time. Third, generate by starting from fresh Gaussian noise and applying learned reverse steps. Large-scale structure usually settles before fine detail because the available information changes across noise levels. Fourth, condition the denoiser on text. Classifier-free guidance amplifies the conditional direction, trading diversity and eventually naturalness for stronger prompt adherence. The central trick is therefore not that one network learns to paint an image in a single leap. It learns a supervised family of noise-removal decisions, and generation emerges when those modest decisions are chained from randomness all the way to an image.