Generative Modeling: From Data Distributions to Deep Generative Models

If you approach generative models as “”networks that create images,”” the field quickly turns into a collection of disconnected architectures.

A more useful developer mental model starts one level lower:

What probability structure could have produced the data, and how can we represent, learn, and infer that structure without making the computation impossible?

That question connects autoregressive models, VAEs, flow-based models, GANs, and diffusion models. Their architectures look very different, but they all respond to the same underlying tension: high-dimensional data distributions are difficult to represent, learn, normalize, sample from, and reason about.

Generative modeling can therefore be organized around three interacting problems:

• Representation: How do we represent a complex high-dimensional joint distribution?
• Learning: How do we make the model distribution approach the data distribution?
• Inference: Given an observation, how do we reason backward about hidden variables or the process that generated it?

Once these three pieces are connected, the major families of deep generative models become much easier to understand.

From prediction to distribution learning

A discriminative model usually begins with a prediction problem. Given an input

$x$

, predict the most likely output

$y$

:

The model focuses directly on the conditional relationship required for prediction.

A generative model asks a broader question. Instead of learning only the path from

$x$

to

$y$

, it models the probability structure from which the data arises.

For class-conditional modeling, for example, we can model

$p(x\mid y)$

together with the prior

$p(y)$

and recover the posterior using Bayes’ rule:

In unsupervised generative modeling, the target becomes the data distribution itself. We assume the training samples come from some unknown distribution:

The model then constructs a parameterized distribution intended to approximate it.

This perspective connects tasks that can otherwise seem separate:

Density estimation asks how probable an observation is under the model. Generation draws new samples from the learned model distribution. Unsupervised representation learning can capture latent structure shared by the data while learning that distribution.

All three begin with the same underlying goal: learning the probability structure of the data.

Why high-dimensional data makes this hard

Real data usually lives in a high-dimensional space. An image, for example, is described by many variables, while valid observations occupy only a limited and complicated region among all possible combinations.

So the first problem is not simply “”use a neural network.”” It is:

How do we represent the joint distribution of many random variables compactly enough to compute with, while keeping enough flexibility to describe the data?

This connects naturally to inverse problems.

A forward problem moves from a cause

$x$

to an observation

$y$

. An inverse problem starts from an observed

$y$

and asks which possible

$x$

could have produced it. Conditional generation extends the same idea by generating possible

$x$

values consistent with a given condition.

Forward:
cause x  ──────────>  observation y

Inverse:
possible cause x  <──────────  observation y

Conditional generation:
condition y  ──────────>  possible samples x

Recovering a high-resolution image from a low-resolution one, estimating color from grayscale data, reconstructing missing regions, or recovering a CT slice from a sparse-view sinogram all share this structure.

The difficulty is that the inverse may not be unique. A single observation can be consistent with multiple possible causes. Inverse problems are therefore not simply about running a function backward; they require reasoning over possible causes that are compatible with the observation.

Latent variables turn inversion into posterior inference

Suppose a model contains an observed variable

$x$

and a hidden variable

$z$

.

The generative direction models the relationship between

$z$

and

$x$

. Once

$x$

has been observed, the reverse question is which values of

$z$

could have produced it.

That is posterior inference:

Posterior inference can require both computing the posterior distribution and calculating expectations under it. The E-step of the EM algorithm is a direct example:

Here the posterior over the unobserved

$z$

is used to compute the expected complete-data log-likelihood.

As the latent space grows and interactions among hidden variables become more complicated, direct posterior computation can become intractable. Increasing model expressivity therefore leads naturally to another question: how can the required inference remain computationally feasible?

more expressive representation
        ↓
more complicated latent structure
        ↓
harder posterior inference
        ↓
need for approximate inference

When the posterior is intractable

Two major approaches are MCMC and Variational Inference. Both approximate a posterior that is difficult to compute directly, but the approximation arises in different ways.

MCMC: approximate with samples

MCMC constructs an ergodic Markov chain whose stationary distribution is the target posterior. Samples generated by running the chain are then used to approximate the posterior and the expectations we need.

target posterior
      ↓
construct Markov chain
      ↓
run the chain
      ↓
collect samples
      ↓
approximate posterior expectations

The practical approximation comes from finite computation. In real problems, only a limited amount of time is available for generating samples, and exploring the relevant probability regions can be expensive in high-dimensional or complex models.

MCMC therefore has a finite-time sampling approximation in practice.

Variational Inference: approximate with optimization

Variational Inference makes a different tradeoff. Instead of sampling directly from the posterior, it defines a tractable family of distributions

$\mathcal{Q}$

and searches within that family for a distribution close to the true posterior:

This turns posterior inference into an optimization problem.

The tractability comes from restricting the search space. If the true posterior cannot be represented by the chosen variational family, more optimization time alone cannot eliminate the mismatch.

The contrast is useful:

MCMC approximation source:
finite-time sampling

Variational Inference approximation source:
restricted distribution family

This distinction becomes concrete in a VAE. Rather than computing a complicated latent posterior directly, the model uses a variational distribution, with the Encoder acting as a Recognition Network that constructs the approximation. The Decoder generates observations from the latent variable.

An abstract inference problem has now become part of a trainable model architecture.

The real bottleneck: flexibility versus tractability

Representation and computation collide when we try to model realistic high-dimensional distributions.

A simple probability model can be easy to normalize and evaluate but may lack enough flexibility to describe complex data. A deep neural network can represent much richer functions, but its output is not automatically a valid probability density.

A probability density must be nonnegative and integrate to one. One way to construct such a density is:

where

Exponentiation gives a positive quantity, while the normalizing constant

Zθ​

ensures that the density integrates to one.

For a complex neural function over a high-dimensional space, however, computing that integral can itself become intractable.

simple probability model
    → tractable
    → limited flexibility

deep neural representation
    → flexible
    → normalization or inference may become intractable

This is the Tractability-Flexibility Tradeoff.

It provides a useful way to compare deep generative models. Rather than asking only which network architecture a model uses, ask which difficult probability computation its structure makes manageable.

Learning a model distribution

In likelihood-based generative modeling, learning can be viewed as reducing the difference between the data distribution and the model distribution:

Expanding this expression separates a term that depends only on the data distribution from a term containing the model log probability. The data-only term is independent of

$\theta$

, so it does not change as the model parameters are optimized.

The learning objective therefore connects to increasing the model’s log probability on observed data. With a finite training set, this gives the progression:

minimize KL divergence
        ↓
minimize cross-entropy
        ↓
minimize negative log-likelihood

The challenge is to make

pmodel​(x;θ)

expressive enough for high-dimensional data while keeping the required probability computations feasible.

This is where the major generative model families diverge.

Autoregressive models: factorize the joint distribution

An autoregressive model handles a high-dimensional joint distribution by decomposing it into sequential conditional distributions:

Instead of directly modeling one complicated joint distribution, the model works with a product of conditional probabilities.

The key structural idea is factorization. The model represents the high-dimensional joint distribution through a sequence of conditional distributions that can be modeled in a computationally manageable form.

VAE: latent variables meet approximate inference

A latent-variable model can express the observed distribution as:

This provides a latent space for representing hidden structure, but it also introduces the posterior problem

$p(z\mid x)$

.

A VAE connects the pieces this way:

observed x
   ↓
Encoder / Recognition Network
   ↓
variational approximation to posterior over z
   ↓
latent z
   ↓
Decoder / Generative Network
   ↓
generated or reconstructed x

Training connects this architecture to Variational Inference through maximizing the ELBO:

latent-variable representation
        ↓
intractable posterior
        ↓
variational approximation
        ↓
Encoder
        ↓
ELBO-based learning

This is why a VAE is more than an Encoder followed by a Decoder. Its structure turns posterior intractability and variational approximation into a trainable generative model.

Flow-based models: use invertibility to preserve tractability

Flow-based models start from a simple distribution and apply a sequence of invertible transformations to reach a more complex target distribution.

Invertibility is the critical structural constraint. Because the transformations can be reversed, probability density can be tracked through them, allowing the model to train with NLL.

architectural restriction
        ↓
invertible transformation
        ↓
density remains computable
        ↓
likelihood-based training becomes possible

The constraint is part of the solution to the tractability problem: it introduces a structural restriction while keeping density evaluation possible.

GAN: learn the generation process instead of explicit density

GANs make a different choice. Instead of directly computing an explicit density function or normalizing constant, they learn the generation process itself.

The Generator tries to produce samples resembling real data, while the Discriminator tries to distinguish generated samples from real ones. The two networks train with opposing objectives in a minimax problem.

latent input
    ↓
Generator
    ↓
generated sample ──┐
                   ├──> Discriminator
real sample ───────┘

GANs use this adversarial minimax process to construct an implicit generative model rather than directly evaluating an explicit model density.

Diffusion: connect noise and data through many steps

Diffusion models take another route. Rather than directly handling the transformation between a simple distribution and a complex data distribution in one step, they connect the two through a sequence of noise levels.

The forward process gradually adds Gaussian noise to real data:

As the process continues, the original data structure gradually disappears and approaches a simple isotropic Gaussian distribution.

Generation runs in the opposite direction:

data
  ↓
add noise
  ↓
add more noise
  ↓
...
  ↓
Gaussian-like noise

generation:

Gaussian noise
  ↓
learned reverse step
  ↓
learned reverse step
  ↓
...
  ↓
data-like sample

Instead of using one direct transformation, diffusion models learn a reverse process that progressively reconstructs data structure from noise.

The score view of diffusion

Score-based generative models can describe this process using the score function:

The score indicates the direction in which the log probability increases.

Two representative approaches are SMLD and DDPM.

SMLD, or Score Matching with Langevin Dynamics, estimates the score at multiple noise scales and uses Langevin Dynamics to sample while the noise scale decreases.

DDPM, or Denoising Diffusion Probabilistic Model, learns a probabilistic reverse model for the steps of the forward noise-corruption process.

Their emphasis can be summarized as:

SMLD:
score estimation
      +
Langevin Dynamics

DDPM:
forward corruption steps
      +
learned reverse probability model

The details differ, but both approaches connect a simple noise distribution to a complex data distribution through multiple noise levels or steps.

That iterative structure also creates a practical tradeoff. Because generation requires multiple diffusion steps, sampling speed can become an important limitation.

One probability problem, different structural answers

The major generative model families are easier to compare when viewed as different computational responses to the same high-dimensional distribution problem:

• Autoregressive models factorize a joint distribution into sequential conditional distributions.
• VAEs introduce latent variables and use Variational Inference to approximate a difficult posterior while training through the ELBO.
• Flow-based models impose invertibility so density evaluation and NLL-based learning remain possible.
• GANs avoid direct explicit-density evaluation and learn the generation process through adversarial minimax training.
• Diffusion models transform generation into a sequence of noise-corruption and reverse-process steps.

The normalizing constant fits into the same picture. Energy-Based Models work with energy-based probability distributions in which the normalizing constant must be handled or approximated. Autoregressive models, flow-based models, and VAEs use computationally manageable structural constraints or model constructions. GANs avoid direct computation of explicit density and its normalizing constant by learning the generation process, while score-based diffusion uses the score rather than directly computing the normalizing constant.

Different architectures make different choices, but the underlying problem is the same: preserve enough flexibility to represent complex data while keeping generative modeling computationally feasible.

A developer mental model for generative architectures

When you encounter a generative architecture, start with three questions rather than its layer diagram.

Representation: What distribution or latent structure is the model trying to represent?

Learning: What objective reduces the difference between the model and the observed data distribution?

Inference: What hidden variables or generating processes must be inferred from observations, and can those quantities be computed directly?

Then look at the structural choice the model uses to keep the relevant computations manageable:

high-dimensional data distribution
          ↓
representation must be expressive
          ↓
probability or posterior computation becomes difficult
          ↓
choose a computational strategy
          ↓
factorize
approximate
restrict
avoid explicit density
or learn a multi-step reverse process

No single family removes every tradeoff. In the comparison considered here, VAEs and flow-based models can show weaknesses in generation quality, GANs can face difficulties with generation diversity, and diffusion models can pay for their iterative generation process with slower sampling.

So the useful comparison is not simply “”Which model is better?””

It is:

Which computational problem does the model solve through its structural choice, and what does that choice cost in quality, diversity, or speed?

Takeaway

Generative modeling is not fundamentally about producing realistic-looking samples. It is about learning an unknown high-dimensional data distribution while keeping Representation, Learning, and Inference computationally manageable.

Autoregressive models, VAEs, flow-based models, GANs, and diffusion models look different because they make different structural choices around the same underlying problem. Once you identify whether an architecture factorizes a distribution, approximates an intractable quantity, imposes a structural constraint, avoids explicit density evaluation, or learns a multi-step reverse process, the landscape of deep generative models becomes much easier to reason about.

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/gan-en/