From Pixel Histograms to Visual Autoregression: How Images Learn to See Themselves

In this post we are going to walk through image generation evolution to understand how neural networks learn to see. Across the evolution of large language models: all the way from most popular token in Markov chains to GPT, compression pressure forces models to increase prediction power. In vision with analogy to LLMs, we show that by changing representation we achieve better intelligence, i.e. higher quality generation. The entire evolution is being able to predict the next token with a better accuracy. As a bonus, we get to see what each representation captures and at which cost.

The core insight is a direct analogy from language models: prediction pressure forces spatial compression. Spatial compression forces hierarchical abstraction. Hierarchical abstraction is what we call seeing.

Stage 0: Random Pixels

We start by predicting random pixels. To do so we sample each pixel uniformly from 0 to 255 RGB, that generates television static. This is the visual equivalent of the same mechanism of sampling random words from a dictionary.

Stage 1: Color Histograms

As a next step, we now sample pixels weighted by the color distribution of a real image. And this is an equivalent of a visual unigram, where histogram compresses 200,000 pixel (256x256x3 image) values into 256 frequency counts. The compression ratio is enormous, so what survives that compression is a palette, but we still have zero spatial structure. Visually we see that colors that are more frequently occuring have a higher chance of being selected.

Stage 2: Local Filters

Now we give the model a concept of neighbors. Instead of treating each pixel in isolation, we look at small local patches. This is what classical computer vision was focusing for decades from 1960s: like edge detectors, Gabor filters, texture synthesis approaches, etc. More recently, Portilla and Simoncelli (2000) showed that a richer set of local statistics (correlations between filter responses across scales and orientations) could produce remarkably convincing textures, but the approach still could not generate objects or scenes.

The result though is significantly better than random histograms. Local filters can produce convincing textures: wood grain, fur, woven fabric, grass. The generated patches look locally plausible, especially when zoomed any small region, it could pass for the real thing as the model has learned neighbourhoods. But since there is no global coherence, the zoomed out picture doesn’t make any sense.

This is the exact analog of Markov chains in text. A bigram model generates locally plausible word pairs that collapse over longer distances. A trigram model produces convincing sentence fragments that contradict themselves a few sentences later. Local filters produce convincing texture patches that never organize into meaningful wholes. In both cases, the model has memorized local co-occurrence statistics and has zero capacity to represent global structure.

The failure mode is also parallel. Markov text generators fall into loops: “the rabbit was not a moment to be no use in the door.” Texture synthesis algorithms fall into tiling: the same 50x50 patch of grass repeats forever because the local matching procedure keeps finding the same nearest neighbor. So the visual Markov loop can be thought as a texture tile.

The $O(V^n)$ Wall and Why Compression Discovers Hierarchy

There is also a visual analogy for the storage problem that overwhelms n-gram language models. If you want to build a lookup table of all possible 3x3 grayscale patches, you need $256^9 \approx 5 \times 10^{21}$ entries. That is already impossibly large. For color patches it becomes $256^{27}$. Extending to 8x8 patches (which are still far too small to capture objects) gives $256^{192}$.

Just as language models hit the $O(V^n)$ wall where expanding the n-gram context creates exponentially more table entries, visual models hit the same wall where expanding the patch size creates exponentially more pixel combinations. So both domains independently evolve to the same direction: replace the discrete lookup table with a continuous learned representation that compresses the space.

This is the key section of the post. The argument for language is well-known: the $O(V^n)$ wall forces Word2Vec-style embeddings, and prediction pressure accidentally discovers king - man + woman = queen.

The argument for vision is the same. A system that recognizes cats cannot memorize images. The same cat under different lighting, angle, and background produces pixel arrays as different as a cat and a truck, so the lookup table becomes impossible. So the system must learn a representation that throws away everything irrelevant (pixels, lighting, background) and keeps only what matters (pointed ears, whiskers, slit pupils).

The compression that best serves recognition turns out to be compositional. Edge detectors are reusable across all categories, so they emerge first. Texture detectors reuse across many categories, so they emerge next. Part detectors reuse within category families. Object detectors are category-specific.

This hierarchy is not designed. It emerges because it is the most parameter-efficient compression for prediction. And it is the visual version of king - man + woman = queen: take the feature for a tabby cat, subtract the tabby texture, add black fur, and you land near the feature for a black cat. “Cat-ness” and “fur pattern” are separable dimensions in feature space, discovered by compression, never requested.

The early layers of a CNN (LeCun et al., 1989; Krizhevsky et al., 2012) reinvent Gabor filters (Hubel and Wiesel, 1962). The deep layers learn part detectors they were not optimized for. Zeiler and Fergus (2014) made this visible by deconvolving activations back to pixel space. Alain and Bengio (2017) proved it with linear probes: freeze the CNN, train a tiny classifier on tasks it was never trained on (indoor/outdoor? water present? natural lighting?), and the probe achieves high accuracy. The compressed representation encodes concepts that were never part of the training signal.

Frozen CNN Feature [d=2048]  ──>  [ Simple Linear Probe ]  ──>  Indoor or Outdoor?

This is another example of how prediction pressure creates knowledge that was never optimized for.

Stage 3: ViT and the Language Bridge

Vision Trasnformers ViT (Dosovitskiy et al., 2020) make the language analogy exact. To generate a sequence of input tokens, the image is divided into 16x16 patches (words), that are embed into a vector of the image (i.e.sentence). Then they run self-attention across all patches, where self-attention connects all patches together. The architecture is identical to a text transformer except the attention maps are now spatial. Because attention is global from the first layer, ViT captures long-range dependencies that CNNs need many stacked layers to approximate. Linear probes on ViT features encode not just object identity but spatial relationships, depth ordering, and geometric consistency.

However, unlike text, images have a natural multi-scale decomposition. This means images offer a choice for autoregressive generation. PixelCNN (van den Oord et al., 2016) goes raster-scan: left to right, top to bottom, committing to local details before knowing the global structure. The result is horizontal banding artifacts. Coarse-to-fine generation avoids this entirely: decide the layout first, add detail later, never generate a region without knowing the global context.

Stage 4: VAR (Visual AutoRegression)

This naturally brings us to Visual Autoregressive models, or VARS.

A CNN takes a full-resolution image and compresses it layer by layer, while VAR starts from the coarsest description and expands scale by scale: a single token becomes layout, then objects, then textures, then a full image - which is a generation! So in a sense, VAR is the CNN’s compression hierarchy run backward: if discrimination results in compression, generation is decompression.

More specifically, VAR (Tian et al., 2024) predicts the next scale, not the next pixel. Generation starts at 1x1 (global color), then 2x2 (spatial layout), then 4x4, 8x8, doubling each time, conditioned on all coarser scales via attention.

Each scale doubles the resolution, followe by details. VAR demonstrates power-law scaling similar to LLMs, suggesting coarse-to-fine is the right inductive bias for visual autoregression the way left-to-right is right for text. The field has already moved past class-conditional: VAR-CLIP enables text-to-image, Infinity (Li et al., 2024) scales to 1024x1024, HART (Tang et al., 2024) hybridizes discrete and continuous tokens, xAR explores flexible resolution schedules.

The Other Roads: GANs and Diffusion

It’s important to mention two other families reached photorealism first. Both independently discovered the same coarse-to-fine hierarchy.

StyleGAN (Karras et al., 2019; 2020) has an explicit multi-resolution architecture: 4x4 constant upsampled through 8x8, 16x16, 32x32, and so on. Coarse styles control pose and shape, while fine styles control skin pores and hair texture. This mirrors VAR’s scale hierarchy, but the training signal is adversarial (fool the discriminator) rather than predictive. The representations are less interpretable because they were optimized for deception, not prediction.

Diffusion models (Ho et al., 2020; Rombach et al., 2022) are not autoregressive at all. They predict noise and subtract it. But Dieleman (2023) showed that denoising implicitly follows a coarse-to-fine schedule: high noise destroys high frequencies first, so early denoising steps recover global composition and late steps recover fine detail. Diffusion does coarse-to-fine without being told to.

What’s interesting here is that since the multi-scale structure of images is not an artifact of any particular architecture, but rather a property of the visual world, three completely different training objectives discovered it independently.

The Ladder

To summarize the word-to-vision model evolution analogy, we show for each row how the model answers on what does survive the compression at this level. The modality is invariant as ompression finds hierarchical structure in both language and images because both describe a world that is hierarchically structured. For example, all models: DALL-E (Ramesh et al., 2021), Chameleon (Meta, 2024), and Infinity already process text and image tokens with the same transformer architecture regardless of the modality input.

Below is an Interactive Playground, where you can switch between stages, select a different source image and watch the generation process at each lebel. At Stage 5, step through VAR’s scale-by-scale generation and see coherence emerge from compression.

References

Alain, G. and Bengio, Y. (2017). Understanding intermediate layers using linear classifier probes. ICLR Workshop.

Canny, J. (1986). A computational approach to edge detection. IEEE TPAMI, 8(6), 679-698.

Dieleman, S. (2023). Diffusion is spectral autoregression. Blog post, sander.ai.

Dosovitskiy, A. et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021.

Efros, A. A. and Leung, T. K. (1999). Texture synthesis by non-parametric sampling. ICCV, 1033-1038.

Goodfellow, I. et al. (2014). Generative adversarial nets. NeurIPS, 27.

Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. NeurIPS, 33.

Hubel, D. H. and Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. Journal of Physiology, 160(1), 106-154.

Karras, T., Laine, S., and Aila, T. (2019). A style-based generator architecture for generative adversarial networks. CVPR, 4401-4410.

Karras, T. et al. (2020). Analyzing and improving the image quality of StyleGAN. CVPR, 8110-8119.

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. NeurIPS, 25.

LeCun, Y. et al. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4), 541-551.

Li, D. et al. (2024). Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv:2412.04431.

Portilla, J. and Simoncelli, E. P. (2000). A parametric texture model based on joint statistics of complex wavelet coefficients. IJCV, 40(1), 49-71.

Ramesh, A. et al. (2021). Zero-shot text-to-image generation. ICML, 8821-8831.

Rombach, R. et al. (2022). High-resolution image synthesis with latent diffusion models. CVPR, 10684-10695.

Sobel, I. (1968). An isotropic 3x3 image gradient operator. Stanford AI Laboratory.

Sohl-Dickstein, J. et al. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. ICML, 2256-2265.

Swain, M. J. and Ballard, D. H. (1991). Color indexing. IJCV, 7(1), 11-32.

Tang, Y. et al. (2024). HART: Efficient visual generation with hybrid autoregressive transformer. arXiv:2410.10812.

Tian, K. et al. (2024). Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurIPS, 37.

van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016). Pixel recurrent neural networks. ICML, 1747-1756.

Vaswani, A. et al. (2017). Attention is all you need. NeurIPS, 30.

Zeiler, M. D. and Fergus, R. (2014). Visualizing and understanding convolutional networks. ECCV, 818-833.

Citation

If you find this work useful, please cite:

@article{levonyan2026pixelhistograms,
  title={From Pixel Histograms to Visual Autoregression: How Images Learn to See Themselves},
  author={Levonyan, Karine},
  year={2026},
  url={https://karinelevonyan.github.io/blog/2026/from-pixel-histograms-to-visual-autoregression/}
}