Diffusion Model

What is a diffusion model and why is it NOT the vapour-diffusion model from building physics?

A diffusion model in machine learning is a generative AI system that creates images by learning to reverse a process of adding random noise. It bears no relationship to the diffusion model in building physics, which describes how water vapor moves through materials under humidity gradients. The collision of names creates confusion in architecture: one is a computational technique for generating visual content, the other a physical process for assessing condensation risk. When you see 'diffusion model' in the context of AI image generation tools (Stable Diffusion, DALL-E, Midjourney), it refers exclusively to the noise-reversal process described here. When discussing envelope performance or air-barrier design, diffusion means vapor transport. Context determines meaning; never assume they are the same.

The denoising-based approach that diffusion models use has become the dominant architecture for generative AI image synthesis. Unlike earlier techniques such as generative adversarial networks (GANs), which train competing networks and are notoriously unstable, diffusion models scale reliably and produce high-quality outputs. They have become the foundation of commercial tools (Stable Diffusion, DALL-E 3) and represent a significant shift in how AI systems approach content generation. For architects, understanding this technique is useful when evaluating whether a generated image can be trusted for design communication or whether it remains a tool only for early-stage mood exploration.

How does the forward and reverse noising process work?

The diffusion process consists of two complementary operations. The forward process, used only during model training, takes a real image and progressively adds Gaussian random noise to it, step by step, until nothing remains but pure noise. After thousands of noise-addition steps, the original image is completely obscured. During training, the model learns to predict what noise was added at each step, learning the statistical relationship between images and noise. This training is done once and is never repeated during use.

The reverse process is how images are generated. It starts with random noise (pure static) and iteratively removes small amounts of noise, guided by a learned denoising network. If you provide a text prompt ('a minimalist residential courtyard, concrete floors, skylights'), the model's denoising predictions are conditioned by that prompt: it predicts noise that, when subtracted, moves the random initialization toward an image that matches the description. After hundreds of denoising steps, an image emerges from the noise. Each step is imperceptibly small; the cumulative effect is coherent image synthesis. This is why diffusion models are computationally expensive: generating one image requires 20 to 100 forward passes through a large neural network.

The advantage of this approach is stability: the model learns a smooth path from noise to image, making training and generation more predictable than GAN-based methods. The disadvantage is speed and computational cost. Generating a single high-resolution image typically takes 10 to 30 seconds on consumer hardware, compared to instantaneous output from some alternative methods. For interactive design workflows, this latency can be frustrating.

What are conditioning and prompts, and how do they guide image generation?

Conditioning is any input signal that guides the diffusion model's denoising process toward desired output. The most common form is text conditioning: a natural-language prompt that describes what you want. The model encodes your text into a numerical representation and uses it to bias each denoising step. A prompt like 'a Scandinavian timber-frame house, snow, evening light' influences every noise-removal prediction, pushing the result toward architectural styles, materials, and lighting that statistically correlate with those words in the training data.

Text prompts are powerful but limited. They provide statistical guidance, not logical constraints. You cannot tell a diffusion model 'make all windows the same size' or 'maintain 7-meter floor height', and have it obey. The model treats every detail as an independent pattern to match. If the training data contains mostly images of residential buildings with varied window sizes, the model will generate varied windows regardless of how many times you ask for regularity. This is a structural limitation of the technique, not a prompt-writing problem.

Beyond text, other conditioning modalities allow geometric control. ControlNet is the most architecturally relevant: you provide a structural guide (a depth map from your 3D model, an edge skeleton, or a spatial layout) and a text prompt, and the model generates images that respect your guide while adding materials, lighting, and detail. This hybrid approach is closer to architectural visualization because your actual geometry is preserved; only surfaces and lighting are generated. A practical workflow: extract a depth or normal-map render from your 3D model, feed it to ControlNet with a prompt like 'morning light, stone and glass facade', and receive variations that maintain your massing while exploring aesthetic direction.

Conditioning strength is adjustable: you can dial text influence up or down. At low strength, the model ignores your prompt and uses the geometric guide; at high strength, text dominates and geometry is warped to match the description. Finding the right balance requires iteration. Most architectural workflows use 60-80% text strength with ControlNet to blend design intent (your prompt) with structural integrity (your model).

What are diffusion models honestly good at, and bad at, in architectural practice?

Diffusion models excel at rapid mood and material exploration. You can generate dozens of variations in minutes: 'this facade in copper', 'this facade weathered, thirty years old', 'this courtyard at dawn vs. dusk'. For a design at concept stage before geometry is locked, exploring material direction and lighting atmosphere is valuable. Clients often respond intuitively to visual mood; generated images can communicate scale, proportion, and ambiance faster than verbal description. Diffusion is also powerful for populating context: surrounding buildings, landscaping, entourage figures, and sky conditions can be generated to place your design in a plausible environment.

Diffusion models are catastrophically bad at what matters most in architectural documentation: geometric consistency, dimensional accuracy, and the enforcement of design rules. Windows shift in position and size across a generated facade. Proportions of storeys vary. Symmetry disappears. Material patterns become random. A colonnade becomes a jumble of columns at inconsistent spacing. Any rule your design depends on (rhythm, proportion, alignment) is violated. This is not a quality-of-prompt issue; it is structural to the technique. The model treats every pixel as an independent prediction task guided by text, without enforcing the global constraints that architecture requires.

For client communication, this is risky. A generated image of your design that looks compelling but violates your geometry can mislead. A client might approve aesthetic direction based on a generated image that is impossible to build. If that generated image becomes the reference for expectations, you have created legal and professional liability. The safest approach is to reserve generated images for internal exploration and early-concept communication explicitly labeled as AI-generated mood studies. For any image meant to represent your actual design, render from your validated 3D model, where every dimension and rule you set is enforced.

How does a diffusion model differ from a generative adversarial network (GAN)?

Both are generative models, but they learn and generate differently. A GAN trains two competing networks: a generator that creates fake images, and a discriminator that tries to detect fakes. The generator learns to fool the discriminator; the discriminator learns to spot fakes. This adversarial process can produce fast, stable-looking results, but training is notoriously unstable and mode collapse is common (the generator forgets how to generate certain categories). A diffusion model, by contrast, trains a single network to predict noise at each step of a forward process. This is mathematically simpler and more stable to train.

In practice, diffusion models produce higher quality, more diverse, and more controllable outputs than GANs. They scale better with data size and model parameters. GANs generate images in one shot (fast but less controllable); diffusion takes many steps (slower but more flexible, allowing ControlNet conditioning and step-by-step refinement). For architectural visualization, diffusion's controllability and quality make it more relevant. Most commercial tools (Stable Diffusion, DALL-E, Midjourney) now use diffusion rather than GAN architectures.

How should architects approach diffusion models in professional practice?

Treat diffusion-generated images as internal exploration tools and concept communication, never as depictive assets for final presentation. Use them to explore mood, material palette, and spatial atmosphere when your design direction is still forming. Do not feed generated images to clients as if they depict your actual design; the geometric unreliability creates false expectations and professional risk. Label all generated imagery clearly as 'AI-generated concept' to avoid confusion.

If geometry-aware generation appeals to your workflow, ControlNet-based systems (RunwayML, Comfy UI) allow you to feed depth or edge maps from your 3D model alongside text prompts, blending your structural intent with aesthetic generation. This is closer to responsible use but still requires judgment: generated surfaces may still hide geometry errors or create details inconsistent with your model.

Be transparent with clients about what is generated and what is rendered from your model. Be explicit about copyright: images created from training data that was scraped without permission carry legal risk if you publish them. The legal landscape remains unsettled, and Slovakia has not yet established clear case law on AI-generated work ownership or training-data liability. Until clarity emerges, treating generated images as internal tools only is the safer professional stance.