Autoregressive Image Generation without Vector Quantization

Challenging Discrete Tokenization

• 💡 Traditional autoregressive image generation relies on Vector Quantization (VQ) to convert continuous image data into discrete tokens.
• ⚠️ This discretization process is inherently lossy, leading to information loss and potential artifacts in generated images.
• 🧩 Training VQ models can be tricky due to sensitive gradient approximations, resulting in unstable training and suboptimal visual vocabularies.

Introducing Diffusion Loss

• 🚀 The paper proposes a novel approach that eliminates the need for discrete tokens by working directly with continuous representations of images.
• 🧠 They leverage diffusion models to model the per-token probability distribution in a continuous-valued space.
• ✅ Instead of categorical cross-entropy, a new Diffusion Loss function is defined to train the model effectively.

How Continuous Modeling Works

• 🎯 The autoregressive model predicts a continuous vector (Z) for each token, acting as a guide.
• 🔬 A denoising network is trained to predict noise added to a token, conditioned on the Z vector and current noise level.
• 🔑 Diffusion Loss measures the denoising network's ability to predict noise, ensuring Z contains necessary information for token reconstruction.

Enhanced Autoregressive Capabilities

• ✨ The method extends to Masked Autoregressive (MAR) models, allowing for simultaneous prediction of multiple tokens in flexible orders.
• 🌐 This enables the model to utilize bidirectional attention, learning richer relationships across different parts of an image.
• 🛠️ The approach offers flexibility, as Diffusion Loss can be applied to continuous representations even with existing VQ models or various tokenizers like KL8 and Consistency Decoder.

Significant Performance Gains

• 📈 Removing VQ leads to higher quality images with fewer artifacts, demonstrated by lower FID scores compared to traditional methods.
• ⚡ The MAR model with Diffusion Loss achieves remarkable speed, generating images in under 0.3 seconds per image.
• 📊 Ablation studies confirmed efficiency, showing the denoising MLP doesn't need to be large and around 100 diffusion steps is optimal.

Broader Implications

• 💡 This research combines autoregression and diffusion in a groundbreaking way, challenging the assumption that autoregressive models are limited to discrete sequences.
• 🌍 The authors suggest this continuous modeling approach could be applied to other continuous data domains, such as audio and video.
• 🔑 It represents a paradigm shift in generative modeling, moving away from reliance on discretization for high-quality generation.