Harsh Tomar

Architecture Components

• SigLIP Vision Encoder: Processes images into embeddings using Vision Transformer with 16×16 patches, generating 196 tokens for 224×224 images with learned positional embeddings
• Gemma Language Model: Decoder-only architecture with RMSNorm, GeLU activations, Rotary Position Encoding (RoPE), and grouped-query attention for efficiency
• Rotary Position Encoding: Sophisticated position encoding applying rotation matrices to query/key vectors, enabling extrapolation beyond training context length
• Grouped-Query Attention: Reduces computational requirements by sharing key-value heads across multiple query heads while maintaining quality
• KV-Cache Mechanism: Efficient autoregressive inference with cached key-value pairs for faster generation

Implementation Features

• Complete from-scratch implementation in PyTorch with detailed comments
• Pre-trained weight loading from Hugging Face (6GB model)
• Visual question answering, image captioning, and multimodal chat capabilities
• Inference script with top-p sampling and temperature control
• Comprehensive documentation with architecture diagrams and research papers

Architecture Implementation

• Patch Embedding: Divides images into 16×16 non-overlapping patches, linearly projects to embedding dimension using Conv2d for efficiency
• Multi-Head Self Attention: Jointly attends to information from different representation subspaces with scaled dot-product attention
• MLP Block: Feed-forward network with GELU activation applied after attention mechanism
• Transformer Block: Combines attention and MLP with residual connections and layer normalization for stable training
• Class Token: Learnable embedding prepended to sequence for classification, similar to BERT's [CLS] token

Implementation Variants

• CIFAR-10: Lightweight model (384 dim, 6 blocks, 6 heads) for educational purposes with attention visualization
• ImageNet: Full-scale model (768 dim, 12 blocks, 12 heads) with flash attention and distributed training support
• Data augmentation strategies (RandAugment, Mixup, CutMix)
• Different pooling strategies (CLS token vs average pooling)
• Comprehensive training analysis with and without augmentation

LoRA Architecture

• Low-Rank Adaptation: Injects trainable rank decomposition matrices (A, B) into frozen pre-trained weights W, computing y = x(W + AB)
• Parameter Efficiency: Trains <1% of parameters with rank r typically 8, 16, or 32
• Scaling Factor: Uses α to control update magnitude with rank-stabilized variant (α·AB/√r)
• Layer Support: LoRALinear, LoRAEmbedding, LoRAConv2d for different layer types
• Memory Reduction: 65% reduction for BERT, 50% for LLaMA-7B

QLoRA Innovations

• 4-bit NF4 Quantization: Normal Float data type optimized for LLM weight distributions
• Double Quantization: Quantizes quantization constants for additional memory savings
• Paged Optimizers: Offloads optimizer states to CPU reducing GPU memory usage
• BF16 LoRA Training: Maintains adapters in BF16 precision for numerical stability
• Memory Efficiency: 85% reduction enabling LLaMA-65B fine-tuning on consumer GPUs

Inference-Time Compute Scaling

• Zero-Shot Prompting: Applied to Llama 3.2 built from scratch for baseline reasoning capabilities
• Beam Search: Demonstration of search-based decoding strategies for improved output quality
• Method Comparison: Comparing different inference-time compute methods against baseline performance
• Model Size Effects: Analysis of how model size impacts accuracy with chain-of-thought reasoning
• Scaling Test Time Compute: Research on computational strategies during inference for better reasoning

Reinforcement Learning Approaches

• Exploration of RL techniques for improving reasoning capabilities
• Policy optimization for multi-step reasoning tasks
• Reward modeling for reasoning quality assessment
• Integration of RL with language model pre-training

Implementation Notebooks

• Applying Zero-Shot Prompting to Llama 3.2 Built From Scratch
• Beam Search Demonstration with detailed visualizations
• Inference-Time Compute Scaling: Comparing Different Methods
• Effect of Model Size on Accuracy with Chain-of-Thought Reasoning