Compositional Visual Retrieval

Potential Distribution Shifts:

• Novel object combinations not seen in training
• Different linguistic phrasings of modifications
• Variations in image quality, lighting, or backgrounds
• Objects in unusual positions or orientations

How LoRA Works: Low-Rank Parameter Updates

Instead of updating weight matrix \(W \in \mathbb{R}^{d \times d}\) directly, LoRA adds a low-rank decomposition:

where \(B \in \mathbb{R}^{d \times r}\), \(A \in \mathbb{R}^{r \times d}\), and \(r \ll d\) (rank=32 ≪ 768)

🎯 Result: LoRA allowed us to adapt FILIP to the specific task (compositional retrieval) while preserving OpenCLIP's broad visual understanding. This proved crucial when test queries used novel object combinations or unexpected phrasings—the model fell back on its pretrained knowledge instead of failing completely.

For each anchor sample with objects {A, B, C}:

• Hard Negative: Must have 2 objects from {A, B, C} + 1 different object D
• Example: Anchor = "chair, train, hat" → Hard Negative = "chair, train, lamp"
• Overlap: |keywords_anchor ∩ keywords_negative| = 2

🔧 Hard Negative Batch Sampler

def create_hard_negative_batch(dataset, batch_size=16):
    """
    Create batches with hard negatives (2 matching objects, 1 different)
    """
    # Index dataset by keywords
    keyword_index = defaultdict(list)
    for idx, (image, caption) in enumerate(dataset):
        keywords = extract_keywords(caption)
        # Store by keyword combinations
        for k in keywords:
            keyword_index[k].append((idx, keywords))
    
    batches = []
    used_indices = set()
    
    for idx, (image, caption) in enumerate(dataset):
        if idx in used_indices:
            continue
            
        anchor_keywords = extract_keywords(caption)
        if len(anchor_keywords) != 3:  # Skip if not exactly 3 objects
            continue
        
        # Find hard negatives: 2 matching + 1 different
        batch = [(idx, anchor_keywords)]
        
        for other_idx, other_keywords in dataset:
            if other_idx == idx or other_idx in used_indices:
                continue
            
            overlap = len(anchor_keywords & other_keywords)
            
            # Hard negative: exactly 2 objects match
            if overlap == 2:
                batch.append((other_idx, other_keywords))
                used_indices.add(other_idx)
                
                if len(batch) >= batch_size:
                    break
        
        if len(batch) >= batch_size // 2:  # At least half filled
            batches.append(batch)
            used_indices.add(idx)
    
    return batches

# Example batch
# Anchor:   "chair, train, hat"      (positive)
# Hard Neg: "chair, train, lamp"     (2 match: chair, train | 1 diff: lamp vs hat)
# Hard Neg: "chair, motor, hat"      (2 match: chair, hat | 1 diff: motor vs train)
# Hard Neg: "bicycle, train, hat"    (2 match: train, hat | 1 diff: bicycle vs chair)

Visual Example of a Hard Negative Batch

Hypothesis: By exposing the model to more challenging negatives, we expected 2-3% improvement in Top-1 accuracy, especially on edge cases where objects differ by only one item.

Recommendation: If implementing hard negative mining, invest time in a robust selection algorithm (embedding-based) and use soft weighting. Our simple keyword approach was insufficient for this task.

🎯 How Gating Works: The sigmoid gate (α) decides how much to "move" from reference image toward text direction. For minor changes (α≈0.2), mostly preserves reference. For major changes (α≈0.8), heavily modifies embedding.

🔀 Parallel Gating Path Architecture

The gating mechanism runs in parallel to the main MLP path to dynamically weight the combination

Mathematical Formulation

where \(B\) = batch size, \(N\) = database size, \(\tau\) = temperature (learned via logit_scale)

PyTorch Implementation

# Compute similarity scores (logits)
logits = pred @ db_features.T  # [B, N] - cosine similarity
logits = logits * combiner.logit_scale.exp()  # scale by learned temperature

# Ground truth: diagonal elements are positive pairs
labels = torch.arange(B, device=pred.device)  # [0, 1, 2, ..., B-1]

# Cross-entropy treats row i, column i as positive
loss1 = F.cross_entropy(logits, labels)

🎯 Why This Works: For each predicted embedding, InfoNCE maximizes similarity with its correct target (numerator) while minimizing similarity with all other targets in the database (denominator). This prevents mode collapse by forcing the model to maintain distinct embeddings for different compositions.

Mathematical Formulation

where \(\mathbf{z}_i^\text{pred} \in \mathbb{R}^{768}\) is the predicted embedding, \(\mathbf{z}_i^\text{target} \in \mathbb{R}^{768}\) is the ground truth

PyTorch Implementation

# Direct L2 distance between predicted and target
loss2 = F.mse_loss(pred, target_feat)

# Equivalent to:
# loss2 = torch.mean((pred - target_feat) ** 2)

🎯 Why This Works: While InfoNCE focuses on relative similarities (ranking), MSE provides absolute positioning. This ensures predictions don't just rank correctly but actually land in the right region of embedding space. Without MSE, predictions might correctly distinguish targets but be geometrically far from ground truth.

Mathematical Formulation

where \(D_{ij}^\text{pred} = \|\mathbf{z}_i^\text{pred} - \mathbf{z}_j^\text{pred}\|_2\) and \(D_{ij}^\text{target} = \|\mathbf{z}_i^\text{target} - \mathbf{z}_j^\text{target}\|_2\)

PyTorch Implementation

# Compute pairwise distance matrices
dist_pred = torch.cdist(pred, pred)              # [B, B]
dist_tgt = torch.cdist(target_feat, target_feat) # [B, B]

# Minimize difference between distance structures
loss3 = F.mse_loss(dist_pred, dist_tgt) * 0.1   # weight = 0.1

# This ensures:
# If target A and B are close → predictions A' and B' should be close
# If target A and C are far → predictions A' and C' should be far

🎯 Why This Works: Consistency loss prevents the model from creating "shortcuts" where all predictions cluster together (mode collapse). By forcing the predicted distance matrix to match the target distance matrix, we ensure the embedding space maintains meaningful structure. The 0.1 weight balances this regularization with the main objectives.

Total Loss Function

Why This Combination Prevents Collapse

🎯 Experimental Validation: Ablation study showed removing any single loss component degraded performance: -InfoNCE (-8.2%), -MSE (-5.7%), -Consistency (-3.1%). The combination creates a robust training signal that maintains clean, well-separated embedding space even after 50+ epochs without collapse.

Simple Python Script for Keyword Swapping:

def reverse_modification(text):
        """
        Reverse add/remove operations in modification text
        """
        # Define keyword mappings
        add_keywords = ['add', 'include', 'place', 'put in', 'insert']
        remove_keywords = ['remove', 'delete', 'get rid of', 'take out', 'eliminate']
        
        # Swap add → remove
        for add_word in add_keywords:
            text = text.replace(add_word, '')
        
        # Swap remove → add
        for remove_word in remove_keywords:
            text = text.replace(remove_word, '')
        
        # Replace temporary markers
        text = text.replace('', 'remove')
        text = text.replace('', 'add')
        
        return text
    
    # Example usage
    original = "remove the bike and add the car"
    reversed = reverse_modification(original)
    # Output: "add the bike and remove the car"

🎯 Why This Works: By swapping operations, the model learns symmetric understanding of compositional transformations. If it can map "A + remove X" → B, it should also map "B + add X" → A. This doubles the effective training data for compositional reasoning.

Distribution of Phrasing Patterns:

• ~85% Majority Pattern: "remove item X and add item Y"
• ~10% Variations: "get rid of item X and add item Y to this"
• ~5% Complex: "add item Y for this and get rid of item Z"

Paraphrasing Examples:

🎯 Impact: These paraphrases were especially valuable for Stage 2 (Combiner training), where the text encoder needed to handle diverse linguistic expressions of the same compositional intent. This augmentation improved robustness to test-time phrasing variations by ~4-5%.

Simple Python Script for Keyword Swapping:

def reverse_modification(text):
    """
    Reverse add/remove operations in modification text
    """
    # Define keyword mappings
    add_keywords = ['add', 'include', 'place', 'put in', 'insert']
    remove_keywords = ['remove', 'delete', 'get rid of', 'take out', 'eliminate']

    # Swap add → remove
    for add_word in add_keywords:
        text = text.replace(add_word, '')

    # Swap remove → add
    for remove_word in remove_keywords:
        text = text.replace(remove_word, '')

    # Replace temporary markers
    text = text.replace('', 'remove')
    text = text.replace('', 'add')

    return text

# Example usage
original = "remove the bike and add the car"
reversed = reverse_modification(original)
# Output: "add the bike and remove the car"

🎯 Why This Works: By swapping operations, the model learns symmetric understanding of compositional transformations. If it can map "A + remove X" → B, it should also map "B + add X" → A. This doubles the effective training data for compositional reasoning.

🔍 How it works: Three learnable latent vectors (queries) attend to all 196 patches (keys/values) via cross-attention. Each latent "pools" information from relevant patches, ideally capturing one object per latent.

💡 Key Lesson: Compositional reasoning benefits from preserving fine-grained spatial details. Aggressive compression trades off the very patch-level awareness that FILIP provides. The Combiner module succeeded by operating on the [CLS] token (global summary) while still leveraging the 196-patch encodings during FILIP pretraining.

TransE-Inspired Vector Arithmetic

Training Configuration:
- Batch Size: 32 (limited by dual encoder VRAM: ~22GB on RTX 4090)
- Epochs: 20
- Learning Rate: 5e-5 (AdamW with cosine warmup)
- Freeze image encoder initially, then apply LoRA (r=32)
- Alternate tasks per batch to prevent interference

Rationale: Multitasking ensures Text Encoder 1 maintains FILIP awareness while Text Encoder 2 learns compositional transformations independently.

💡 Key Lesson: Elegant architectures fail under hardware constraints. The dual encoder showed poor performance (62.5% baseline) and couldn't scale effectively. The final Combiner module succeeded by being simpler, smaller (single encoder + MLP), and more robust to linguistic variations through gated residual fusion instead of rigid vector arithmetic.

🎯 Takeaway: The gated residual fusion in the Combiner (\(\text{final} = \text{fused} + \alpha \cdot \text{text} + (1-\alpha) \cdot \text{img}\)) proved far more flexible than rigid vector arithmetic. The sigmoid gate \(\alpha\) adapts per query: low for minor changes, high for major transformations.