Chapter 5

Constraint-Driven Optimization

Learn to incorporate runtime constraints and validation directly into your optimization process for robust, high-quality DSPy systems.

📋 Prerequisites

  • Previous Section: Choosing Optimizers
  • Chapter 3: Assertions Module (for understanding constraints)
  • Concept: Basic optimization theory
  • Level: Advanced

Introduction

Constraint-driven optimization extends DSPy's standard optimization by ensuring that your compiled programs not only maximize a metric (like accuracy) but also adhere to strict requirement boundaries (like format, length, or safety).

💡

The Shift: Instead of just asking "Did it get the right answer?", we ask "Did it get the right answer and follow the rules?"

Core Concepts

1. Constraint-Aware Metrics

The simplest way to enforce constraints is to embed them into your metric function. If a prediction violates a hard constraint, it receives a score of 0, regardless of the content's quality.

def constrained_metric(example, pred, trace=None):
    # 1. Check Hard Constraints (Format, Type)
    if not validate_format(pred):
        return 0.0  # Immediate failure
        
    # 2. Check Soft Constraints (Style, Length)
    penalty = 0.0
    if len(pred.output) > 500:
        penalty += 0.1
        
    # 3. Calculate Base Score
    base_score = answer_f1_score(example, pred)
    
    return max(0.0, base_score - penalty)

2. Hard vs. Soft Constraints

Type Description Example Handling Strategy
Hard Must pass for the output to be valid. Valid JSON syntax, specific output schema. Return 0 in metric if failed. Use Assertions.
Soft Preferences that improve quality. Conciseness, tone, specific vocabulary. Apply scalar penalties to the metric score.

🚀 Optimization Strategies

1. Constraint-Guided Example Selection

When using BootstrapFewShot, you can ensure that only examples satisfying your constraints are selected as demonstrations.

2. Progressive Constraint Enforcement

For difficult constraints, it helps to start leniently and tighten requirements over time. This prevents the optimizer from getting stuck early on when few candidates satisfy all rules.

# Concept: Gradual tightening
levels = [
    [validate_format],                  # Level 1: Just format
    [validate_format, validate_length], # Level 2: + Length
    [validate_format, validate_length, validate_style] # Level 3: All
]

for constraints in levels:
    # Optimizing loop...
    pass

Advanced Techniques

Constraint Transfer Learning

If you have successfully optimized constraints for one task (e.g., "Always output JSON"), you can transfer those learned patterns (instructions or demonstrations) to a new, related task.

RAG System Optimization

For RAG systems, constraints often involve citation handling and hallucination prevention.

constraints = {
    'min_evidence': 2,       # Cite at least 2 sources
    'max_hallucination': 0.1 # <10% hallucinated content
}

📊 Monitoring and Analysis

It's crucial to track Violation Rates during optimization. If 90% of your candidates are failing a specific constraint, you may need to:

  • Relax the constraint temporarily.
  • Improve the base prompt instructions.
  • Provide better manually-written demonstrations that satisfy the constraint.
Next: Reflective Prompt Evolution