---

name: optimization-benchmark description: Comprehensive performance benchmarking, regression detection, automated testing, and performance validation. Use for CI/CD quality gates, baseline comparisons, and systematic performance testing.

Benchmark Suite Skill

Overview

This skill provides comprehensive automated performance testing capabilities including benchmark execution, regression detection, performance validation, and quality assessment for ensuring optimal system performance.

When to Use

• Running performance benchmark suites before deployment
• Detecting performance regressions between versions
• Validating SLA compliance through automated testing
• Load, stress, and endurance testing
• CI/CD pipeline performance quality gates
• Comparing performance across configurations

Quick Start

# Run comprehensive benchmark suite
npx claude-flow benchmark-run --suite comprehensive --duration 300

# Execute specific benchmark
npx claude-flow benchmark-run --suite throughput --iterations 10

# Compare with baseline
npx claude-flow benchmark-compare --current <results> --baseline <baseline>

# Quality assessment
npx claude-flow quality-assess --target swarm-performance --criteria throughput,latency

# Performance validation
npx claude-flow validate-performance --results <file> --criteria <file>

Architecture

+-----------------------------------------------------------+
|                   Benchmark Suite                          |
+-----------------------------------------------------------+
|  Benchmark Runner  |  Regression Detector  |  Validator   |
+--------------------+-----------------------+--------------+
         |                     |                    |
         v                     v                    v
+------------------+  +-------------------+  +--------------+
| Benchmark Types  |  | Detection Methods |  | Validation   |
| - Throughput     |  | - Statistical     |  | - SLA        |
| - Latency        |  | - ML-based        |  | - Regression |
| - Scalability    |  | - Threshold       |  | - Scalability|
| - Coordination   |  | - Trend Analysis  |  | - Reliability|
+------------------+  +-------------------+  +--------------+
         |                     |                    |
         v                     v                    v
+-----------------------------------------------------------+
|              Reporter & Comparator                         |
+-----------------------------------------------------------+

Benchmark Types

Standard Benchmarks

Benchmark	Metrics	Duration	Targets
Throughput	requests/sec, tasks/sec, messages/sec	5 min	min: 1000, optimal: 5000
Latency	p50, p90, p95, p99, max	5 min	p50<100ms, p99<1s
Scalability	linear coefficient, efficiency retention	variable	coefficient>0.8
Coordination	message latency, sync time	5 min	<50ms
Fault Tolerance	recovery time, failover success	10 min	<30s recovery

Test Campaign Types

1. Load Testing: Gradual ramp-up to sustained load
2. Stress Testing: Find breaking points
3. Volume Testing: Large data set handling
4. Endurance Testing: Long-duration stability
5. Spike Testing: Sudden load changes
6. Configuration Testing: Different settings comparison

Core Capabilities

1. Comprehensive Benchmarking

// Run benchmark suite
const results = await benchmarkSuite.run({
  duration: 300000,      // 5 minutes
  iterations: 10,        // 10 iterations
  warmupTime: 30000,     // 30 seconds warmup
  cooldownTime: 10000,   // 10 seconds cooldown
  parallel: false,       // Sequential execution
  baseline: previousRun  // Compare with baseline
});

// Results include:
// - summary: Overall scores and status
// - detailed: Per-benchmark results
// - baseline_comparison: Delta from baseline
// - recommendations: Optimization suggestions

2. Regression Detection

Multi-algorithm detection:

Method	Description	Use Case
Statistical	CUSUM change point detection	Detect gradual degradation
Machine Learning	Anomaly detection models	Identify unusual patterns
Threshold	Fixed limit comparisons	Hard performance limits
Trend	Time series regression	Long-term degradation

# Detect performance regressions
npx claude-flow detect-regression --current <results> --historical <data>

# Set up automated regression monitoring
npx claude-flow regression-monitor --enable --sensitivity 0.95

3. Automated Performance Testing

// Execute test campaign
const campaign = await tester.runTestCampaign({
  tests: [
    { type: 'load', config: loadTestConfig },
    { type: 'stress', config: stressTestConfig },
    { type: 'endurance', config: enduranceConfig }
  ],
  constraints: {
    maxDuration: 3600000,  // 1 hour max
    failFast: true         // Stop on first failure
  }
});

4. Performance Validation

Validation framework with multi-criteria assessment:

Validation Type	Criteria
SLA Validation	Availability, response time, throughput, error rate
Regression Validation	Comparison with historical data
Scalability Validation	Linear scaling, efficiency retention
Reliability Validation	Error handling, recovery, consistency

MCP Integration

// Comprehensive benchmark integration
const benchmarkIntegration = {
  // Execute performance benchmarks
  async runBenchmarks(config = {}) {
    const [benchmark, metrics, trends, cost] = await Promise.all([
      mcp.benchmark_run({ suite: config.suite || 'comprehensive' }),
      mcp.metrics_collect({ components: ['system', 'agents', 'coordination'] }),
      mcp.trend_analysis({ metric: 'performance', period: '24h' }),
      mcp.cost_analysis({ timeframe: '24h' })
    ]);

    return { benchmark, metrics, trends, cost, timestamp: Date.now() };
  },

  // Quality assessment
  async assessQuality(criteria) {
    return await mcp.quality_assess({
      target: 'swarm-performance',
      criteria: criteria || ['throughput', 'latency', 'reliability', 'scalability']
    });
  }
};

Key Metrics

Benchmark Targets

const benchmarkTargets = {
  throughput: {
    requests_per_second: { min: 1000, optimal: 5000 },
    tasks_per_second: { min: 100, optimal: 500 },
    messages_per_second: { min: 10000, optimal: 50000 }
  },
  latency: {
    p50: { max: 100 },   // 100ms
    p90: { max: 200 },   // 200ms
    p95: { max: 500 },   // 500ms
    p99: { max: 1000 },  // 1s
    max: { max: 5000 }   // 5s
  },
  scalability: {
    linear_coefficient: { min: 0.8 },
    efficiency_retention: { min: 0.7 }
  }
};

CI/CD Quality Gates

Gate	Criteria	Action on Failure
Performance	< 10% degradation	Block deployment
Latency	p99 < 1s	Warning
Error Rate	< 0.5%	Block deployment
Scalability	> 80% linear	Warning

Load Testing Example

// Load test with gradual ramp-up
const loadTest = {
  type: 'load',
  phases: [
    { phase: 'ramp-up', duration: 60000, startLoad: 10, endLoad: 100 },
    { phase: 'sustained', duration: 300000, load: 100 },
    { phase: 'ramp-down', duration: 30000, startLoad: 100, endLoad: 0 }
  ],
  successCriteria: {
    p99_latency: { max: 1000 },
    error_rate: { max: 0.01 },
    throughput: { min: 80 }  // % of expected
  }
};

Stress Testing Example

// Stress test to find breaking point
const stressTest = {
  type: 'stress',
  startLoad: 100,
  maxLoad: 10000,
  loadIncrement: 100,
  duration: 60000,  // Per load level
  breakingCriteria: {
    error_rate: { max: 0.05 },    // 5% errors
    latency_p99: { max: 5000 },   // 5s latency
    timeout_rate: { max: 0.10 }   // 10% timeouts
  }
};

Integration Points

Integration	Purpose
Performance Monitor	Continuous monitoring data for benchmarking
Load Balancer	Validates load balancing effectiveness
Topology Optimizer	Tests topology configurations
CI/CD Pipeline	Automated quality gates

Best Practices

1. Consistent Environment: Run benchmarks in consistent, isolated environments
2. Warmup Period: Always include warmup to eliminate cold-start effects
3. Multiple Iterations: Run multiple iterations for statistical significance
4. Baseline Maintenance: Keep baseline updated with expected performance
5. Historical Tracking: Store all benchmark results for trend analysis
6. Realistic Workloads: Use production-like workload patterns

Example: CI/CD Integration

#!/bin/bash
# ci-performance-gate.sh

# Run benchmark suite
RESULTS=$(npx claude-flow benchmark-run --suite quick --output json)

# Compare with baseline
COMPARISON=$(npx claude-flow benchmark-compare \
  --current "$RESULTS" \
  --baseline ./baseline.json)

# Check for regressions
if echo "$COMPARISON" | jq -e '.regression_detected == true' > /dev/null; then
  echo "Performance regression detected!"
  echo "$COMPARISON" | jq '.regressions'
  exit 1
fi

echo "Performance validation passed"
exit 0

Related Skills

• optimization-monitor - Real-time performance monitoring
• optimization-analyzer - Bottleneck analysis and reporting
• optimization-load-balancer - Load distribution optimization
• optimization-topology - Topology performance testing

---

Version History

• 1.0.0 (2026-01-02): Initial release - converted from benchmark-suite agent with comprehensive benchmarking, regression detection, automated testing, and performance validation