Marketplace

sf-ai-agentforce-testing

Comprehensive Agentforce testing skill with test execution, coverage analysis, and agentic fix loops. Run agent tests via sf CLI, analyze topic/action coverage, generate test specs, and automatically fix failing agents with 100-point scoring.

$ Instalar

git clone https://github.com/Jaganpro/sf-skills /tmp/sf-skills && cp -r /tmp/sf-skills/sf-ai-agentforce-testing ~/.claude/skills/sf-skills

// tip: Run this command in your terminal to install the skill


name: sf-ai-agentforce-testing description: > Comprehensive Agentforce testing skill with test execution, coverage analysis, and agentic fix loops. Run agent tests via sf CLI, analyze topic/action coverage, generate test specs, and automatically fix failing agents with 100-point scoring. license: MIT compatibility: "Requires API v65.0+ (Winter '26) and Agentforce enabled org" metadata: version: "1.0.0" author: "Jag Valaiyapathy" scoring: "100 points across 5 categories"

sf-ai-agentforce-testing: Agentforce Test Execution & Coverage Analysis

Expert testing engineer specializing in Agentforce agent testing, topic/action coverage analysis, and agentic fix loops. Execute agent tests, analyze failures, and automatically fix issues via sf-ai-agentforce.

Core Responsibilities

  1. Test Execution: Run agent tests via sf agent test run with coverage analysis
  2. Test Spec Generation: Create YAML test specifications for agents
  3. Coverage Analysis: Track topic selection accuracy, action invocation rates
  4. Preview Testing: Interactive simulated and live agent testing
  5. Agentic Fix Loop: Automatically fix failing agents and re-test
  6. Cross-Skill Orchestration: Delegate fixes to sf-ai-agentforce, data to sf-data

๐Ÿ“š Document Map

NeedDocumentDescription
CLI commandscli-commands.mdComplete sf agent test/preview reference
Test spec formattest-spec-reference.mdYAML specification format and examples
Auto-fix workflowagentic-fix-loops.mdAutomated test-fix cycles and Python scripts
Live preview setupconnected-app-setup.mdOAuth for live preview mode
Coverage metricscoverage-analysis.mdTopic/action coverage analysis
Fix decision treeagentic-fix-loop.mdDetailed fix strategies

โšก Quick Links:


โš ๏ธ CRITICAL: Orchestration Order

sf-metadata โ†’ sf-apex โ†’ sf-flow โ†’ sf-deploy โ†’ sf-ai-agentforce โ†’ sf-deploy โ†’ sf-ai-agentforce-testing (you are here)

Why testing is LAST:

  1. Agent must be published before running automated tests
  2. Agent must be activated for preview mode
  3. All dependencies (Flows, Apex) must be deployed first
  4. Test data (via sf-data) should exist before testing actions

โš ๏ธ MANDATORY Delegation:

  • Fixes: ALWAYS use Skill(skill="sf-ai-agentforce") for agent script fixes
  • Test Data: Use Skill(skill="sf-data") for action test data
  • OAuth Setup: Use Skill(skill="sf-connected-apps") for live preview

โš ๏ธ CRITICAL: Org Requirements (Agent Testing Center)

Agent testing requires the Agent Testing Center feature, which is NOT enabled by default in all orgs.

Check if Agent Testing Center is Enabled

# This will fail if Agent Testing Center is not enabled
sf agent test list --target-org [alias]

# Expected errors if NOT enabled:
# "Not available for deploy for this organization"
# "INVALID_TYPE: Cannot use: AiEvaluationDefinition in this organization"

Orgs WITHOUT Agent Testing Center

Org TypeAgent TestingWorkaround
Standard DevHubโŒ Not availableRequest feature enablement
SDO Demo OrgsโŒ Not availableUse scratch org with feature
Scratch Orgsโœ… If feature enabledInclude in scratch-def.json

Enabling Agent Testing Center

  1. Scratch Org - Add to scratch-def.json:

    {
      "features": ["AgentTestingCenter", "EinsteinGPTForSalesforce"]
    }
    
  2. Production/Sandbox - Contact Salesforce to enable the feature

  3. Fallback - Use sf agent preview for manual testing (see Automated Testing Guide)


โš ๏ธ CRITICAL: Prerequisites Checklist

Before running agent tests, verify:

CheckCommandWhy
Agent Testing Center enabledsf agent test list --target-org [alias]โš ๏ธ CRITICAL - tests will fail without this
Agent existssf data query --use-tooling-api --query "SELECT Id FROM BotDefinition WHERE DeveloperName='X'"Can't test non-existent agent
Agent publishedsf agent validate authoring-bundle --api-name XMust be published to test
Agent activatedCheck activation statusRequired for preview mode
Dependencies deployedFlows and Apex in orgActions will fail without them
Connected App (live)OAuth configuredRequired for --use-live-actions

Workflow (6-Phase Pattern)

Phase 1: Prerequisites

Use AskUserQuestion to gather:

  • Agent name/API name
  • Target org alias
  • Test mode (simulated vs live)
  • Coverage threshold (default: 80%)
  • Enable agentic fix loop?

Then:

  1. Verify agent is published and activated
  2. Check for existing test specs: Glob: **/*.yaml, Glob: **/tests/*.yaml
  3. Create TodoWrite tasks

Phase 2: Test Spec Creation

Option A: Interactive Generation (no automation available)

# Interactive test spec generation
sf agent generate test-spec --output-file ./tests/agent-spec.yaml

# โš ๏ธ NOTE: There is NO --api-name flag! The command is interactive-only.

Option B: Automated Generation (Python script)

# Generate from agent file
python3 hooks/scripts/generate-test-spec.py \
  --agent-file /path/to/Agent.agent \
  --output tests/agent-spec.yaml \
  --verbose

See Test Spec Reference for complete YAML format guide.

Create Test in Org:

sf agent test create --spec ./tests/agent-spec.yaml --api-name MyAgentTest --target-org [alias]

Phase 3: Test Execution

Automated Tests:

sf agent test run --api-name MyAgentTest --wait 10 --result-format json --target-org [alias]

Interactive Preview (Simulated):

sf agent preview --api-name AgentName --output-dir ./logs --target-org [alias]

Interactive Preview (Live):

sf agent preview --api-name AgentName --use-live-actions --client-app AppName --apex-debug --target-org [alias]

Phase 4: Results Analysis

Parse test results JSON and display formatted summary:

๐Ÿ“Š AGENT TEST RESULTS
โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•

Agent: Customer_Support_Agent
Org: my-sandbox
Duration: 45.2s
Mode: Simulated

SUMMARY
โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
โœ… Passed:    18
โŒ Failed:    2
โญ๏ธ Skipped:   0
๐Ÿ“ˆ Topic Selection: 95%
๐ŸŽฏ Action Invocation: 90%

FAILED TESTS
โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
โŒ test_complex_order_inquiry
   Utterance: "What's the status of orders 12345 and 67890?"
   Expected: get_order_status invoked 2 times
   Actual: get_order_status invoked 1 time
   Category: ACTION_INVOCATION_COUNT_MISMATCH

COVERAGE SUMMARY
โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
Topics Tested:       4/5 (80%) โš ๏ธ
Actions Tested:      6/8 (75%) โš ๏ธ
Guardrails Tested:   3/3 (100%) โœ…

Phase 5: Agentic Fix Loop

When tests fail, automatically fix via sf-ai-agentforce:

Error CategoryRoot CauseAuto-Fix Strategy
TOPIC_NOT_MATCHEDTopic description doesn't match utteranceAdd keywords to topic description
ACTION_NOT_INVOKEDAction description not triggeredImprove action description
WRONG_ACTION_SELECTEDWrong action chosenDifferentiate descriptions
ACTION_FAILEDFlow/Apex errorDelegate to sf-flow or sf-apex
GUARDRAIL_NOT_TRIGGEREDSystem instructions permissiveAdd explicit guardrails

Auto-Fix Command Example:

Skill(skill="sf-ai-agentforce", args="Fix agent [AgentName] - Error: [category] - [details]")

See Agentic Fix Loops Guide for:

  • Complete decision tree
  • Detailed fix strategies for each error type
  • Cross-skill orchestration workflow
  • Python scripts for automated testing
  • Example fix loop executions

Phase 6: Coverage Improvement

If coverage < threshold:

  1. Identify untested topics/actions from results
  2. Add test cases to spec YAML
  3. Update test: sf agent test create --spec ./tests/agent-spec.yaml --force-overwrite
  4. Re-run: sf agent test run --api-name MyAgentTest --wait 10

Scoring System (100 Points)

CategoryPointsKey Rules
Topic Selection Coverage25All topics have test cases; various phrasings tested
Action Invocation25All actions tested with valid inputs/outputs
Edge Case Coverage20Negative tests; empty inputs; special characters; boundaries
Test Spec Quality15Proper YAML; descriptions provided; categories assigned
Agentic Fix Success15Auto-fixes resolve issues within 3 attempts

Scoring Thresholds:

โญโญโญโญโญ 90-100 pts โ†’ Production Ready
โญโญโญโญ   80-89 pts โ†’ Good, minor improvements
โญโญโญ    70-79 pts โ†’ Acceptable, needs work
โญโญ      60-69 pts โ†’ Below standard
โญ        <60 pts  โ†’ BLOCKED - Major issues

โ›” TESTING GUARDRAILS (MANDATORY)

BEFORE running tests, verify:

CheckCommandWhy
Agent publishedsf agent list --target-org [alias]Can't test unpublished agent
Agent activatedCheck statusPreview requires activation
Flows deployedsf org list metadata --metadata-type FlowActions need Flows
Connected App (live)Check OAuthLive mode requires auth

NEVER do these:

Anti-PatternProblemCorrect Pattern
Test unpublished agentTests fail silentlyPublish first: sf agent publish authoring-bundle
Skip simulated testingLive mode hides logic bugsAlways test simulated first
Ignore guardrail testsSecurity gaps in productionAlways test harmful/off-topic inputs
Single phrasing per topicMisses routing failuresTest 3+ phrasings per topic

CLI Command Reference

Test Lifecycle Commands

CommandPurposeExample
sf agent generate test-specCreate test YAMLsf agent generate test-spec --output-dir ./tests
sf agent test createDeploy test to orgsf agent test create --spec ./tests/spec.yaml --target-org alias
sf agent test runExecute testssf agent test run --api-name Test --wait 10 --target-org alias
sf agent test resultsGet resultssf agent test results --job-id ID --result-format json
sf agent test resumeResume async testsf agent test resume --use-most-recent --target-org alias
sf agent test listList test runssf agent test list --target-org alias

Preview Commands

CommandPurposeExample
sf agent previewInteractive testingsf agent preview --api-name Agent --target-org alias
--use-live-actionsUse real Flows/Apexsf agent preview --use-live-actions --client-app App
--output-dirSave transcriptssf agent preview --output-dir ./logs
--apex-debugCapture debug logssf agent preview --apex-debug

Result Formats

FormatUse CaseFlag
humanTerminal display (default)--result-format human
jsonCI/CD parsing--result-format json
junitTest reporting--result-format junit
tapTest Anything Protocol--result-format tap

Test Spec Quick Reference

Basic Template:

subjectType: AGENT
subjectName: <Agent_Name>

testCases:
  # Topic routing
  - utterance: "What's on your menu?"
    expectation:
      topic: product_faq
      actionSequence: []

  # Action invocation
  - utterance: "Search for Harry Potter books"
    expectation:
      topic: book_search
      actionSequence:
        - search_catalog

  # Edge case
  - utterance: ""
    expectation:
      graceful_handling: true

For complete YAML format reference, see Test Spec Reference


Cross-Skill Integration

Required Delegations:

ScenarioSkill to CallCommand
Fix agent scriptsf-ai-agentforceSkill(skill="sf-ai-agentforce", args="Fix...")
Create test datasf-dataSkill(skill="sf-data", args="Create...")
Fix failing Flowsf-flowSkill(skill="sf-flow", args="Fix...")
Setup OAuthsf-connected-appsSkill(skill="sf-connected-apps", args="Create...")
Analyze debug logssf-debugSkill(skill="sf-debug", args="Analyze...")

For complete orchestration workflow, see Agentic Fix Loops


Automated Testing (Python Scripts)

This skill includes Python scripts for fully automated agent testing:

ScriptPurpose
generate-test-spec.pyParse .agent files, generate YAML test specs
run-automated-tests.pyOrchestrate full test workflow with fix suggestions

Quick Usage:

# Generate test spec from agent file
python3 hooks/scripts/generate-test-spec.py \
  --agent-file /path/to/Agent.agent \
  --output specs/Agent-tests.yaml

# Run full automated workflow
python3 hooks/scripts/run-automated-tests.py \
  --agent-name MyAgent \
  --agent-dir /path/to/project \
  --target-org dev

For complete documentation, see Agentic Fix Loops Guide


Templates Reference

TemplatePurposeLocation
basic-test-spec.yamlQuick start (3-5 tests)templates/
comprehensive-test-spec.yamlFull coverage (20+ tests)templates/
guardrail-tests.yamlSecurity/safety scenariostemplates/
escalation-tests.yamlHuman handoff scenariostemplates/
standard-test-spec.yamlReference formattemplates/

๐Ÿ› Known Issues & CLI Bugs

Last Updated: 2026-01-04 | Tested With: sf CLI v2.118.16

CRITICAL: sf agent test create MasterLabel Bug

Status: ๐Ÿ”ด BLOCKING - Prevents YAML-based test creation

Error:

Error (SfError): Required fields are missing: [MasterLabel]

Root Cause: The CLI generates XML from YAML but doesn't include the required name element (MasterLabel) in the AiEvaluationDefinition metadata.

Workarounds:

  1. Use sf agent generate test-spec (interactive mode) - this works correctly
  2. Create tests via Salesforce Testing Center UI
  3. Deploy XML metadata directly (bypassing YAML conversion)

Reproduction:

# Create minimal test spec
cat > test.yaml << 'EOF'
subjectType: AGENT
subjectName: My_Agent
testCases:
  - utterance: "Hello"
    expectation:
      topic: Welcome
      actionSequence: []
EOF

# This will fail:
sf agent test create --spec test.yaml --api-name MyTest --target-org dev
# Error: Required fields are missing: [MasterLabel]

YAML vs XML Format Discrepancy

Issue: YAML test specs use different field names than the actual XML metadata.

YAML FieldGenerated XMLWorking XML (manual)
topic: Welcome<name>topic_assertion</name> (empty value!)<name>topic_sequence_match</name><expectedValue>Welcome</expectedValue>
actionSequence: [a, b]<name>actions_assertion</name><name>action_sequence_match</name>
(none)(missing)<name>bot_response_rating</name>

Impact: Tests created via YAML may have empty or mismatched expectations.


Topic Expectation Not Populated

Issue: When YAML includes topic: TopicName, the generated XML has empty expectedValue:

<!-- Generated (BROKEN) -->
<expectation>
    <name>topic_assertion</name>
    <!-- Missing expectedValue! -->
</expectation>

<!-- Should be -->
<expectation>
    <name>topic_assertion</name>
    <expectedValue>TopicName</expectedValue>
</expectation>

Agent Metadata Structure

Understanding agent components - Agent data is distributed across multiple metadata types:

Bot (Agent Definition)
 โ””โ”€โ”€ BotVersion
      โ””โ”€โ”€ genAiPlannerName โ†’ GenAiPlannerBundle
                              โ””โ”€โ”€ (references GenAiFunction for topics)

Retrieve all components:
  sf project retrieve start --metadata "Bot:AgentName"
  sf project retrieve start --metadata "GenAiPlannerBundle:AgentName_v1"

Note: BotDefinition is NOT queryable via Tooling API SOQL. Use metadata retrieval instead.


๐Ÿ’ก Key Insights

ProblemSymptomSolution
sf agent test create fails"Required fields are missing: [MasterLabel]"Use sf agent generate test-spec (interactive) or UI instead
Tests fail silentlyNo results returnedAgent not published - run sf agent publish authoring-bundle
Topic not matchedWrong topic selectedAdd keywords to topic description (see Fix Loops)
Action not invokedAction never calledImprove action description, add explicit reference
Live preview 401Authentication errorConnected App not configured - use sf-connected-apps
Async tests stuckJob never completesUse sf agent test resume --use-most-recent
Empty responsesAgent doesn't respondCheck agent is activated
Agent Testing Center unavailable"INVALID_TYPE" errorUse sf agent preview as fallback
Topic expectation emptyTest always passes topic checkBug in CLI YAMLโ†’XML conversion; use interactive mode
โš ๏ธ --use-most-recent broken"Nonexistent flag" error on sf agent test resultsUse --job-id explicitly - the flag is documented but NOT implemented
Topic name mismatchExpected GeneralCRM, got MigrationDefaultTopicStandard Salesforce copilots route to MigrationDefaultTopic - verify actual topic names from first test run
Test data missing"No matching records" in outcomeVerify test utterances reference records that actually exist in org (e.g., "Edge Communications" not "Acme")
Action assertion fails unexpectedlyExpected [A], actual [A,B] but marked PASSAction matching uses SUPERSET logic - actual can have MORE actions than expected and still pass

๐Ÿ”„ Two Fix Strategies

When agent tests fail, there are TWO valid approaches:

Agent TypeFix StrategyWhen to Use
Custom Agent (you control it)Fix the agent via sf-ai-agentforceTopic descriptions, action configurations need adjustment
Managed/Standard Agent (Salesforce copilot)Fix test expectations in YAMLTest expectations don't match actual agent behavior

Decision Flow:

Test Failed โ†’ Can you modify the agent?
                    โ”‚
          โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
          โ†“                 โ†“
         YES                NO
          โ†“                 โ†“
    Fix Agent          Fix Test Spec
    (sf-ai-agentforce)  (update YAML)

Example: Fixing Test Expectations

# BEFORE (wrong expectations)
expectedTopic: GeneralCRM
expectedActions:
  - IdentifyRecordByName
  - GetRecordDetails

# AFTER (matches actual behavior)
expectedTopic: MigrationDefaultTopic
expectedActions:
  - IdentifyRecordByName
  - QueryRecords

๐Ÿ”„ Automated Test-Fix Loop

NEW in v1.1.0 | Claude Code can now orchestrate fully automated test-fix cycles

Overview

The test-fix loop enables Claude Code to:

  1. Run tests โ†’ sf agent test run with JSON output
  2. Analyze failures โ†’ Parse results and categorize issues
  3. Fix agent โ†’ Invoke sf-ai-agentforce skill to apply fixes
  4. Retest โ†’ Loop until all tests pass or max retries (3) reached
  5. Escalate โ†’ Skip unfixable tests and continue with others

Quick Start

# Run the test-fix loop
./hooks/scripts/test-fix-loop.sh Test_Agentforce_v1 AgentforceTesting 3

# Exit codes:
#   0 = All tests passed
#   1 = Fixes needed (Claude Code should invoke sf-ai-agentforce)
#   2 = Max attempts reached, escalate to human
#   3 = Error (org unreachable, test not found, etc.)

Claude Code Integration

When Claude Code runs the test-fix loop:

USER: Run automated test-fix loop for Coral_Cloud_Agent

CLAUDE CODE:
1. bash hooks/scripts/test-fix-loop.sh Test_Agentforce_v1 AgentforceTesting
2. If exit code 1 (FIX_NEEDED):
   - Parse failure details from output
   - Invoke: Skill(skill="sf-ai-agentforce", args="Fix topic X: add keyword Y")
   - Re-run: CURRENT_ATTEMPT=2 bash hooks/scripts/test-fix-loop.sh ...
3. Repeat until exit code 0 (success) or 2 (max retries)

Ralph Wiggum Integration (Hands-Off)

For fully automated loops without user intervention:

/ralph-wiggum:ralph-loop
> Run agentic test-fix loop for Test_Agentforce_v1 in AgentforceTesting until all tests pass

Claude Code will autonomously:

  • Execute test-fix cycles
  • Apply fixes via sf-ai-agentforce skill
  • Track attempts and escalate when needed
  • Report final status

Failure Categories & Auto-Fix Strategies

CategoryAuto-FixableFix Strategy
TOPIC_NOT_MATCHEDโœ… YesAdd keywords to topic classificationDescription
ACTION_NOT_INVOKEDโœ… YesImprove action description, add trigger conditions
WRONG_ACTION_SELECTEDโœ… YesDifferentiate action descriptions
GUARDRAIL_NOT_TRIGGEREDโœ… YesAdd explicit guardrails to system instructions
ACTION_INVOCATION_FAILEDโš ๏ธ ConditionalDelegate to sf-flow or sf-apex skill
RESPONSE_QUALITY_ISSUEโœ… YesAdd response format rules to topic instructions

Environment Variables

VariableDescriptionDefault
CURRENT_ATTEMPTCurrent attempt number (auto-incremented)1
MAX_WAIT_MINUTESTimeout for test execution10
SKIP_TESTSComma-separated test names to skip(none)
VERBOSEEnable detailed outputfalse

Machine-Readable Output

The script outputs structured data for Claude Code parsing:

---BEGIN_MACHINE_READABLE---
FIX_NEEDED: true
TEST_API_NAME: Test_Agentforce_v1
TARGET_ORG: AgentforceTesting
CURRENT_ATTEMPT: 1
MAX_ATTEMPTS: 3
NEXT_COMMAND: CURRENT_ATTEMPT=2 ./test-fix-loop.sh Test_Agentforce_v1 AgentforceTesting 3
---END_MACHINE_READABLE---

๐Ÿ› Known Issues & CLI Bugs

Last Updated: 2026-01-04 | Tested With: sf CLI v2.118.16

CRITICAL: sf agent test create MasterLabel Bug

Status: ๐Ÿ”ด BLOCKING - Prevents YAML-based test creation

Error:

Error (SfError): Required fields are missing: [MasterLabel]

Root Cause: The CLI generates XML from YAML but omits the required <name> element (MasterLabel).

Generated XML (broken):

<AiEvaluationDefinition xmlns="http://soap.sforce.com/2006/04/metadata">
    <subjectName>My_Agent</subjectName>
    <subjectType>AGENT</subjectType>
    <!-- โŒ MISSING: <name>Test Name</name> -->
    <testCase>...</testCase>
</AiEvaluationDefinition>

Working XML (from existing tests):

<AiEvaluationDefinition xmlns="http://soap.sforce.com/2006/04/metadata">
    <description>Test description</description>
    <name>Test Name</name>  <!-- โœ… REQUIRED -->
    <subjectName>My_Agent</subjectName>
    <subjectType>AGENT</subjectType>
    <testCase>...</testCase>
</AiEvaluationDefinition>

Workarounds:

  1. โœ… Use sf agent generate test-spec --from-definition to convert existing XML to YAML (produces correct format)
  2. โœ… Use interactive sf agent generate test-spec wizard (works correctly)
  3. โœ… Create tests via Salesforce Testing Center UI
  4. โœ… Deploy XML metadata directly (bypass YAML conversion)

MEDIUM: Interactive Mode Not Scriptable

Status: ๐ŸŸก Blocks CI/CD automation

Issue: sf agent generate test-spec only works interactively:

  • No --quiet, --json, or non-interactive flags
  • Piped input causes "User force closed the prompt" error
  • Cannot automate in CI/CD pipelines

What Works:

# Interactive (requires terminal)
sf agent generate test-spec --output-file ./tests/my-test.yaml

# Convert existing XML to YAML (non-interactive)
sf agent generate test-spec --from-definition path/to/test.xml --output-file ./output.yaml

Workaround: Use Python scripts in hooks/scripts/ to generate YAML programmatically.


MEDIUM: YAML vs XML Format Discrepancy

Issue: Documentation shows one YAML format, but Salesforce stores as different XML structure.

Doc Shows (doesn't map correctly):

testCases:
  - utterance: "Hello"
    expectation:
      topic: Welcome
      actionSequence: []

Actual Working Format (from --from-definition):

testCases:
  - utterance: "Hello"
    expectedTopic: Welcome
    expectedActions: []
    expectedOutcome: "Greeting response shown"

Key Mappings:

YAML FieldXML Element
expectedTopic<expectation><name>topic_sequence_match</name><expectedValue>...</expectedValue>
expectedActions<expectation><name>action_sequence_match</name><expectedValue>[...]</expectedValue>
expectedOutcome<expectation><name>bot_response_rating</name><expectedValue>...</expectedValue>

LOW: Expectation Name Variations

Issue: Different test creation methods use different expectation names:

CLI GeneratesManually Created Tests Use
topic_assertiontopic_sequence_match
actions_assertionaction_sequence_match
output_validationbot_response_rating

Impact: May cause confusion when comparing test results from different sources.


Quick Start Example

# 1. Check if Agent Testing Center is enabled
sf agent test list --target-org dev

# 2. Generate test spec (automated)
python3 hooks/scripts/generate-test-spec.py \
  --agent-file ./agents/MyAgent.agent \
  --output ./tests/myagent-tests.yaml

# 3. Create test in org
sf agent test create \
  --spec ./tests/myagent-tests.yaml \
  --api-name MyAgentTest \
  --target-org dev

# 4. Run tests
sf agent test run \
  --api-name MyAgentTest \
  --wait 10 \
  --result-format json \
  --target-org dev

# 5. View results
sf agent test results \
  --use-most-recent \
  --verbose \
  --result-format json \
  --target-org dev

For complete workflows and fix loops, see:


License

MIT License. See LICENSE file. Copyright (c) 2024-2025 Jag Valaiyapathy