AI Observability

For AI-First Products (ChatGPT-like, AI Agents, Code Generators, AI Writers)

Traditional click-based analytics don't work for AI products with non-deterministic interfaces. Track task completion, response quality, and user satisfaction instead.

Why AI Products Need Different Analytics

Did the AI actually help?
Track task completion rate, not just "clicks"
Output quality:
Accept vs reject rates, how much users edit outputs
Where users struggle:
High regeneration = poor initial quality
Abandonment points:
Where users give up (prompt, response, editing)

Core AI Events

Prompt & Response Flow

// When user submits a prompt
WysLeap.trackEvent('ai_prompt_submitted', {
  promptLength: 120,
  taskType: 'code_generation'
});

// When AI generates response
WysLeap.trackEvent('ai_response_generated', {
  responseLength: 500,
  responseTime: 1200,  // ms
  modelUsed: 'gpt-4'
});

User Feedback on Output

// User accepts output (good quality)
WysLeap.trackEvent('ai_response_accepted', {
  taskType: 'code_generation'
});

// User rejects output
WysLeap.trackEvent('ai_response_rejected', {
  taskType: 'writing'
});

// User edits output
WysLeap.trackEvent('ai_response_edited', {
  editPercentage: 30
});

// User requests regeneration
WysLeap.trackEvent('ai_regeneration_requested', {
  regenerationCount: 2
});

Task Success & Completion

// Task completed successfully
WysLeap.trackEvent('ai_task_completed', {
  taskType: 'writing',
  timeToCompletion: 180  // seconds
});

// Task abandoned
WysLeap.trackEvent('ai_task_abandoned', {
  taskType: 'code_generation',
  abandonedAt: 'response'
});

AI Metrics You'll See

Quality Metrics:

Accept rate (target: >70%)
Reject rate (target: <20%)
Edit rate and avg % edited
Avg regenerations (target: <2)

Task Performance:

Task completion rate (target: >80%)
Abandonment rate (target: <20%)
Time to completion
Performance by task type

Real-World AI Product Examples

AI Writing Assistant

Jasper, Copy.ai - Track content generation and acceptance

AI Code Generator

GitHub Copilot, Cursor - Track code generation quality

AI Chat Assistant

ChatGPT, Claude - Track conversation flow and satisfaction

AI Image Generator

Midjourney, DALL-E - Track generation and iteration patterns

Want to Record User Sessions?

Enable session replay to watch how users interact with your product.

Next: Session Replay

Onboarding Guide