Documentation

Onboarding guide

Get from zero to live analytics in minutes: installation, custom events, SaaS metrics, AI observability, and session replay.

Step 06 of 9: AI Observability

Onboarding guide

AI Observability

For AI-First Products (ChatGPT-like, AI Agents, Code Generators, AI Writers)

Traditional click-based analytics don't work for AI products with non-deterministic interfaces. Track task completion, response quality, and user satisfaction instead.

Why AI Products Need Different Analytics

Did the AI actually help?
Track task completion rate, not just "clicks"
Output quality:
Accept vs reject rates, how much users edit outputs
Where users struggle:
High regeneration = poor initial quality
Abandonment points:
Where users give up (prompt, response, editing)

Core AI Events

Prompt & Response Flow

javascript

// When user submits a prompt
WysLeap.trackEvent('ai_prompt_submitted', {
  promptLength: 120,
  taskType: 'code_generation'
});

// When AI generates response
WysLeap.trackEvent('ai_response_generated', {
  responseLength: 500,
  responseTime: 1200,  // ms
  modelUsed: 'gpt-4'
});

User Feedback on Output

javascript

// User accepts output (good quality)
WysLeap.trackEvent('ai_response_accepted', {
  taskType: 'code_generation'
});

// User rejects output
WysLeap.trackEvent('ai_response_rejected', {
  taskType: 'writing'
});

// User edits output
WysLeap.trackEvent('ai_response_edited', {
  editPercentage: 30
});

// User requests regeneration
WysLeap.trackEvent('ai_regeneration_requested', {
  regenerationCount: 2
});

Task Success & Completion

javascript

// Task completed successfully
WysLeap.trackEvent('ai_task_completed', {
  taskType: 'writing',
  timeToCompletion: 180  // seconds
});

// Task abandoned
WysLeap.trackEvent('ai_task_abandoned', {
  taskType: 'code_generation',
  abandonedAt: 'response'
});

AI Metrics You'll See

Quality Metrics:

Accept rate (target: >70%)
Reject rate (target: <20%)
Edit rate and avg % edited
Avg regenerations (target: <2)

Task Performance:

Task completion rate (target: >80%)
Abandonment rate (target: <20%)
Time to completion
Performance by task type

Example AI Product Types

AI Writing Assistant

Track content generation and acceptance (e.g. Jasper, Copy.ai).

AI Code Generator

Track code generation quality (e.g. GitHub Copilot, Cursor).

AI Chat Assistant

Track conversation flow and satisfaction (e.g. ChatGPT, Claude).

AI Image Generator

Track generation and iteration patterns (e.g. Midjourney, DALL-E).

Want to Record User Sessions?

Enable session replay to watch how users interact with your product.

Next: Session Replay →