AI Observability
For AI-First Products (ChatGPT-like, AI Agents, Code Generators, AI Writers)
Traditional click-based analytics don't work for AI products with non-deterministic interfaces. Track task completion, response quality, and user satisfaction instead.
Why AI Products Need Different Analytics
- Did the AI actually help? Track task completion rate, not just "clicks"
- Output quality: Accept vs reject rates, how much users edit outputs
- Where users struggle: High regeneration = poor initial quality
- Abandonment points: Where users give up (prompt, response, editing)
Core AI Events
Prompt & Response Flow
// When user submits a prompt
WysLeap.trackEvent('ai_prompt_submitted', {
promptLength: 120,
taskType: 'code_generation'
});
// When AI generates response
WysLeap.trackEvent('ai_response_generated', {
responseLength: 500,
responseTime: 1200, // ms
modelUsed: 'gpt-4'
});User Feedback on Output
// User accepts output (good quality)
WysLeap.trackEvent('ai_response_accepted', {
taskType: 'code_generation'
});
// User rejects output
WysLeap.trackEvent('ai_response_rejected', {
taskType: 'writing'
});
// User edits output
WysLeap.trackEvent('ai_response_edited', {
editPercentage: 30
});
// User requests regeneration
WysLeap.trackEvent('ai_regeneration_requested', {
regenerationCount: 2
});Task Success & Completion
// Task completed successfully
WysLeap.trackEvent('ai_task_completed', {
taskType: 'writing',
timeToCompletion: 180 // seconds
});
// Task abandoned
WysLeap.trackEvent('ai_task_abandoned', {
taskType: 'code_generation',
abandonedAt: 'response'
});AI Metrics You'll See
Quality Metrics
- Accept rate (target: >70%)
- Reject rate (target: <20%)
- Edit rate and avg % edited
- Avg regenerations (target: <2)
Task Performance
- Task completion rate (target: >80%)
- Abandonment rate (target: <20%)
- Time to completion
- Performance by task type
Example AI Product Types
AI Writing Assistant
Track content generation and acceptance (e.g. Jasper, Copy.ai).
AI Code Generator
Track code generation quality (e.g. GitHub Copilot, Cursor).
AI Chat Assistant
Track conversation flow and satisfaction (e.g. ChatGPT, Claude).
AI Image Generator
Track generation and iteration patterns (e.g. Midjourney, DALL-E).
Next Steps
Want to record user sessions? Enable session replay to watch how users interact with your product.
SaaS / PLG Tracking
For SaaS companies: Track product-led growth metrics like Time to First Value (TTFV), activation rate, Product Qualified Leads (PQLs), trial conversion, and churn risk.
Session Replay
Record and replay user sessions to understand behavior, debug issues, and improve UX. Session replay is off by default. Turn it on with sessionReplay.enabled: true.