Support tickets dropped 73% when we stopped letting LLMs write error messages. Here's the hilarious journey from 3-paragraph error essays to actually helpful 12-word messages.
"Let AI handle the error messages," they said. "It'll be more user-friendly," they said. Fast forward to production: User uploads a file that's too large. The AI-generated error message? A philosophical meditation on the nature of file size limitations in modern computing.
I'm not making this up.
The Spectrum of AI Error Message Failures
// What we asked AI to do
const errorMessage = await generateErrorMessage({
error: 'FILE_TOO_LARGE',
context: 'user_upload',
tone: 'helpful'
});
// What AI delivered
"I understand you're trying to upload a file. File uploading is a complex process that involves transferring data from your local machine to our servers. In this particular instance, it appears that the file you've selected exceeds our maximum file size limit of 10MB. This limit exists to ensure optimal performance for all users. You might consider compressing your file using various compression algorithms, or perhaps splitting it into smaller chunks. Alternatively, you could review the content to see if any portions are unnecessary..."
User has already left.
// The "helpful" error generator
function generateHelpfulError(error) {
return ai.complete(`
Generate a helpful, user-friendly error message for: ${error}
Be thorough and educational.
`);
}
// Result: 3-paragraph essays on every error
These are actual error messages our AI generated:
"Your password doesn't meet our requirements. But don't worry! Creating a strong password is like building a fortress for your digital identity. Consider using a mix of uppercase and lowercase letters, like the peaks and valleys of a mountain range..."
"We're having trouble connecting to our database. This could be due to various factors including network latency, server load, or cosmic rays affecting our data center. Please try again in a few moments."
Cosmic rays. Really.
"The email address you entered doesn't appear to follow the standard format defined in RFC 5322. An email typically consists of a local part, followed by the @ symbol, followed by a domain..."
Nobody needs an RFC citation in their error message.
AI doesn't understand user frustration. When someone sees an error, they're already annoyed. They don't want a lecture.
AI learned from documentation, tutorials, and Stack Overflow. Those are teaching contexts, not error contexts.
Tell AI to be "helpful" and it becomes a overeager teaching assistant. Tell it to be "concise" and you get "Error occurred."
After months of iteration, here's the prompt that generates good error messages:
const BETTER_ERROR_PROMPT = `
Generate an error message with EXACTLY this format:
1. Problem (5-10 words max)
2. Solution (1 specific action)
3. Technical detail in parentheses
Example:
"File too large. Choose a file under 10MB (current: 25MB)"
DO NOT:
- Apologize
- Explain why the limit exists
- Suggest multiple solutions
- Use more than 20 words total
`;
// The error message validator
function validateErrorMessage(message) {
const wordCount = message.split(' ').length;
const hasApology = /sorry|apologize|regret/i.test(message);
const hasExplanation = /because|since|as a result|in order to/i.test(message);
return {
valid: wordCount <= 20 && !hasApology && !hasExplanation,
issues: {
tooLong: wordCount > 20,
hasApology,
hasExplanation
}
};
}
// Before and after examples
const errorExamples = {
before: {
FILE_TOO_LARGE: "I understand you're trying to upload a file. File uploading is a complex process...",
INVALID_EMAIL: "The email address you entered doesn't appear to follow the standard format defined in RFC 5322...",
NETWORK_ERROR: "We're having trouble connecting to our database. This could be due to various factors including..."
},
after: {
FILE_TOO_LARGE: "File too large. Choose a file under 10MB (current: 25MB)",
INVALID_EMAIL: "Invalid email format. Check for typos (example: [email protected])",
NETWORK_ERROR: "Connection failed. Try again or check your internet"
}
};
// The final error message generator
class ErrorMessageGenerator {
constructor() {
this.cache = new Map();
}
generate(errorCode, context) {
// Check cache first
const cacheKey = `${errorCode}:${JSON.stringify(context)}`;
if (this.cache.has(cacheKey)) {
return this.cache.get(cacheKey);
}
// Generate with strict constraints
const message = this.generateWithConstraints(errorCode, context);
// Validate before caching
const validation = validateErrorMessage(message);
if (!validation.valid) {
return this.fallbackMessage(errorCode, context);
}
this.cache.set(cacheKey, message);
return message;
}
fallbackMessage(errorCode, context) {
// Human-written fallbacks for common errors
const fallbacks = {
FILE_TOO_LARGE: `File too large. Max: ${context.maxSize}MB`,
INVALID_EMAIL: "Invalid email format",
NETWORK_ERROR: "Connection failed. Try again"
};
return fallbacks[errorCode] || "An error occurred. Try again";
}
}
For complex errors, we built a context-aware system:
// Context-aware error system
const contextualErrors = {
FILE_UPLOAD: {
FILE_TOO_LARGE: (context) => {
if (context.fileType === 'image') {
return `Image too large. Resize to under ${context.maxSize}MB (current: ${context.currentSize}MB)`;
}
if (context.fileType === 'video') {
return `Video exceeds ${context.maxSize}MB limit. Compress or trim it`;
}
return `File too large. Max: ${context.maxSize}MB`;
},
UNSUPPORTED_TYPE: (context) => {
const allowed = context.allowedTypes.join(', ');
return `File type not supported. Use: ${allowed}`;
}
},
FORM_VALIDATION: {
REQUIRED_FIELD: (context) => `${context.fieldName} is required`,
INVALID_FORMAT: (context) => {
const examples = {
phone: '(555) 123-4567',
date: 'MM/DD/YYYY',
zipcode: '12345'
};
return `Invalid ${context.fieldName}${examples[context.fieldType] ? ` (example: ${examples[context.fieldType]})` : ''}`;
}
}
};
We tried giving our errors "personality":
- Friendly: "Oops! Looks like that file is a bit too chunky!"
- Professional: "File size exceeds maximum permitted threshold."
- Casual: "Whoa, that file's huge! Try something smaller?'
Users hated all of them. Turns out nobody wants personality from error messages. They want clarity.
Then we tried to localize AI errors:
// AI localization attempt
const localizedErrors = {
en: "File too large. Max: 10MB",
es: "El archivo es demasiado grande. El tamaño máximo permitido es de 10MB para garantizar un rendimiento óptimo",
de: "Die Datei überschreitet die maximal zulässige Größe von 10MB. Bitte wählen Sie eine kleinere Datei aus",
fr: "Le fichier que vous avez sélectionné dépasse la taille maximale autorisée de 10MB",
ja: "ファイルサイズが大きすぎます。最大許容サイズは10MBです。",
zh: "文件太大。请选择小于10MB的文件"
};
// AI made some languages way too verbose
Same meaning, wildly different lengths. Our UI broke in 6 languages.
- State the problem clearly - What exactly went wrong?
- Provide the fix - One specific action
- Include the actual value - "File too large (25MB)"
- Skip the why - Users don't care about your file size philosophy
- No apologies - "Sorry" doesn't fix the problem
- Test with angry users - They'll tell you if it's clear
- Support tickets about "confusing errors": -73%
- User retry success rate: +41%
- Average error message length: 12 words (down from 67)
- Developer sanity: Restored
// The final implementation
const errorConfig = {
// Strict constraints for AI
maxWords: 20,
format: 'problem_solution_detail',
forbiddenPhrases: ['sorry', 'apologize', 'because', 'in order to', 'RFC'],
// Human fallbacks for critical errors
fallbacks: {
PAYMENT_FAILED: "Payment declined. Try another card",
AUTH_EXPIRED: "Session expired. Sign in again",
SERVER_ERROR: "Something went wrong. Try again"
},
// A/B testing showed these perform best
templates: {
validation: "{field} {issue}. {example}",
limit: "{resource} exceeds {limit}. {action}",
connection: "{service} unavailable. {retry}"
}
};
// Usage
const error = generateError('FILE_TOO_LARGE', {
currentSize: '25MB',
maxSize: '10MB'
});
// Returns: "File too large. Choose a file under 10MB (current: 25MB)"