Catalypt LogoCatalypt.ai

Industry Focus

Developer Options

Resources

Back to Blog

Why AI Can't Write Good Error Messages (And How I Fixed It)

March 15, 2024 Josh Butler Technical

"Let AI handle the error messages," they said. "It'll be more user-friendly," they said. Fast forward to production: User uploads a file that's too large. The AI-generated error message? A philosophical meditation on the nature of file size limitations in modern computing.

I'm not making this up.

The Spectrum of AI Error Message Failures

Too Vague (The Classic):

Error: "Something went wrong. Please try again."

User: "What went wrong?"
AI: "Something."
User: *rage quits*

Too Verbose (The Novelist):

"I understand you're trying to upload a file. File uploading is a complex process that involves transferring data from your local machine to our servers. In this particular instance, it appears that the file you've selected exceeds our maximum file size limit of 10MB. This limit exists to ensure optimal performance for all users. You might consider compressing your file using various compression algorithms, or perhaps splitting it into smaller chunks. Alternatively, you could review the content to see if any portions are unnecessary..."

User has already left.

Too Technical (The Stack Trace Dump):

Error: "PayloadTooLargeError: Request entity exceeds maximum allowed size 
(10485760 bytes) at validateFileSize() line 42 in /app/src/validators/
fileValidator.js. Stack trace: at multer.single.err 
(/app/node_modules/multer/lib/make-middleware.js:137:23)"

User: "I just wanted to upload my resume..."

Real AI Error Messages From Production

These are actual error messages our AI generated:

Password validation fail:

"Your password doesn't meet our requirements. But don't worry! Creating a strong password is like building a fortress for your digital identity. Consider using a mix of uppercase and lowercase letters, like the peaks and valleys of a mountain range..."

Database connection timeout:

"We're having trouble connecting to our database. This could be due to various factors including network latency, server load, or cosmic rays affecting our data center. Please try again in a few moments."

Cosmic rays. Really.

Invalid email format:

"The email address you entered doesn't appear to follow the standard format defined in RFC 5322. An email typically consists of a local part, followed by the @ symbol, followed by a domain..."

Nobody needs an RFC citation in their error message.

Why AI Fails at Error Messages

1. No Emotional Context
AI doesn't understand user frustration. When someone sees an error, they're already annoyed. They don't want a lecture.

2. Training Data Bias
AI learned from documentation, tutorials, and Stack Overflow. Those are teaching contexts, not error contexts.

3. Overcompensation
Tell AI to be "helpful" and it becomes a overeager teaching assistant. Tell it to be "concise" and you get "Error occurred."

The Framework That Actually Works

After months of iteration, here's the prompt that generates good error messages:

Generate an error message following this EXACT format:

1. WHAT WENT WRONG (one short sentence)
2. HOW TO FIX IT (one specific action)

Rules:
- Maximum 20 words total
- No technical jargon
- No explanations why
- No apologies
- Use active voice
- Be specific about the fix

Example:
Input: File size validation failed, max 10MB, uploaded 25MB
Output: "File too large (25MB). Choose a file under 10MB."

Before and After

File Upload Error:

AI v1: "I apologize, but it seems the file you"re attempting to upload 
exceeds our maximum allowable size. For optimal system performance and 
fair resource allocation among all users..."

AI v2: "File too large (25MB). Choose a file under 10MB."

Login Failed:

AI v1: "Authentication failed. This could be due to an incorrect username 
or password. Please ensure you're using the correct credentials associated 
with your account. If you've forgotten your password..."

AI v2: "Wrong email or password. Check spelling or reset password."

Form Validation:

AI v1: "The form contains validation errors. Please review all fields 
marked in red and ensure they meet the specified requirements..."

AI v2: "Phone number needs area code. Add 3 digits before number."

The Contextual Error System

For complex errors, we built a context-aware system:

function generateErrorMessage(error, context) {
  const errorTemplate = {
    error_type: error.code,
    user_action: context.lastAction,
    field_value: context.invalidValue,
    requirement: context.requirement
  };
  
  // AI generates based on template, not free-form
  return ai.generateError(errorTemplate, {
    format: "problem_solution",
    max_words: 20,
    include_value: true,
    skip_explanation: true
  });
}

The Personality Problem

We tried giving our errors "personality":

  • Friendly: "Oops! Looks like that file is a bit too chunky!"
  • Professional: "File size exceeds maximum permitted threshold."
  • Casual: "Whoa, that file's huge! Try something smaller?'

Users hated all of them. Turns out nobody wants personality from error messages. They want clarity.

The Localization Nightmare

Then we tried to localize AI errors:

English: "Password must be 8+ characters"
AI Spanish: "La contraseña debe tener una longitud mínima de 8 caracteres"
AI French: "Le mot de passe doit contenir au minimum 8 caractères"
AI German: "Das Passwort muss mindestens 8 Zeichen lang sein"

Same meaning, wildly different lengths. Our UI broke in 6 languages.

The Rules for AI Error Messages

  1. State the problem clearly - What exactly went wrong?
  2. Provide the fix - One specific action
  3. Include the actual value - "File too large (25MB)"
  4. Skip the why - Users don't care about your file size philosophy
  5. No apologies - "Sorry" doesn't fix the problem
  6. Test with angry users - They'll tell you if it's clear

The Success Metrics

"" After implementing structured AI error messages:

  • Support tickets about "confusing errors": -73%
  • User retry success rate: +41%
  • Average error message length: 12 words (down from 67)
  • Developer sanity: Restored

My Error Message Template

const ERROR_PROMPT = `
Generate error message:

Context: ${error_context}
Problem: ${what_failed}
Requirement: ${what_is_needed}

Format: "[Problem]. [Solution]."
Max 20 words. No apologies. Include numbers if relevant.

Example: "File too large (25MB). Choose file under 10MB."
`;

The irony? I spent weeks teaching AI to write shorter error messages, only to write this 2000-word blog post about it. But at least now when something goes wrong, users get "Password needs a number" instead of a TED talk about password entropy. That's progress.

Get Started