I build my own automations, so I own every failure
I don't hire a team to build the automations that run parts of my business. I build them myself, in the same tools I'd expect a small business client to use, and that means when something breaks I'm the one who finds it, not a support ticket sitting in someone else's queue. That changes how I test. I'm not testing to prove the thing works once. I'm testing to find out how it fails, because it will.
Six runs, six failures, before anything went live
I recently built a cloud-based agent to pull stories for a newsletter and draft the copy. I ran it six times while building it and it failed six different ways, and each failure forced a specific fix before I moved on. Run one broke before it even reached the stories themselves; the feeds coming in were inconsistent enough that the parser choked, so I rebuilt that first. Run two found 21 valid stories then hit a 410 error from a retired GitHub Models API route, so I swapped it for a live one. Run three produced drafts in five minutes flat but the copy used phrases I'd banned, so I added a cleanup pass and a repair round to catch that automatically. Run four never got a verdict back from the editor step at all, so I increased how much review time the step was allowed before it timed out. Run five finished in eight minutes but the FAQ section was truncated and quoted a number I couldn't trace to a source, so I added a rule that rejects output past a certain token ceiling rather than letting it trail off mid-sentence. Run six finished clean on the surface, until I noticed a repair pass had cut the internal links in the digest from 12 down to 8, so I added a second repair round specifically for links. Not one of those six went anywhere near a live system.
The mistake that slips past a working test, a mailbox that reverted to a fake sender name and reached a real inbox
The failure that worries me most isn't the obvious one. On a cold email automation, one mailbox reverted its sending identity to a placeholder name, Tom Parker, while the actual email copy underneath was fine. I'd checked the message. I hadn't checked the identity fields wrapped around it, because a working test run doesn't flag a field you didn't think to look at. A real person replied and addressed him as Tom. The email content passed every test I ran. The sender didn't, and that's the part that stayed with me, that a test can pass completely and still let something false out the door.
The failure that only shows up after launch, a fixed-fee project that worked until messy client data made it misfire on live records
Some failures don't show up in testing at all, because they need real, messy conditions to trigger them. I built a hiring chatbot and workflow automation for a recruitment client on a fixed fee of 8,500 pounds. The build itself came in on budget and worked in every test I ran against it, using clean sample data I'd put together myself. Once it was live against the client's actual CRM data, it misfired on roughly one in six candidate records, because their real data was messier in ways my test set never was: missing fields, inconsistent formats, duplicate entries the build hadn't been asked to handle.
What changed in my process after the mailbox sent as someone else
Now, before any mailbox in one of my automations sends anything live, I open the sending profile on its own and check the display name against my own name, separate from checking the message copy underneath it. I treat the identity fields as their own test item, not something a working draft happens to prove by default. The mailbox that came back as Tom Parker is still the reason I do that check on every campaign now, not just the one where it went wrong.
What changed in my process after the second invoice
The recruitment client invoice changed what I ask for before I agree a fixed fee on a build. I now ask for a sample of the client's actual CRM data before I call the build finished, not just the clean sample I put together myself to test against. I still remember totting up the extra 22 hours and writing a second invoice for 2,750 pounds, about a third of the original fixed fee, and deciding that the conversation about messy data needs to happen before the contract is signed, not after the build is already live.