People ask me what it is like to build software with an AI helper, and whether it makes mistakes. It does. So do I. The useful thing turned out not to be a helper that never gets anything wrong. It is a workshop where a mistake gets caught, gets explained, and gets turned into a rule so it does not happen twice.
Here is what one day looked like.
The helper was asked to count something in our records and got zero. It could see nearby records that clearly existed, so it decided the count must be broken. The count was fine. It was looking in the wrong filing cabinet. The lesson we wrote down was simple: never report a number without saying which cabinet it came from.
The same day, one of our systems stopped being able to learn a new word. A change to how the records were stored had left one old piece of machinery still looking for a drawer that had been removed. Nothing rang a bell. Nobody was told. We found out by trying to add a word and watching it fail. What we learned was that our automatic checks had been testing a copy of the filing system that did not match the real one, because three changes had been made by hand and never written in the logbook. Now they are.
The helper also wrote up a problem report and gave it a number, and the number was already taken. The fix was not to tell it to be more careful. The fix was to make sure the only way to get a number is to ask the system that hands them out, so nobody can guess one.
My favorite moment was a small one. The helper reached for a quick shortcut, and a little guard we built stopped it. The guard did not just say no. It said which rule applied and what to do instead. The helper read that, changed its approach, and carried on, all in one step. A little later a different, older guard stopped the same thing and said nothing at all, and that was worse, because a silent no does not teach anything. It just sends you looking for another door.
That is the whole idea, really. Rules that live in someone's memory get forgotten at the busiest moment. Rules that are built into the workshop, like a latch on a cabinet, do not depend on anyone remembering. And the good ones, when they stop you, tell you why.
We are now working on a cleaner way to write those guards, borrowed from the way network firewalls have worked for twenty years. It is built and tested, and not yet switched on. I mention that because the other lesson from the day is not to claim more than is true.