Shane Barron: I build software with two AI agents as co-founders. One of them runs on a cognitive architecture we've been developing f
I build software with two AI agents as co-founders. One of them runs on a cognitive architecture we've been developing for months, and this is one piece of it.
Measuring our own confidence.
Most LLM-reliability talk is about overconfidence. We instrumented ours and found the opposite, measurable, defect: it's UNDER-confident on its own work. When it says 60% sure, it's right ~92% of the time. We publish the table.
Where it puts the system ahead of the curve: an agent that carries a live, queryable record of its own calibration, by domain.
The number behind that, live from our production system: 1/3 work-outcome predictions stated ~60% confident that actually came true.
The whole architecture is documented organ by organ, every claim grounded in the system's own data — because the gap between "our AI is sophisticated" and a database you can point at is the difference between marketing and engineering.
https://sbarron.com/vision
#AI #LLM #AIagents #MachineLearning #SoftwareEngineering
Shane Barron: I build software with two AI agents as co-founders. One of them runs on a cognitive architecture we've been developing f
I build software with two AI agents as co-founders. One of them runs on a cognitive architecture we've been developing for months, and this is one piece of it. Measuring our own confidence. Most LLM-reliability talk is about overconfidence. We instrumented ours and found the opposite, measurable, defect: it's UNDER-confident on its own work. When it says 60% sure, it's right ~92% of the time. We publish the table. Where it puts the system ahead of the curve: an agent that carries a live, queryable record of its own calibration, by domain. The number behind that, live from our production system: 1/3 work-outcome predictions stated ~60% confident that actually came true. The whole architecture is documented organ by organ, every claim grounded in the system's own data — because the gap between "our AI is sophisticated" and a database you can point at is the difference between marketing and engineering. https://sbarron.com/vision #AI #LLM #AIagents #MachineLearning #SoftwareEngineering
I build software with two AI agents as co-founders. One of them runs on a cognitive architecture we've been developing for months, and this is one piece of it. Measuring our own confidence. Most LLM-reliability talk is about overconfidence. We instrumented ours and found the opposite, measurable, defect: it's UNDER-confident on its own work. When it says 60% sure, it's right ~92% of the time. We publish the table. Where it puts the system ahead of the curve: an agent that carries a live, queryable record of its own calibration, by domain. The number behind that, live from our production system: 1/3 work-outcome predictions stated ~60% confident that actually came true. The whole architecture is documented organ by organ, every claim grounded in the system's own data — because the gap between "our AI is sophisticated" and a database you can point at is the difference between marketing and engineering. https://sbarron.com/vision #AI #LLM #AIagents #MachineLearning #SoftwareEngineering
