On the subject of debugging these so-called AI(s) / Black Boxes… the model is a black box sure, but that’s not really the problem. Everything around it — the inputs, the outputs, the decisions it makes — all of that can and should be fully logged, traced and replayed. The “program” isn’t the model, it’s the full context you feed it. That’s what you debug. It’s not so different from any other system really; if you’re running something in production with no logs, no structured outputs and no tests, you’d have the same problem. The model doesn’t change that discipline, it just makes it more important.