Microsoft Xbox · Tech Mahindra
A safety check that runs right before a release, alongside the existing tests. It catches the kinds of failures normal tests miss, on a simple rule: strict checks can block a release, AI only gives advice, and people make the final call.
Implementation complete and ready for review. My own concept and architecture; full implementation details shared on request.
Problem
Between the tests passing and someone approving the release, nothing actually checked that the live app really worked. Blank pages, broken logins, 'access denied' errors on data, and dead services all slipped through. One real case: an admin page started returning an 'access denied' error after a release, went unnoticed for days, and needed a manual investigation, while all the tests stayed green the whole time.
Design Decisions
The AI never blocks a release on its own opinion. Only clear, provable checks can block; people make the final call.
It runs at the same time as the existing tests, so it adds almost no extra wait, around 30 seconds.
It works with two AI providers behind a single setting, so teams that need stricter data control can switch with no code changes.
Turning it off is a one-line change, so teams trust a tool they can remove instantly.
How It Works
It runs next to the existing tests: three strict checks that can block a release, and three AI checks that only give advice.
A real browser opens each app and confirms it actually displays, the menus work, and the code and styles load.
It checks the key services to catch ones that are dead on arrival.
It confirms required services are reachable and that security certificates have not expired.
An AI looks at screenshots for visual problems a simple status code cannot catch.
When a strict check fails, an AI explains the likely cause to speed up the fix.
An AI rates how risky the release is based on what changed, to inform the reviewer.
Engineering Challenges
Every objection a platform or security reviewer might raise has a built-in answer.
It runs at the same time as the existing tests, so it adds only about 30 seconds.
AI checks are advice only. Only the strict, provable checks can block a release.
The strict checks retry and time out sensibly and publish full reports, so a person can override in seconds.
By default the data stays inside Microsoft's systems; a stricter mode keeps it inside the team's own cloud, and secrets are removed before any AI call.
Built With
A simple release-pipeline step that drives a real browser and can swap its AI provider, publishing standard reports.
Impact
30 sec
Check runtime
None
Added release wait
Human
Final call
What This Shows
The key idea is the 'block, advise, decide' rule: provable checks hold the gate, AI adds judgment where it helps, and people stay in charge. That is what makes an AI safety tool something the people who own production can trust.