I recently talked with a company building AI safety tools for senior living and schools.
I expected to spend most of the conversation thinking about cameras and model accuracy. Instead, I kept thinking: the camera is the easy part.
You can point at a camera. You can put a box around something on a screen. You can demo an alert arriving three seconds later.
Fine. Now what?
That is where the product starts getting interesting.
In October 2025, an AI security system at a Maryland high school flagged a bag of Doritos as a possible firearm.
The school’s security department reviewed the alert and canceled it. The principal did not immediately know that, reported the situation to the school resource officer, and police were called. The student was searched and handcuffed.
The AI made a mistake. A human caught it. The system kept moving anyway.
That is the part I cannot get past.
If a person can cancel an alert and the consequence still happens, then “human in the loop” is mostly decoration.
The useful version
Senior living is where I can see the good version of this idea.
People fall, wander, and get out of bed at night when no one is standing there. Staff already have too much to do. Families want reassurance. Residents still deserve privacy.
The CDC says more than 14 million older adults, about 1 in 4, report falling each year. There is a real reason to notice a change sooner. There is also a real reason not to turn someone’s bedroom into a live feed.
An Axios Phoenix story described a system called Paul at Fellowship Square Mesa. The facility had averaged about 20 falls per month among 125 residents before using it. Paul uses radar-based motion detection instead of a camera or audio recorder. It looks for changes in gait, posture, stability, and movement at night.
Helpany’s CEO made the most important point in the story: the system is not preventing falls. Staff members are, because they have better information.
That is how I want an AI product described.
Not “the AI prevents falls.” The staff does the work. The tool helps them notice something useful sooner.
My product brain goes straight to 2:17 in the morning.
What does the caregiver actually receive when three other things are already going wrong? What changed? Why does it matter now? What should they check? Can they disagree with the alert?
I have built enough dashboards to know a screen can make the buyer feel informed while making the operator feel doomed.
A video feed is not a workflow. A risk score is not a workflow. An alert is not a workflow if it just asks someone who is already busy to become more busy.
The useful version says: this changed, here is why we think it matters, and here is the next human step.
Then it lets the human say, “No, that is not what happened,” and makes that answer count.
The questions I care about
I do not need a grand framework for this. I want answers to a few boring questions.
Who gets the alert first?
What do they see besides a scary label?
If they cancel it, does that cancellation reach everyone?
What happens when the same false alert fires ten more times?
Can someone pause or narrow the system without opening a support ticket and waiting until Tuesday?
Those questions are not as exciting as a detection demo. They are also the difference between a useful tool and a machine that creates work, fear, or both.
In senior living, a bad alert can waste staff time and make people trust the next alert less.
In a school, it can put a student on his knees in front of police.
Same basic promise: notice something sooner. Very different cost of being wrong.
Show me the bad day
I do not think the answer is to never use AI in these environments.
I can see the useful version. In senior living, it means less staring and better reasons to check on someone. In schools, it probably means narrower use, fewer automatic consequences, and a lot more humility about what the system does not know.
I do not want more eyes on people. I want fewer, better interruptions.
The camera is easy to sell because everyone understands watching. The model is easy to demo because accuracy fits on a slide.
The workflow is the actual product.
Do not show me only what happens when the system is right.
Show me the bad day. Show me who can stop it. Show me whether “cancel” actually means cancel.
That is the demo I care about.