The outsized importance of using AI 'just right' in consumer health apps

AI promises to reshape everything from dinner reservations to medical research, but experts warn of persistent blind spots in the technology's ethical framework and real-world performance.

We spoke with Renée Deehan, PhD, SVP Science and Artificial Intelligence at InsideTracker, an AI-powered platform providing personal health analysis and data-driven wellness guides, to discuss the intersection of AI, ethics, and trust.

Core challenge: The fundamental issue with AI isn't just technical capability – it's about trust and reliability. "An ethical 'code' still has to be programmed in," says Deehan. "I may know not to perform obviously nefarious tasks … but not all scenarios can be accounted for."

Entrenched bias: The medical field presents particularly thorny challenges for ethical AI. Deehan points to deeper concerns: "We know marginalized groups are not as represented in clinical research – trans people, women of color. Outputs from agents may not be smart enough to account for that bias."

Such limitations mean expert oversight remains critical: "The more sophisticated tasks still need to be left to researchers trained in how to handle that bias, and know how to quality control the output of the agent. Obviously, we need to be very careful with sensitive financial, personal or health information," Deehan explains.

Real-world testing: Even in controlled scientific settings, AI falls short of being able to fully manage tasks. When Deehan tasked an AI system with analyzing scientific research papers and organizing findings into a spreadsheet, the results were mixed. "It actually did pretty well, but it missed a lot of material and I still had to QA the output," she says.

Deehan sees the human oversight necessary to get quality outputs a loss of important time and energy. Citing attempting to use an AI agent to make a simple dinner reservation she says, "It takes the agent longer to do the task than it does to instruct the agent to perform the task. I didn't find that it saved me any time."

Future potential: Yet Deehan sees promise in specific applications, particularly in streamlining customer service and personalizing recommendations. "The agents will increasingly 'learn' my own preferences — which restaurants and neighborhoods do I prefer? What are the shopping items that I frequently order and when? Hopefully it will be helpful for websites that are increasingly difficult to navigate, like some of the major e-commerce giants, where unless you are looking for something highly specific, you may have to wade through a lot of noise."