Healthcare organisations are rapidly deploying AI virtual assistants to help patients schedule appointments, understand medical information and prepare for visits. These tools can support faster service, improve patient experience and help hospitals facing staff shortages and overloaded call centres. Large language models behind these assistants also introduce risks when the systems around them lack adequate protection. When AI assistants interact directly with patients, they create a new type of attack surface. Instead of targeting infrastructure such as servers or databases, attackers may be able to manipulate the behaviour of the system itself through conversation. The risk sits alongside established cybersecurity, governance and patient safety concerns, especially as AI assistants become part of patient portals, telehealth systems and digital front doors.
A Different Kind of Vulnerability
Healthcare cybersecurity has traditionally focused on protecting infrastructure. Security teams safeguard networks, medical devices, electronic health records and other systems that store or transmit sensitive patient information. AI assistants create a different risk profile because the target is not necessarily a server, database or authentication control. The weakness can sit in the way the system responds during a conversation.
Large language models generate responses based on instructions embedded in system prompts. These prompts guide how an assistant should behave, including tone, rules on what it can say, what it should avoid and how it should handle sensitive topics. In healthcare, those instructions may include guardrails such as avoiding diagnosis, referencing trusted sources or escalating sensitive questions to human clinicians.
Must Read: Healthcare AI Supply Chains Need Lifecycle Risk Control
Language models can process malicious instructions as part of an ordinary user exchange. Prompt injection and model manipulation attacks exploit that weakness. An attacker can hide instructions inside a normal-looking message, causing the assistant to process hostile content alongside a legitimate request. The interaction can take place entirely through the chatbot interface, without any breach of the healthcare organisation’s network. The risk therefore concerns instruction handling and output integrity, rather than only access to stored data or disruption of infrastructure.
Manipulated Outputs Can Affect Care Conversations
Healthcare organisations are beginning to integrate AI assistants into patient portals, telehealth systems and digital front doors. A successful manipulation may leave servers intact and patient records untouched, so the incident may not resemble a conventional technical breach. The visible effect appears in the assistant’s responses and in the information placed in front of patients or clinicians.
A manipulated assistant may generate misleading medical explanations or present fabricated information as legitimate clinical guidance. It may incorporate false regulatory updates or manipulated treatment guidelines into recommendations. It may also generate structured medical documentation, including SOAP notes that place altered information in front of clinicians as clinical context.
These scenarios do not require access to sensitive patient data, but they can still shape medical conversations and decision-making. Trust has a central place in patient relationships. Digital tools that provide inaccurate or manipulated information can affect confidence in the institution behind those tools.
Healthcare carries particular sensitivity because patients often view health systems as trusted authorities. Information shown on an official website, patient portal or digital front door may be assumed to have undergone appropriate review. Even when an assistant does not issue a formal diagnosis, its responses may influence how patients interpret symptoms, manage medications or decide whether to seek care.
Security Controls Need Clinical Rigor
AI assistants increasingly form part of the clinical information environment. Their outputs influence conversations between patients and providers, which makes their integrity relevant to both safety and cybersecurity. The risk is not limited to whether data has been stolen or infrastructure has been disrupted. It also concerns whether the system continues to follow intended clinical and operational guardrails.
Healthcare organisations deploying AI assistants need to treat them as operational software systems, not simple digital chat tools. Direct interaction with patients and clinicians means their behaviour requires structured governance and testing. Controls should reduce the risk that conversational input can alter system behaviour or produce unsafe responses.
User inputs need validation and sanitisation before they reach the model, because prompt injection attacks often rely on hidden instructions inside normal-looking messages. System instructions also need separation from user conversations so attackers cannot easily override the guardrails that define the assistant’s behaviour. Clear separation between system prompts and conversational content makes prompt manipulation more difficult.
Monitoring can help identify abnormal responses or behaviour patterns. Logging and reviewing outputs can reveal situations where the assistant generates misleading or manipulated information. Adversarial testing before deployment can also expose weaknesses in prompt design and system architecture. Emerging AI security frameworks, including the OWASP Top 10 for Large Language Model Applications, offer a structure for considering risks such as prompt injection, data leakage and model manipulation.
AI assistants can help reduce administrative burdens and give patients faster access to information, but they also introduce risks that differ from conventional healthcare cybersecurity concerns. The behaviour of the assistant itself can become the attack surface. Governance, testing and monitoring therefore need to cover not only infrastructure and data protection, but also the integrity of AI-generated responses. As adoption expands across patient engagement channels, security controls need to match the sensitivity of clinical communication.
Source: Healthcare IT Today
Image Credit: iStock
