Using AI safely in CF care

If you read only this

Each sourced sentence on this page is checked by a script against an exact quote from what we read of its source. The advice is the lab's own and is marked as ours. The sources, and how much of each we read, are listed at the end.

Why people turn to AI

What can go wrong

Each topic starts with what to do; that advice is ours. Open Sources under it for the sourced sentences, how strong they are, and the originals. Strong means a source answers the question directly and we read it in full; Limited, that it is small, indirect or partly read; Not found, that we found nothing to rely on. No topic is rated Strong yet. Several topics cite the American Psychological Association (APA), whose advisory on AI chatbots in mental health care we read in part.

• Made-up facts and sources

  Open every source a chatbot gives you. If you cannot find it, or it does not say what the chatbot said, do not rely on that point.

  Sources Evidence: limited

  In plain words. In one study of four AI search engines, only about half of what they wrote was fully backed by the sources they cited. In another study, a chatbot made up a large share of its sources, more with one version of its model than with another.

  In a study of four popular AI search engines, on average 51.5% of the sentences they generated were fully supported by their citations. In that study, on average 74.5% of citations supported their sentence. In another study, a chatbot wrote short literature reviews: 55% of the citations made with a version of its model were made up, and 18% with another version.

  Limited. Both studies were read in full from open-access text; one is a preprint. While both confirm citation fabrication and unsupported statements, they evaluate general search engines and multi-topic literature prompts rather than direct cystic fibrosis care. Originals: Liu and others (preprint); Walters and Wilder (Scientific Reports).

• A good test score is not good help

  A high score on a medical exam says little about how well a tool helps a person. Bring what it said to your care team rather than acting on it alone.

  Sources Evidence: limited

  In plain words. On its own, a chatbot named the right conditions in almost every case in this study. People who used the same chatbots did much worse, and no better than people who used any source they chose.

  Near-perfect scores on medical licensing exams do not necessarily mean accurate help in real-world use. In a study with 1,298 people and ten medical scenarios, chatbots tested alone named the right conditions in 94.9% of cases. People who used the same chatbots named relevant conditions in less than 34.5% of cases. They did no better than a comparison group, who were randomly assigned to use a source of their choice. The authors recommend testing with real people before these tools are offered to the public in health care.

  Limited. One preprint, read in full from open-access text. While it tests 1,298 participants across ten simulated medical vignettes, it evaluates lay diagnosis rather than clinical care or cystic fibrosis. Original: Bean and others (preprint).

• Answers that agree with you

  Ask open questions. Instead of “My plan is fine, right?”, ask “What could be wrong with this plan, and what should I ask my care team?” You can also ask for the best case against the answer it gave. For anything new or worrying about your health, follow the first rule above instead.

  Sources Evidence: limited

  In plain words. In tests on writing tasks, five leading AI assistants consistently showed sycophancy: answers that match what a person believes rather than what is true. People, and the systems used to train these assistants, sometimes preferred convincing, agreeable answers to correct ones.

  Researchers use the word sycophancy for answers that match what a person believes over what is true. In a study, five leading AI assistants consistently showed sycophancy across four kinds of writing tasks. People, and the models used to train assistants, sometimes preferred convincing, agreeable answers over correct answers. The APA notes that many chatbots are designed to validate and agree with what users say.

  Limited. One preprint, read in full from open-access text, which reports sycophancy in tests of writing tasks in general rather than clinical health; the APA advisory also raises it. Originals: Sharma and others (preprint); APA health advisory (PDF).

• Trusting the machine too much

  If a chatbot and your care team disagree, ask your team about it. Care teams: form your own view first, then look at the tool.

  Sources Evidence: limited

  In plain words. Combining some of its studies, a review of computer tools that advise clinicians found that people using a tool followed wrong advice more often than people without one. The review was not about chatbots.

  Pooling some of its studies, a review of clinical decision support systems (computer tools that advise clinicians) found that wrong advice was followed more often by people using these tools than by people without them.

  Limited for chatbots. The systematic review was read in full from open-access text; it examines clinical decision support tools for clinicians in general rather than consumer chatbots or cystic fibrosis. Original: Goddard and others (review).

• Privacy: where your words go

  Leave out names, dates of birth and anything that identifies you or someone else. Leave out photos and screenshots of letters, results and medicine lists too. Check your own tool's setting on training with chats, and its privacy page.

  Sources Evidence: limited

  In plain words. The US Federal Trade Commission (FTC) says its rule requiring notice when health information is hacked or shared without permission probably covers a health app that collects information and can sync with a fitness tracker, but not hospitals, doctor’s offices or insurers covered by HIPAA. One AI company says it may use consumer chats to improve its AI if the user allows it, and may use chats flagged for safety review. The APA says to be extremely cautious about sharing sensitive information and to avoid entering details that identify you.

  The US Federal Trade Commission (FTC) says a health app that collects information from people and can sync with a fitness tracker is probably covered by its Health Breach Notification Rule. Under that rule, a breach is not limited to a hack: sharing covered health information without the person's permission can also require notice. The rule does not apply to hospitals, doctor's offices or health insurers that are covered by HIPAA.

  An example: An AI company's help page says it may use consumer chats to improve its models if the user chooses to allow it. That page also says the company may use or analyze conversations that are flagged for safety review. The APA advises the public to be extremely cautious about sharing sensitive information and to avoid entering details that identify them.

  Limited. Each sentence matches what these pages said when we read them, but the evidence is thin overall: US rules only, one company as an example, and we could not read the US health department's pages on HIPAA, so we do not say how HIPAA applies to chatbots. Originals: FTC guidance; the company’s help page; APA health advisory (PDF).

• AI for emotional support

  If anyone is thinking about hurting themselves, or is in a crisis, call your local emergency number or a crisis line where you live now. Your care team can help with what comes next. If chats leave you feeling worse or more alone, take a break and talk with someone you trust.

  Sources Evidence: limited

  In plain words. The APA says AI chatbots and wellness apps should not replace a qualified mental health care provider, though they may support ongoing therapy. It reports unsafe interactions with vulnerable people, such as children, and says their ability to handle a crisis safely is limited and unpredictable. It also says large, high-quality clinical trials of these tools in mental health care are lacking.

  The APA says AI chatbots and wellness apps should not replace a qualified mental health care provider, though they may be a support alongside ongoing therapy. It reports that these chatbots have already had unsafe interactions with vulnerable people, such as children. It says their ability to handle a person in crisis safely is limited and unpredictable. It also says there is a lack of large, high-quality clinical trials of these tools in mental health care.

  Limited. This is the advice of a professional body (the APA), which itself says the trials are missing. We read the first seven pages in full and searched the rest by keyword. Original: APA health advisory (PDF).

• Language and translation

  For anything about medicines or care, ask your care team for a professional interpreter or translated material. If you use a chatbot to translate, keep the original. Do not rely on a chatbot translation of a medicine or care plan unless your care team or a professional interpreter has checked it.

  Sources Evidence: limited

  In plain words. In one study of the instructions families get when a child leaves the hospital, chatbot and machine translations into Haitian Creole had more errors that could matter for a child’s care than professional translations did. For Spanish and Portuguese, the study found them comparable to professional translations. The APA adds that most AI language models are trained mainly on English-language content from the internet that is centered on Western countries.

  The APA notes that the training data for most AI language models are not public, and consist mainly of English-language, Western-centric content from the internet. In a study of discharge instructions for children, chatbot and machine translations into Haitian Creole contained more potentially clinically significant errors than professional translations. For Spanish and Portuguese, that study found them comparable to professional translations.

  Limited. One translation study, and we read a summary of its abstract, not the paper. Originals: APA health advisory (PDF); translation study in Pediatrics.

• Your own rewrite has no checker

  Use a rewrite to understand, and keep the original. Ask your care team about anything that would change what you do. Care teams: an AI rewrite or translation is a draft that a person checks before it is used.

  When a clinic rewrites a leaflet, someone can check it before it goes out. When you ask a chatbot to rewrite a letter or a result for you, nobody checks it. That is our reasoning, not a study finding.

  Sources Evidence: limited

  In plain words. One study found that chatbots might be able to make online patient materials easier to read while keeping them accurate. Its authors say the models varied and made mistakes, so a person needs to review the result.

  A study found that chatbots have the potential to make online patient materials easier to read while keeping them accurate. Its authors say the models varied and made mistakes, and that this shows the need for human review.

  Limited. One study of rewritten web materials; none of the studies we read looked at rewrites people make for themselves. Original: Will and others (JMIR).

• Who checks the tool?

  Ask who made a tool, who tested it, and whether it is approved for your use. More questions are below.

  Sources Evidence: limited

  In plain words. The APA says most AI chatbots and wellness apps used for mental health are not designed to provide treatment and have not received regulatory approval, and that current regulatory frameworks are not adequate for AI in mental health. The US Food and Drug Administration (FDA) says its digital health policies still cover software that counts as a medical device, even when the software is meant for patients or caregivers.

  The APA says most AI chatbots and wellness apps used for mental health are not designed to provide treatment and have not received regulatory approval. The APA also says current regulatory frameworks are inadequate for the realities of AI in mental health. In the US, the Food and Drug Administration (FDA) says its digital health policies still apply to software that counts as a medical device, including software meant for patients or caregivers.

  Limited. The APA writes about mental health tools, and for the FDA we read only the page that introduces its guidance, not the guidance itself. Originals: APA health advisory (PDF); FDA page.

• Facts that change or are easy to get wrong

  Treat insurance rules, benefit amounts, phone numbers and web addresses from a chatbot as unchecked until you find them on the official page. Medicine labels change, so check the date on the label you read. Questions about who can get a medicine belong with your care team.

  Sources Evidence: not found

  We found no study on these, so the rules above are the lab’s own. Originals: none.

Tell your care team

Questions to ask about any AI tool

What the lab does about this

What this page does not do

Where this comes from