Broker guide
AI Hallucination: Why AI Makes Things Up and How to Check
AI tools can invent lender rules, figures and sources. See why hallucinations happen, real Australian cases and how a broker checks every answer.
- Published
- Updated
An AI hallucination is an answer that presents false or unsupported information as fact. In mortgage broking, that can mean an invented lender rule or a real policy quoted with a condition missing.
Confidence doesn’t tell you whether the answer is correct. Before relying on a policy answer, open its source and compare the claim with the lender’s wording.
What an AI Hallucination Looks Like
An AI hallucination can sound like a complete, well-reasoned answer even when its central claim is wrong. It can also include a citation that looks genuine.
In a hypothetical example, an assistant says a lender accepts a self-employed applicant with one year of trading. The supplied policy actually requires two years. The assistant has turned a plausible lending rule into a false statement about that lender.
The same problem appears when an answer invents a document title, changes a percentage or attributes a quotation to the wrong source. A real document can also be cited beside a claim it doesn’t support.
Treat fluent wording and a source link as separate from factual support. The question is whether the source establishes the particular claim you want to use.
Why AI Hallucinates
A language model generates likely text from patterns learned during training and the context supplied to it. A likely continuation can be factually wrong.
A connected search or document tool can supply evidence, but the model still generates the explanation. It can misread a passage, combine incompatible conditions or fill a gap with a plausible answer. The OAIC’s guidance on AI products explains how this predictive process can produce credible-looking inaccuracies.
For a broker, a broad question such as “Which lenders accept this income?” also leaves material facts unstated. An assistant that assumes the missing facts can return an answer that doesn’t fit your client.
Old information creates another accuracy problem. A model can repeat a rule that was once correct after the lender has changed it. Even an accurate quotation needs a current, applicable source.
Asking the same assistant whether it is sure gives you another generated answer. Use an external source to resolve the claim.
How Often AI Gets It Wrong
AI hallucination rates depend on the model, task and scoring method. A result from a legal research test doesn’t tell you the error rate for Australian lender-policy answers.
Two published studies show how much the task changes the result.
| Study | Task tested | Reported hallucination rate | What the figure describes |
|---|---|---|---|
| Dahl and colleagues, 2024, Large Legal Fictions | Specific, verifiable questions about randomly selected US federal court cases | 58% for GPT-4 and 88% for Llama 2 | Those model versions answering that legal task |
| Magesh and colleagues, 2025, Hallucination-Free? | Questions put to three specialist legal research systems using document retrieval | Between 17% and 33% across the systems | The study’s legal questions, systems and definition of hallucination |
These are historical study results. They aren’t a current accuracy rating for ChatGPT or a prediction that a particular percentage of your client files will contain errors.
The denominator also changes the meaning. A percentage of responses containing an error differs from a percentage of individual factual claims. Refusals and unanswered questions need their own treatment, because declining to answer is different from inventing an answer.
For a lender-policy question, assess the claim you will rely on. A low error rate elsewhere can’t establish that this lender accepts this income or security.
Real AI Hallucination Cases
Documented cases show invented references surviving into professional work, including an Australian government report and court material.
Deloitte’s Australian Government Report
In 2025, Deloitte agreed to repay the final instalment for a report prepared for the Department of Employment and Workplace Relations. The report contained fabricated references and an incorrect purported court quotation, according to reporting by the Australian Computer Society.
A revised report corrected references and disclosed generative AI use. The department’s report page also records a further corrected version in 2026.
The case illustrates a checking failure. A formal report with footnotes can still contain a source that doesn’t exist.
Invented Authorities in an Australian Court
In Dayal, a 2024 Australian family-law matter, a solicitor tendered AI-generated authorities that didn’t exist. The court’s reasons record that neither the solicitor nor another practitioner had checked the output.
The Victorian Legal Services Board later varied the solicitor’s practising certificate in 2025. The professional using the answer remained responsible for checking it.
Fake Citations in Mata v. Avianca
In the 2023 US case Mata v. Avianca, lawyers submitted nonexistent judicial opinions with fabricated quotations and citations generated by ChatGPT. The court’s sanctions decision describes their continued reliance on the material after its existence was questioned.
Each case involves a source that needed checking outside the generated answer. Repeating the citation in another document wouldn’t make it genuine.
How to Check an AI Answer
Check an AI answer by finding the source, reading the relevant passage and confirming that it applies to the client’s circumstances. Complete the check before copying the claim into file notes or using it to shortlist a lender.
- Ask for the exact source and passage supporting the claim. Request the lender’s document name and the relevant section or page.
- Open the source yourself. Use the lender’s broker portal or official document where access is required, rather than relying on a generated summary.
- Compare the quotation with the original. Read the surrounding conditions and exceptions, including any footnotes that change eligibility.
- Confirm the policy’s date and applicability. Check whether a replacement or update changes the rule for the loan type or application you are assessing.
- Recheck figures independently. Keep pay frequency, assessment periods and calculation inputs beside the amounts they affect.
- Record what you verified. Retain the source version and relevant wording with the file, along with any lender clarification needed for an ambiguous case.
For the hypothetical trading-history example, the check fails when the source says two years but the assistant says one. Correct the claim and reassess the lender shortlist before using the answer.
An AI hallucination checker or detector can flag a possible contradiction. It doesn’t replace the source comparison. Asking a second chatbot is useful only when you also inspect the evidence it supplies.
Choose Tools That Show Their Sources
Tools that retrieve and quote source documents make checking easier because you can compare the answer with the evidence it used. Retrieval can still select the wrong passage, and a generated explanation can still misstate a condition.
Bulma’s Policy Advisor answers lender-policy questions across 52+ lenders and quotes the policy wording behind each answer. It also shows the lender, policy area and when Bulma last updated that policy. That update date describes Bulma’s record, so distinguish it from the lender’s own document date.
Use the quoted wording as the starting point for your check. Open the source and resolve any condition that changes the client’s eligibility before relying on the answer.
