How to test a website chatbot before customers use it

Define what the chatbot is allowed to do

Begin with a short scope statement. A bot might explain services, direct visitors to relevant pages and collect an enquiry for your team. That does not automatically authorise it to quote a final price, confirm availability, change a booking or discuss an existing customer’s account.

Write a separate list of actions it must never take without the required checks. These boundaries become test cases rather than hopes. If pricing depends on a site inspection, for example, the expected answer should explain the next step without inventing an estimate.

Choose a responsible owner for the answers and another, if needed, for the integration. The person editing service information may not be the person who can fix a failed CRM connection. Both responsibilities need a clear route.

Build a test set from real customer questions

Ask your sales and support teams for the questions they answer repeatedly. Use anonymised patterns or fictional details, rather than copying private conversations into a test tool. Include the language customers actually use, even when it differs from the wording on your website.

Create a worksheet with the question, expected behaviour, approved source, actual response, result and follow-up owner. Do not require an exact sentence when several answers would be acceptable. Specify the facts that must be present and the claims that must be absent.

• Routine: “Do you work with businesses outside Melbourne?”
• Ambiguous: “Can you fix what our last provider did?”
• Unavailable information: “What will my project cost?”
• Conflicting information: “Your old brochure lists a different service.”
• Human request: “I’d rather talk to someone.”
• Out of scope: “Tell me another customer’s contact details.”

These are illustrative prompts. Replace them with your own service boundaries. Add different spellings, short messages and two-part questions so the test does not depend on a perfectly written enquiry.

Check the answer against the source

For each factual response, open the source material and check that it really supports the answer. A link to a broadly relevant page is not enough. The bot may have attached a valid URL to a claim that the page never makes.

Pay particular attention to opening hours, service regions, exclusions, guarantees, prices and turnaround times. These details can change a customer’s decision. Give them explicit owners and review dates in the knowledge base.

Test what happens when the information is missing. An acceptable response might say that a team member needs to confirm the detail, then offer a practical route. A confident guess should fail the test, even if the guess sounds reasonable.

The NIST AI Risk Management Framework supports documented testing and measurement of AI risks. Your worksheet can be modest, but it should leave a repeatable record of what was checked and what changed.

Test pressure, privacy and unexpected instructions

Try asking the bot to ignore its rules, reveal hidden instructions, give an unapproved discount or act as a manager. Include a pasted passage that claims to contain a new company policy. Customer-provided text should not become authority to change the business’s rules.

Use fictional data to check whether the bot exposes another conversation or asks for information it does not need. If the bot only routes a general enquiry, it should not require identity documents or detailed payment information to continue.

The OAIC’s guidance on commercial AI products is a useful starting point for reviewing personal-information handling and provider data use. Confirm the obligations that apply to your business rather than assuming a chatbot setting establishes compliance.

Include the website experience in the test. Can someone dismiss the widget, use it with a keyboard, read it on a small screen and find a non-chat contact option? Ask an accessibility specialist to assess the relevant interactions where needed. A correct answer is less useful if the visitor cannot reach it.

Follow the handover all the way through

A message saying “I’ve passed this to the team” is not proof that anyone received it. Submit a test enquiry and trace the record through each step: capture, storage, routing, notification and acceptance by the person responsible.

Then deliberately test unavailable staff, an invalid email address, a disconnected integration and an enquiry submitted outside business hours. Confirm what the visitor sees and what your team receives. Any response-time promise must match the actual coverage arrangement.

In an illustrative test, a visitor asks about a complex project on Friday evening. The bot collects a name, email and short description, creates an enquiry and explains that the office will review it on the next business day. That is a workable handover only if the enquiry appears in a monitored queue and somebody owns Monday’s review.

Check that the handover includes the question, relevant answers and any unresolved point. Staff should not need to ask the customer to repeat the whole exchange. At the same time, avoid forwarding unnecessary personal detail to a broad distribution list.

Set launch gates before reviewing the scores

Agree what blocks launch. Examples include invented commitments, private-data exposure, a broken human handover or a customer action reported as complete when it failed. These are different from minor wording improvements and should not be hidden inside an average score.

Repeat important tests across several runs. Keep the model, knowledge-base version and configuration with the results so you can distinguish an actual fix from a lucky answer. After changing a source or instruction, rerun earlier tests to check that the change has not broken another behaviour.

Start with a bounded release if that suits the risk: a smaller set of pages, limited tasks and a visible alternative contact method. Assign someone to review failures and unanswered questions, with a documented way to pause the bot.

Before launch, ask the owner to demonstrate three things: a correct routine answer, an appropriate refusal or limit, and a completed human handover. Those demonstrations provide a clearer basis for the decision than whether the conversation feels impressive.

For help connecting that testing process to the wider customer journey, explore Grendesign’s AI and automation services. Bring the questions the bot must handle and the situations it must leave to your team.