Rented a Car, Got My License Sold: What That Teaches You About Testing New AI Tools

The Rental Counter Problem

Are you handing your driver's license to a rental car kiosk without knowing what happens to that scan afterward? Most people are not asking that question at the counter, and most businesses are not asking the equivalent question before they plug a new AI tool into their workflow.

Rental car companies increasingly scan and store license data during checkout, sometimes routing that information through third-party verification vendors whose data-handling practices customers never see and never agree to in any meaningful way. You hand over the card, get your keys, and drive off with no idea where that image of your license actually lives now or who has access to it.

That is the same transaction happening every time a marketing team feeds real customer records into a new LLM just to see what it can do. Someone wants to test whether a model writes decent email copy, so they paste in an actual customer list to make the demo feel real. The model performs well. Nobody asks where that data went, whether it was used for training, or whether it sits in a log file on a server outside the company's control.

Testing a new AI tool and testing it safely are not the same exercise.

What Counts As A Demo Mistake

A demo mistake is not always dramatic. Most of the time it looks like enthusiasm.

Someone on the support team wants to see whether a chatbot can handle tricky tickets, so they copy three real customer complaints into a public tool and ask it to draft replies. The names are still attached. The account numbers are still attached. That transcript now lives somewhere outside the company, and nobody logged it as a data exposure because it did not feel like one.

Marketing does the same thing with unreleased copy. Someone wants to check whether a model can match brand tone before a product launch, so they paste in the actual campaign draft, the one embargoed until next Tuesday, into a chatbot with a free-tier account. The output is fine. The problem is that the draft is now sitting in a system with no contractual obligation to protect it, and no one on the launch team signed off on that risk.

Uploading a real client contract to see if a model can summarize it. Pasting payroll figures to test a spreadsheet assistant. None of these look like security incidents in the moment. They look like normal Tuesday testing, which is exactly why they keep happening.

A Sandbox Checklist Before You Test Anything

Before you paste a single real record into a new model, check the vendor's data retention policy. Every legitimate AI provider publishes one. It tells you whether your inputs get stored, for how long, and whether they get used to train future versions of the model. If you cannot find that page in under two minutes, treat that as your answer.

Business-tier plans usually disable training on your inputs by default, but usually is not confirmed. Log into the account settings and look for the toggle yourself. OpenAI, Anthropic, and Google all offer this control on paid business tiers. Free tiers generally do not, which is exactly why free tiers are where most of these mistakes happen.

Once you know the policy, build your test data instead of borrowing it. Write three fake customer complaints with fake names and fake account numbers that mimic the structure of real ones. Draft a placeholder campaign line instead of the embargoed one. The model does not need your actual data to show you whether it can write in your tone or summarize a contract clause correctly.

Open a throwaway account for the first round of testing. Separate email, no billing tied to your company, nothing connected to your CRM or your file storage. Confirm the tool behaves the way you expect before a real account ever touches it.

Judging The Model Without Exposing The Brand

Once you have a throwaway account and fake data, the actual evaluation work still needs to happen, and this is where most tests stall out. You are checking three things: does the output sound like your brand, does it come back fast enough to be usable, and does the math work at volume.

Tone matching is the easiest to check and the one people skip because it feels subjective. It is not. Feed the model five or six pieces of your fake but structurally accurate copy and ask it to write a sixth in the same voice. Read it next to something a real writer on your team produced last month. If a coworker cannot tell which one is which, that is a pass. If it reads like a press release when your brand is casual, that is your answer before you spend another hour on it.

Latency matters more than most teams budget for. A chatbot that takes four seconds to respond feels broken to a customer waiting on hold. Time ten fake queries end to end and average it.

Cost per call is math, not vibes. Multiply the vendor's per-token or per-minute rate by your expected volume for a month, not a day. Small tests hide big bills.