A homeowner asks ChatGPT for a plumber. They ask again. They get a different answer.
A contractor checks ChatGPT for their business name and sees it in the results. They check again the next day and it is gone. They check a third time and it is back, but with a different competitor listed alongside.
This is not a glitch. It is how AI search works.
Rand Fishkin and the team at SparkToro published research confirming what many contractors have already noticed firsthand: AI tools are probability engines. Ask the same question 100 times and you will get close to 100 different answers. The business that appears in one response may disappear from the next. The list of recommendations changes length, order, and content with nearly every query.
For a contractor who checked once, saw their business, and assumed the problem was solved, this is a problem. One good result on a Tuesday afternoon does not mean the same homeowner asking the same question on Wednesday morning will see the same answer.
Why AI Answers Change Every Time
Traditional search engines rank pages. AI search engines generate answers. The distinction matters.
Google reads its index, applies a ranking algorithm, and returns the same ten links in roughly the same order for the same query. The results shift over time as the algorithm updates and competitors optimize, but the basic structure is stable. Rank one on Tuesday, rank one on Wednesday.
ChatGPT, Perplexity, and Gemini work differently. They run a fresh retrieval for each query, pull from whatever sources are currently available and relevant, and synthesize a unique response. The response is generated word by word based on probability, not pulled from a pre-built ranking. Each generation is a new draw from the same pool of sources, but the draw is never identical.
SOCi's 2026 Local Visibility Index describes this as the difference between ranking and recommendation. Google ranks businesses. AI tools recommend them. Ranking is deterministic. Recommendation is probabilistic. The same homeowner asking the same question twice may get two different plumbers.
What This Means for Trades Contractors
The implication is straightforward but uncomfortable. You cannot check AI search once and conclude you are visible or invisible. One query is a single sample from a distribution that changes every time.
Here is what that means in practice for a plumber, HVAC contractor, or electrician.
A single check is not a measurement. If a contractor types "best plumber in Bergen County" into ChatGPT, sees a competitor, and assumes they are losing, they may be wrong. The competitor may appear in 30% of queries and the contractor may appear in 25%. The gap is real but narrow. One query does not reveal that.
Visibility is a percentage, not a yes or no. The right question is not "does ChatGPT recommend my business?" The right question is "what percentage of queries does my business appear in?" A contractor who appears in 20% of ChatGPT local queries for their trade and county is in a better position than one who appeared once and never again.
The distribution favors businesses with stronger signals. AI tools are probabilistic, but the probabilities are not random. Businesses with more third-party mentions, more detailed reviews, more complete directory listings, and better-structured data appear in a higher percentage of generated answers. The inconsistency is in the output, not the input. The inputs that drive AI recommendations are stable and improvable.
How to Measure AI Visibility When Answers Keep Changing
Since a single query is not a measurement, contractors need a sampling approach. The method is simple but requires repetition.
Run the same query 10 to 20 times across different days and different times of day. Use the exact phrasing a homeowner would use: "I need a plumber in [county]. Who do you recommend?" Record which businesses appear in each response. Track the percentage of queries where your business shows up.
That percentage is your share of voice for that query. A contractor who appears in 4 out of 20 queries has a 20% share of voice. A competitor who appears in 12 out of 20 has 60%. The gap between those numbers tells the real story.
Do this across multiple query variations. "Best plumber near me," "emergency plumber [city]," "who fixes burst pipes in [county]" are all different queries that pull from different parts of the AI's source pool. A contractor might dominate one phrasing and be absent from another.
Run the same test on ChatGPT, Perplexity, and Gemini separately. Each platform has its own retrieval patterns and source preferences. A contractor who is invisible on ChatGPT but visible on Gemini has a specific gap to close, not a general failure.
The Action Plan for Inconsistent AI Search
- Run 10 to 20 queries per platform, per trade, per county. Use the exact phrasing homeowners use. Record which businesses appear. Calculate your share of voice as a percentage.
- Test multiple query variations. "Best plumber near me," "emergency plumber [city]," "plumber for water heater installation [county]" are all different searches to the AI even if they feel similar to you.
- Check all three major platforms. ChatGPT, Perplexity, and Gemini each pull from different sources and generate different answers. Track them separately.
- Re-test monthly. AI answers shift as sources update, new content gets indexed, and platforms adjust their retrieval patterns. A single snapshot goes stale within weeks.
- Focus on the inputs, not the outputs. You cannot control which answer an AI generates on a given query. You can control your review content, directory completeness, third-party mentions, and structured data. Those inputs shift the probability in your favor over time.
- Do not panic over a single bad result. If your business disappears from one query, run nine more. The pattern across 20 queries tells the story. One query is noise.
The Bigger Picture for Trades Contractors
45% of consumers used AI tools for local business recommendations in the past year, up from 6% the year before, according to BrightLocal's 2026 Local Consumer Review Survey. Those consumers are not running 20 queries and calculating share of voice. They ask once, get an answer, and act on it.
That means every single query matters, even though the results are probabilistic. A contractor who appears in 40% of ChatGPT queries for their trade and county is the business a homeowner will see on their first try, most of the time. A contractor who appears in 5% will almost never be the answer a homeowner gets on the first try.
The goal is not to appear in every AI answer. The goal is to appear in enough of them that the first query a homeowner runs is likely to include your business. That requires consistent signals across Yelp, your Google Business Profile, and the broader web. Learn more about what AI visibility means for your trade.
FAQ
Why does ChatGPT give different answers every time I ask the same question?
AI tools are probability engines, not ranking systems. Each query triggers a fresh retrieval and generation process. The AI pulls from currently available sources and synthesizes a unique answer based on probability, not a pre-built list. The same question can produce different results each time because the generation process is non-deterministic.
How many times should I check ChatGPT for my business?
Run the same query 10 to 20 times across different days and times. Record which businesses appear in each response. Your share of voice is the percentage of queries where your business shows up. One check is a single sample, not a measurement.
If AI results are inconsistent, does AI visibility even matter?
Yes. The inconsistency is in the output, not the inputs. Businesses with stronger signals (more reviews, better directory listings, more third-party mentions) appear in a higher percentage of generated answers. A contractor who appears in 40% of queries will be the answer most homeowners see on their first try. A contractor at 5% will almost never be the first answer.
Should I track ChatGPT, Perplexity, and Gemini separately?
Yes. Each platform uses different retrieval patterns and source preferences. A contractor might be visible on Gemini but invisible on ChatGPT. Track each platform separately and identify which specific gaps to close.
How often should I re-check my AI visibility?
Monthly at minimum. AI answers shift as sources update, new content gets indexed, and platforms adjust their retrieval algorithms. A snapshot from one month may not reflect the next month's results.
Run your free AI visibility audit to see what ChatGPT, Gemini, and Perplexity find when homeowners search for contractors in your area.