What can we learn from ChatGPT Search Sources?

Can we learn/infer anything about the prompt/inner workings of ChatGPT’s search when sources change?

It’s been some time since I dug into ChatGPT’s Search telemetry, but since the search sources have started being exposed, I wanted to understand whether ChatGPT consistently uses the same retrieval source for the same prompt and, if not, whether changing search source materially changes the websites used to generate answers.

Thanks to Mark Willliams-Cook for his inspiration here.

Can we learn/infer anything about the prompt/inner workings of ChatGPT’s search when sources change?

To try and understand this, I have taken 1,000 prompts - devided into different intent types and actions (not based on “realistic” traits, but geared to illicit different response types, intents) and tracked them 10x each.

Interestingly only 9,946 runs successfully completed (not the full 10k), which shows that not every query returned search results.

From this, I captured the telemetry data (this all via Bright Data as a scrape source - ironic given that it’s a ChatGPT Search source AND it also presents potential issues in the data, more on both below!).

It’s also worth noting Suganthan’s excellent research on this same subject here.

Findings

Up front, it is clear that this particular set biases one search source above the rest:

Primary search sources observed:

• Labrador: 88.1%

• Bright: 9.9%

• Oxylabs: 1.7%

• SERP: 0.3%

This seems stable, as 88.4% of prompts always have the same source across runs, and that Labrador in most circumstances.

However, 11.6% changed primary search source during repeated executions, while 13.4% prompts showed changes in their observed search source combinations. 

Although a minority, this is meaningful and demonstrates that backend source selection is not completely fixed. What impacts this? Great question.

The most common intra-prompt (changes between checking the same prompts) search source changes involved Bright(data) appearing alongside Labrador or Oxylabs, suggesting that Labrador appears to act as the default retrieval source while Bright(data) is introduced for specific types of queries.

As you might expect, different search sources returned meaningfully different result sets. When the same prompt remained on the same search source, URL overlap between runs was substantially higher than when the backend changed.

Domain overlap showed the same pattern. Mixed-source prompts produced larger and more diverse domain selections than prompts that consistently stayed on a single backend.

• URL overlap drops from 0.273 to 0.149 when search source changes.

• Domain overlap drops from 0.265 to 0.155 when search source changes.

• That is roughly a 45% lower URL overlap and 42% lower domain overlap.

Although backend selection changes for only a minority of prompts, those changes have a disproportionately large impact on the retrieved URL and domain sets. In other words, routing appears relatively stable, but when it does change, it meaningfully changes what ChatGPT considers as evidence.

For tracking and measuring outputs, the search source is a meaningful datapoint to capture - where possible - as it’s an important factor in what is returned.

Rather than thinking about “ranking in ChatGPT” as a single ecosystem, there are multiple, retrieval ecosystems, each drawing on partially different sets of websites and domains.

Observed Search Source Characteristics

This is a small data set, but we can see some emerging characteristics:

• Labrador generally returns smaller, more focused retrieval sets.

• Bright retrieves more URLs, more citations and a wider variety of domains.

• Oxylabs appears relatively infrequently but shows comparatively diverse domain selection.

• SERP appears rarely and is strongly associated with news-style prompts.

Prompt characteristics appear associated with search source selection. 

Evergreen educational and factual prompts overwhelmingly use Labrador, while prompts involving current events, regulation, surveys, recent developments or explicit date references are considerably more likely to trigger Bright, Oxylabs or SERP.

The amount of URLs retrieved meaningfully changes by search source also:

https://datawrapper.dwcdn.net/VpqdF/1/

The telemetry also contains A/B testing metadata, including experiment names that reference shopping and Bright Data feed serving. While interesting, these fields were in really low number - too low to say anything useful with.  

They do, however, indicate that more elements of retrieval are being actively experimented with.

What else may impact Source Selection?

The telemetry includes a web_search_query field (for fan-out), but within this dataset it almost always mirrors the original user prompt rather than exposing internal query fan-outs. 

Because all telemetry was collected through Bright Data, this should currently be interpreted as representing Bright Data-mediated ChatGPT sessions rather than all ChatGPT users. The lack of richer fan-out telemetry is likely a limitation of the capture method rather than evidence that ChatGPT performs only a single search query. If I search any of these prompts “logged-in” via a paid account, the output is much richer.

As a result, query expansion remains an important unanswered question. This element may heavily impact other parts of the process, source selection just being one.

Another limitation is that search source names such as Labrador, Bright, Oxylabs and SERP appear to represent search sources, not end-user search engines. Their precise role within OpenAI’s retrieval architecture is not publicly documented.

Implications

For a small test, there are some fairly significant implementations

• Search source should become another dimension recorded during AI visibility studies wherever telemetry makes it available. Ignoring search source risks treating materially different retrieval processes as though they were identical.

• Optimizing for ChatGPT is different from other chatbots - we knew that already - but we introduce another element of optimizing (or at least) understanding the different search sources within the tool. How do we optimize for those?

• Potentially some even larger differences between tracking logged-out, free instances of ChatGPT vs those logged-in and/or paid for.

We need to capture more, test more and be most fastidious to really be conscious of all these facets and still there are parts that we’re making educated guesses at (at best). The longer I look at this data, the more I think Dixon (and WAIKAY’s) approach to being more interested in your presence within the model data (rather than search pipeline) is a sane(r) route to analyzing this data.

Further points to investigate.

There is so much more to look at here, some I’ll try and get to, but part of my hopes the community may want to take a look at also.

• Comparing Bright Data captures with normal browser telemetry from logged-in ChatGPT accounts. Most likely to show different elements I think!

• Measuring retrieval differences between Free, Plus and Pro accounts.

• Determining whether small prompt changes consistently alter backend routing. The data here suggests that it is more stable than expected, but this can be probed a lot more.

• Identifying the characteristics that trigger Bright, Oxylabs and SERP instead of Labrador.

• Analysing the websites and domains most strongly associated with each search source. Can we use this to check against Google/Bing/Brave SERPs and other data sources to understand what the customer-facing version of these data sources may be.

• Understanding how search source selection impacts the final answer generation

• Investigating internal query fan-outs using alternative capture methods capable of exposing richer retrieval telemetry. If the source and fanned-out queries change, these are significant multipliers.

If you want access to any of the data/further insights - please reach out!