We probed OpenAI's plugin search with 1,000 queries and a paired prompt evaluation. The catalog is small, skewed toward business software, and models rarely consult it unless the user asks.

Yesterday OpenAI announced that it would support dynamic discovery of plugins, in other words, would suggest using plugins when a user query matched the capabilities of a plugin present in the store.

To test to what extent these capabilities had already been rolled out, we ran 1,000 search queries through the plugin search that OpenAI exposes to its models in Codex, and across all of them it surfaced 152 distinct plugins. Of those, 81 serve business users only, 57 serve both audiences, and 13 are aimed squarely at consumers. In a separate evaluation still in progress, the models we tested never searched for a plugin on their own in 18 task-shaped prompts, even when the task plainly called for one. Asked directly whether a plugin could help, they searched almost every time.

How plugin discovery works today

Plugin discovery runs through a plugin-management skill with two tools, one that searches the catalog and one that formally suggests a plugin to the user, who can then install it. According to a leaked version of the skill's description, the model should reach for it when the user asks about plugins or when a task would materially benefit from an external app, service, or data source it can't otherwise access. Nothing in that description forces a search, so whether discovery happens at all is left to the model's judgment on each turn.

A catalog of 152, weighted toward business software

The 1,000 queries covered several hundred verticals, from restaurant reservations and telehealth to maritime logistics. The business-only plugins are mostly CRMs, analytics, finance, and project management tools, while the consumer side comes down to four travel brands (Booking.com, Expedia, Skyscanner, Tripadvisor), Instacart, and a handful of fitness trackers.

Linear, Attio, and HubSpot each appeared in roughly 240 of the 1,000 result sets, largely because a query like "retail CRM sales pipeline" or "banking CRM sales pipeline" returns the same cluster of sales tools whatever the industry.

Consumer categories are sparse enough that 24 queries returned nothing at all, among them music streaming, telehealth, pet sitting, secondhand fashion, meditation, and homework tutoring.

Some kinks to work out…

776 of the 1,000 queries hit the ten-result ceiling, though many of those results had little to do with the query. "Restaurant reservations" returned WordPress.com, and "medical appointment booking" returned Calendly alongside QuickBooks, NetSuite, Skyscanner, and GitBook.

A model receiving these results has to filter the noise itself, and our evaluation shows it often concludes that nothing fits.

Models look for plugins when asked, and rarely otherwise

Alongside the catalog probe, we're running a prompt evaluation to see when models reach for plugin search on their own. It takes 200 user tasks, half business and half consumer, and phrases each one twice, once as a user naturally would and once followed by "Does ChatGPT have a plugin that can help with this?" Every prompt runs in a fresh Codex context with account access disabled.

The evaluation is still running, but our initial findings suggest that plain prompts virtually never trigger a plugin search, while the explicit question triggers more than 90% of the time. However, even when the search is triggered, the times when a formal suggestion is made drops off steeply. We'll update this article once the sample is large enough to support firmer conclusions.

What comes next

Plugin search gives a first look at how OpenAI intends to route users to apps, but the flow is clearly still in progress, and the catalog, the search results, and the models' willingness to use them will likely shift in the coming weeks. We've set up Atlas to track plugin search over time, so you can follow those changes as they happen!
Check it out at https://atlas.alpic.ai/discoverability