Searching OpenAPI Specifications Without Filling Context

In the previous article, we learned what Code Mode is and how useful it can be to let the model write orchestration code instead of calling tools directly.

In this third part of the series, we will explore how we can use Code Mode when the context window is limited or when we have a document—an OpenAPI specification in our case—that is larger than the context window.

The context window problem ​

When developing a service or an application, we often build an API to allow other services to interact with our system. Developers traditionally learn how to use that API through documentation websites, such as the Cloudflare API reference.

To make those websites easier to build, maintain, and keep in sync with the actual API, the industry created a specification format: OpenAPI.

openapi: 3.0.0
info:
  title: Sample API
  version: 1.0.0
paths:
  /users:
    get:
      summary: Get all users
      responses:
        '200':
          description: A list of users
          content:
            application/json:
              schema:
                type: array
                items:
                  type: object
                  properties:
                    id:
                      type: string
                    name:
                      type: string

This example shows a simple OpenAPI specification for an API that has a single endpoint /users which returns a list of users. The specification describes the endpoint, the HTTP method, the expected response, and the data format.

This is powerful for programmatic use and can generate useful human-facing documentation. However, a large raw specification is difficult to navigate manually and can exceed a model's context window. In those cases, we cannot simply provide the complete document to the model and expect it to use it effectively.

• GitHub's GHES 3.21 OpenAPI specification is 11.1 MB.
• Cloudflare's OpenAPI specification is 22.7 MB.

These files evolve, so their exact sizes will change. The point is that real API descriptions can be far larger than the information needed for a single task.

The Cloudflare snapshot used in this series contains 23,792,523 characters. The rough shortcut of four characters per token suggests about 6 million tokens, but tokenization varies by model and content. Either way, it is much larger than many available context windows.

We cannot simply provide the entire specification to the model.

OpenAPI specifications are for people and machines ​

OpenAPI is designed to let both people and software understand an API. A documentation generator can turn it into a website; a client generator can turn it into code. For a model, however, returning the complete specification as context is still usually the wrong interface. The model needs the relevant endpoints and schemas for the task, not every operation in the API.

However, nothing prevents us from building a tool that can search the OpenAPI specification and return only the relevant information. Imagine, you're trying to integrate a marketplace platform and you want to know how to create a return.

Instead of using Ctrl+F or Cmd+F on the OpenAPI specification, why couldn't we use this algorithm to search it?

searchTerm = "return"
results = []

for each URL path and path item in spec.paths:
    for each HTTP method and operation in the path item:
        if the entry is not an HTTP operation:
            continue

        searchableText = combine:
            - the URL path
            - the operation ID
            - the summary
            - the description
            - the operation tags

        if searchableText does not contain searchTerm:
            continue

        endpoint = {
            method: the HTTP method in uppercase,
            path: the URL path,
            operationId: the operation ID,
            summary: the operation summary,
            parameters: [
                for each parameter:
                    keep its name, location, and required flag
            ],
            requestBody: keep only its content types and required flag,
            responses: [
                for each response:
                    keep its status code and description
            ]
        }

        add endpoint to results

return results

That way, we extract only the relevant information. But we can go even further.

For now, even with an algorithm like this, we still have to run the code manually and there is no way of knowing if our algorithm will handle all the cases. For example, to create a return, we could filter the results to only keep the endpoints that have a POST method. Maybe filtering using tags is better than trying to guess the right wording to use in the search term in summary and description. We could also filter the results to only keep the endpoints that have a 200 response code. There are so many different ways to filter the results, depending on the client, the data we need, and the task we want to perform, that writing a generic algorithm is far too complex.

We need to write a custom algorithm for each use case.

LLMs know code and algorithms ​

Or we can delegate this task to a LLM.

One practical strength of LLMs is generating code. They can turn a task-specific request into a small search program that developers can review and execute.

So, instead of writing a generic algorithm—or thousands of them to try to cover all the cases—we could just ask the LLM to write the algorithm for us. Then, we just have to execute it. That way, the LLM gets exactly the information it needs, and we do not have to worry about the context window problem.

The first time I heard about this idea was in a Cloudflare blog post named Code Mode and I was "what?". I honestly had to read it more than twice to understand what it was about. Let me give you an example.

Imagine you want to create a return on a marketplace platform. Instead of giving the entire OpenAPI specification to the model, you could write the following method:

async () => {
  const results = []

  for (const [path, methods] of Object.entries(spec.paths)) {
    const operation = methods.post

    if (operation?.tags?.some(tag => tag.toLowerCase() === 'returns')) {
      results.push({
        method: 'POST',
        path,
        summary: operation.summary,
        requestBody: operation.requestBody
      })
    }
  }

  return results
}

With the code above, we can quickly search the OpenAPI specification and return only the relevant information.

But who wants to write this code? Nobody.

Fortunately, LLMs can generate this code for us. Give them access to the OpenAPI specification and a couple of examples, and LLMs can search the entire specification to find only what is relevant.

Tool, description and language ​

In our MCP, the search tool would look like this:

server.registerTool(
  'search',
  {
    title: 'Search OpenAPI Specification',
    description: '...',
    inputSchema: z.object({
      code: z
        .string()
        .max(MAX_CODE_LENGTH)
        .describe('An async arrow function that searches the OpenAPI spec')
    }),
  },
  async ({ code }) => {
    try {
      const result = await evaluateSearchCode(code)
      return { content: [{ type: 'text', text: formatSearchResult(result) }] }
    }
    catch (error) {
      return {
        content: [{ type: 'text', text: `Search failed: ${errorMessage(error)}` }],
        isError: true
      }
    }
  }
)

From a high-level perspective, there is nothing more than what we have already seen in the previous article. The details are in the description and the evaluation function.

The description starts like any other tool description: what is the tool about, and what can the model do with it?

Search the MarketHub OpenAPI specification with JavaScript.

The specification stays inside the server. Your code must be an async arrow function and should return only the small result needed for the next decision. All local $refs are resolved before the code runs.

Then, things start to get interesting.

We can provide TypeScript interfaces to give the model a typed facade against which to write code. This compact contract improves the chance that the model writes a working program on its first attempt.

interface OperationInfo {
  operationId?: string
  summary?: string
  description?: string
  tags?: string[]
  parameters?: Array<{
    name?: string
    in?: string
    required?: boolean
    description?: string
    schema?: unknown
  }>
  requestBody?: unknown
  responses?: Record<string, unknown>
}

declare const spec: {
  paths: Record<string, {
    get?: OperationInfo
    post?: OperationInfo
    put?: OperationInfo
    patch?: OperationInfo
    delete?: OperationInfo
  }>
}

The full OpenAPI specification stays on the server. The generated program can inspect it through this facade, while the model receives only the compact interface and the final search result.

And finally, here are a couple of examples to show the model how to use the tool and what is expected as a result.

// Inspect one operation's request body
async () => spec.paths['/products']?.post?.requestBody

// Inspect endpoint parameters
async () => spec.paths['/products']?.get?.parameters

The model can now search the OpenAPI specification without receiving the complete document in its context.

But why ask the LLM to write JavaScript code instead of any other language?

We ask the model to write JavaScript because it is widely used, familiar to models, and straightforward to execute in the target runtime. The TypeScript interface above is documentation for the model; it provides static types for the facade, but it is not executed.

Discovery is not execution ​

Knowing which endpoints to call is a good start, but not enough. Based on all the data the LLM has, it could generate the code to call the endpoint. That would close the loop. From nothing, the LLM could both learn and interact with the API.

In Part 4, we will add a separate execution capability. It will let generated code call a narrow, host-owned api.request() helper and return only the part of the response the model needs next.