Continue.dev Cost Control: Codebase Context Expansion, Autocomplete Model Amplification, Agent Re-evaluation Loops, and Docs Crawl Storms

Continue.dev is an open-source AI code assistant for VS Code and JetBrains. It supports any LLM backend — Anthropic, OpenAI, Azure, Google, Ollama, and a growing list of provider adapters — and it ships four distinct AI-powered surfaces: @codebase semantic context retrieval, tab autocomplete, agent-mode multi-step task execution, and @docs documentation indexing. Each surface has different triggering semantics, different context assembly logic, and different token consumption behavior.

The per-request mental model that most developers use for cost estimation — "a chat message sends N tokens, gets back M tokens" — holds for the simplest Continue.dev usage (open a chat, type a question, send). It breaks down for the other three surfaces. Autocomplete fires on every keypress. The @codebase provider retrieves dozens of code chunks per query. The agent carries all prior tool output in each new call. The @docs provider crawls entire documentation sites on first use. None of these behaviors are visible in a single-request trace.

Continue.dev does not enforce a per-session, per-day, or per-project spend ceiling. It routes every LLM call through the model you configured in config.json — your API key, your provider, your rates. When retrieval over-fetches, when the autocomplete model is set to something expensive, when an agent gets stuck, or when docs indexing triggers unexpectedly, the bill accumulates without the tool reporting it.

Pattern 1: @codebase Context Window Expansion

Continue.dev's @codebase context provider builds a local embedding index over your repository using LanceDB. When a user sends a query prefixed with @codebase, Continue performs a vector similarity search and retrieves the top-K most similar code chunks, injects them verbatim into the chat context, and sends the combined context to your configured LLM. The default nRetrievedFiles is typically 10–15, and each retrieved chunk can be 1,500–3,000 tokens — the chunk size depends on your indexing configuration and the average size of files in your repository.

On a small codebase (10K lines, 50 files), the top-15 retrieved chunks might total 8,000 tokens. On a large codebase — a Next.js monorepo with API routes, server components, utility modules, and test files spanning 200K+ lines — the same top-15 retrieval returns 25,000–40,000 tokens of context injection before the user's actual question is even added. Across a team making 50 @codebase queries per day, the cost of context injection alone is the dominant LLM spend line.

The amplification compounds when @codebase is used inside an agent task rather than a single chat message. The agent may invoke @codebase lookup as part of a multi-step plan — each lookup injects a fresh 37,500 tokens into the already-growing context window. By step 4 of a 10-step agent task, the running context can exceed 100,000 tokens from retrieval injections alone.

The guard: CodebaseContextGuard

The guard intercepts the retrieved chunk list before it is assembled into the final context payload, counts the tokens in each chunk, and returns a pruned list that fits within a configurable retrieval token budget. It leaves the chat history, system prompt, and user message space intact, reserving headroom for the model's response:

from dataclasses import dataclass, field
from typing import Optional, List

@dataclass
class CodebaseContextGuard:
    max_retrieval_tokens: int = 12_000  # ceiling on @codebase chunk injection
    system_prompt_tokens: int = 1_000   # estimate: system prompt + prior history
    user_message_tokens: int = 300      # estimate: current user message
    headroom_tokens: int = 4_000        # reserve for model response
    model_context_limit: int = 200_000  # model window (claude-sonnet-4-6 = 200K)
    _retrieval_tokens_used: int = field(default=0, init=False)
    _chunks_dropped: int = field(default=0, init=False)
    _trip_reason: Optional[str] = field(default=None, init=False)

    @property
    def tripped(self) -> bool:
        return self._trip_reason is not None

    def _count_tokens(self, text: str) -> int:
        try:
            import tiktoken
            enc = tiktoken.get_encoding("cl100k_base")
            return len(enc.encode(text))
        except Exception:
            return len(text) // 4

    def evaluate_retrieval(
        self,
        retrieved_chunks: List[str],
        user_message_tokens: Optional[int] = None,
    ) -> List[str]:
        """
        Prune retrieved_chunks to fit within the retrieval token ceiling.
        Returns the approved subset (highest-ranked chunks that fit the budget).
        Sets trip_reason when any chunks are dropped.
        """
        if self.tripped:
            return []

        msg_tokens = user_message_tokens or self.user_message_tokens
        overhead = self.system_prompt_tokens + msg_tokens + self.headroom_tokens
        available_for_retrieval = min(
            self.max_retrieval_tokens,
            self.model_context_limit - overhead,
        )

        approved: List[str] = []
        running = 0

        for chunk in retrieved_chunks:
            chunk_tokens = self._count_tokens(chunk)
            if running + chunk_tokens > available_for_retrieval:
                self._chunks_dropped = len(retrieved_chunks) - len(approved)
                self._retrieval_tokens_used = running
                self._trip_reason = (
                    f"@codebase retrieval ceiling: accepted {len(approved)}/{len(retrieved_chunks)} "
                    f"chunks ({running} tokens) — {self._chunks_dropped} chunk(s) dropped to stay "
                    f"under {available_for_retrieval}-token retrieval budget"
                )
                break
            approved.append(chunk)
            running += chunk_tokens

        self._retrieval_tokens_used = running
        return approved

    def status(self) -> dict:
        return {
            "retrieval_tokens_used": self._retrieval_tokens_used,
            "chunks_dropped": self._chunks_dropped,
            "tripped": self.tripped,
            "trip_reason": self._trip_reason,
        }


def guarded_codebase_query(
    continue_client,
    user_message: str,
    guard: CodebaseContextGuard,
) -> dict:
    """
    Retrieves @codebase context, applies the guard, and sends the pruned
    context to the LLM. Callers receive the guard_status alongside the response
    so they can log or alert on trips.
    """
    raw_chunks = continue_client.retrieve_codebase_chunks(
        query=user_message,
        n_results=25,  # over-retrieve, then prune to budget
    )

    approved_chunks = guard.evaluate_retrieval(raw_chunks)

    response = continue_client.chat(
        message=user_message,
        context_chunks=approved_chunks,
    )

    return {
        "response": response,
        "guard_status": guard.status(),
        "chunks_used": len(approved_chunks),
        "chunks_available": len(raw_chunks),
    }

The guard over-retrieves intentionally (asking for 25 chunks) and then prunes to budget. This ensures the highest-ranked semantically similar chunks are always included first — the pruning drops the lowest-ranked chunks, not arbitrary ones. When a trip occurs, the caller receives a guard_status with the exact token count and drop reason, which can be logged for tuning the nRetrievedFiles default in config.json.

Pattern 2: Autocomplete Model Amplification

Continue.dev's tab autocomplete fires on every keypress that passes the debounce threshold (default ~300ms of inactivity). Each autocomplete request sends the file prefix (code above the cursor), the file suffix (code below the cursor), and model-specific instructions — typically 2,000–4,000 tokens of input context — to the configured tabAutocompleteModel. The response is a short completion (20–150 tokens). In an hour of active coding, a developer generates 200–600 autocomplete triggers.

The amplification failure mode is a configuration gap in Continue.dev's config.json: if tabAutocompleteModel is not set, Continue falls back to the default model in the models array — typically the same expensive model the developer uses for chat. A developer who configured claude-sonnet-4-6 or gpt-4o as their primary chat model, without explicitly setting a cheaper model for autocomplete, is billing every keypress at chat-model rates. Purpose-built autocomplete models (deepseek-coder-v2-lite, starcoder2-7b, Codestral, or a local Ollama model) run at 10–100× lower cost per request.

The misconfiguration is easy to make and hard to spot. Continue.dev's model selection UI shows only the chat model prominently. The tabAutocompleteModel field in config.json is a separate optional key — its absence is silent. Teams that onboard developers with a shared config template are particularly exposed if the template omits the autocomplete model: every developer in the team silently bills chat-model rates for every keystroke.

The guard: AutocompleteBudgetGuard

The guard tracks autocomplete token consumption on a rolling hourly and daily window, trips when thresholds are exceeded, and exposes a model-check function that detects when an expensive model is configured as the autocomplete provider. At trip, the caller can disable autocomplete for the current session rather than silently continuing to accumulate cost:

import time
from dataclasses import dataclass, field
from typing import Optional

EXPENSIVE_MODELS = frozenset({
    "claude-sonnet", "claude-opus", "gpt-4o", "gpt-4-turbo",
    "claude-3-5-sonnet", "claude-sonnet-4", "claude-sonnet-4-6",
    "gemini-1.5-pro", "gemini-2.0-flash",
})

@dataclass
class AutocompleteBudgetGuard:
    max_triggers_per_hour: int = 300
    max_input_tokens_per_day: int = 2_000_000
    _hourly: list = field(default_factory=list, init=False)      # trigger timestamps
    _daily: list = field(default_factory=list, init=False)        # (timestamp, tokens)
    _trip_reason: Optional[str] = field(default=None, init=False)

    @property
    def tripped(self) -> bool:
        return self._trip_reason is not None

    def _prune(self) -> None:
        now = time.time()
        self._hourly = [t for t in self._hourly if now - t < 3600]
        self._daily = [(t, n) for t, n in self._daily if now - t < 86400]

    def check_model(self, model_name: str) -> Optional[str]:
        """
        Returns a warning string if model_name matches an expensive model.
        Call this at Continue.dev config load time to detect misconfiguration.
        """
        lower = model_name.lower()
        for name in EXPENSIVE_MODELS:
            if name in lower:
                return (
                    f"tabAutocompleteModel is set to an expensive model ('{model_name}'). "
                    f"Consider setting a dedicated low-cost autocomplete model: "
                    f"codestral, deepseek-coder-v2-lite, starcoder2-7b, or a local Ollama model. "
                    f"Every keystroke triggers an API call at this model's input rate."
                )
        return None

    def record_trigger(self, input_tokens: int = 3_200) -> None:
        if self.tripped:
            return
        self._prune()
        now = time.time()
        self._hourly.append(now)
        self._daily.append((now, input_tokens))

        hourly_count = len(self._hourly)
        daily_tokens = sum(n for _, n in self._daily)

        if hourly_count >= self.max_triggers_per_hour:
            self._trip_reason = (
                f"autocomplete trigger ceiling: {hourly_count} triggers "
                f"in the last hour >= {self.max_triggers_per_hour} limit — "
                f"autocomplete paused for this session"
            )
        elif daily_tokens >= self.max_input_tokens_per_day:
            self._trip_reason = (
                f"daily autocomplete token budget: {daily_tokens:,} input tokens "
                f">= {self.max_input_tokens_per_day:,} daily limit"
            )

    def status(self) -> dict:
        self._prune()
        return {
            "hourly_triggers": len(self._hourly),
            "daily_input_tokens": sum(n for _, n in self._daily),
            "tripped": self.tripped,
            "trip_reason": self._trip_reason,
        }

The check_model function should be called once at IDE startup when Continue loads its config — it catches the misconfiguration before any tokens are spent. The rolling-window trigger and token counters catch the volumetric case: even with a correctly configured cheap model, high-frequency autocomplete in very large files can exceed daily budgets. At trip, the caller returns null completions (disabling autocomplete for the session) rather than continuing to accumulate cost.

Pattern 3: Agent Tool Failure Re-evaluation Loop

Continue.dev's agent mode (/agent) executes multi-step tasks using a ReAct-style loop: the model plans and reasons, calls a tool (run terminal command, edit a file, create a file, read a file), observes the tool output, then plans the next step. The full conversation history — including every prior reasoning step and tool output — is carried forward in each new LLM call. This is by design: the agent needs its history to make coherent decisions.

The cost amplification failure mode arises when a tool call fails and the agent enters a diagnosis-and-retry cycle. Each retry adds the failure output to the accumulated context. If the underlying problem is not solvable by the agent — a missing system dependency, a permission error, a fundamental misunderstanding of the codebase structure — the agent keeps generating plans and retrying with a context window that grows by 2,000–5,000 tokens per iteration. A 10-step stuck agent accumulates significantly more input tokens than a successful 10-step agent, because failure output is verbose and error messages often include stack traces.

The pattern is particularly expensive for tasks that touch system-level operations — package installation, database migration, CI/CD configuration — where the agent cannot resolve the failure without operator intervention but will attempt every variant it can reason about before giving up. Continue.dev's agent currently has no built-in iteration ceiling or context-growth ceiling; it runs until the model stops generating tool calls or the session is killed manually.

The guard: AgentIterationGuard

The guard tracks per-step context token growth, consecutive tool failures, and total iteration count. It trips on whichever ceiling is hit first, returning the partial work product and the trip reason so the developer can inspect what failed rather than watching the agent silently exhaust its context window:

from dataclasses import dataclass, field
from typing import Optional, List

@dataclass
class AgentIterationGuard:
    max_iterations: int = 10
    max_context_tokens: int = 60_000
    max_consecutive_failures: int = 3
    _iterations: int = field(default=0, init=False)
    _context_tokens: int = field(default=0, init=False)
    _consecutive_failures: int = field(default=0, init=False)
    _tool_log: List[dict] = field(default_factory=list, init=False)
    _trip_reason: Optional[str] = field(default=None, init=False)

    @property
    def tripped(self) -> bool:
        return self._trip_reason is not None

    def record_step(
        self,
        tool_name: str,
        tool_output: str,
        context_tokens: int,
        success: bool,
    ) -> None:
        if self.tripped:
            return

        self._iterations += 1
        self._context_tokens = context_tokens
        self._tool_log.append({
            "step": self._iterations,
            "tool": tool_name,
            "success": success,
            "output_preview": tool_output[:150],
        })

        if success:
            self._consecutive_failures = 0
        else:
            self._consecutive_failures += 1

        if self._iterations >= self.max_iterations:
            self._trip_reason = (
                f"agent iteration ceiling: {self._iterations} steps "
                f">= {self.max_iterations} limit "
                f"(context: {context_tokens:,} tokens)"
            )
        elif context_tokens >= self.max_context_tokens:
            self._trip_reason = (
                f"agent context ceiling: {context_tokens:,} tokens "
                f">= {self.max_context_tokens:,} after {self._iterations} steps"
            )
        elif self._consecutive_failures >= self.max_consecutive_failures:
            last = self._tool_log[-1]
            self._trip_reason = (
                f"consecutive tool failures: {self._consecutive_failures} failures "
                f"on '{last['tool']}' — last output: {last['output_preview']!r}"
            )

    def status(self) -> dict:
        return {
            "iterations": self._iterations,
            "context_tokens": self._context_tokens,
            "consecutive_failures": self._consecutive_failures,
            "tool_log": self._tool_log,
            "tripped": self.tripped,
            "trip_reason": self._trip_reason,
        }


def run_agent_with_guard(
    agent_session,
    initial_task: str,
    guard: AgentIterationGuard,
) -> dict:
    """
    Drives a Continue.dev agent session step-by-step, enforcing the iteration
    guard after each tool call. Returns partial work product on trip.
    """
    work_product = []

    while not guard.tripped:
        step = agent_session.next_step()

        if step is None or step.get("type") == "done":
            break

        if step["type"] == "tool_call":
            result = agent_session.execute_tool(step["tool"], step["args"])
            success = result.get("exit_code", 0) == 0

            guard.record_step(
                tool_name=step["tool"],
                tool_output=result.get("output", ""),
                context_tokens=agent_session.context_token_count(),
                success=success,
            )
            work_product.append({
                "tool": step["tool"],
                "success": success,
                "output_preview": result.get("output", "")[:200],
            })

    return {
        "work_product": work_product,
        "guard_status": guard.status(),
        "completed": not guard.tripped,
    }

The consecutive-failure detector is the most important ceiling for the re-evaluation pattern: when the same tool is failing three times in a row, the agent is stuck in a local retry loop and the problem requires developer attention, not more LLM reasoning. Surfacing this trip early — at 3 consecutive failures rather than 10 iterations — returns control to the developer before the context has grown to the point where reading the trip log becomes difficult.

Pattern 4: @docs Crawl Storm on Cloud Embeddings

Continue.dev's @docs context provider indexes external documentation sites so they can be searched semantically alongside your codebase. When a user adds a new docs entry to config.json (e.g., the React docs, Next.js docs, or Prisma docs), Continue.dev crawls the site on first query: it fetches each page, splits the content into chunks, and embeds them into a local LanceDB index.

By default, Continue.dev uses Transformers.js to run embeddings locally — no API cost for embedding calls. The amplification failure mode activates when teams configure a cloud embeddingsProvider in config.json (e.g., openai with text-embedding-3-large, or voyage, or cohere) to improve retrieval quality. In that configuration, every page crawled from a documentation site generates an embedding API call at cloud rates. A large documentation site (Next.js docs: ~380 pages; React docs: ~120 pages; Django docs: ~450 pages) generates hundreds of embedding calls in a single first-crawl event.

The crawl can also be triggered by calling continue.reindexDocs programmatically, which some teams do in workspace setup scripts to ensure the docs index is fresh. If that script runs in CI or on every container start, it silently re-crawls and re-embeds every configured docs source on every invocation. The LanceDB index is local to the developer's machine, so there is no shared index — each developer's environment incurs the full crawl cost independently.

The guard: DocsIndexGuard

The guard enforces per-source and total page count ceilings before a crawl begins, and tracks embedding tokens as pages are indexed. It intercepts the pre-crawl decision — blocking the crawl before any API calls are made — rather than trying to abort a crawl already in progress:

from dataclasses import dataclass, field
from typing import Optional

@dataclass
class DocsIndexGuard:
    max_pages_per_source: int = 150          # max pages crawled from one docs site
    max_total_pages: int = 600               # max across all configured sources
    max_embedding_tokens: int = 6_000_000   # embedding API token ceiling per reindex
    avg_tokens_per_page: int = 2_500        # rough estimate per docs page
    _pages_by_source: dict = field(default_factory=dict, init=False)
    _total_pages: int = field(default=0, init=False)
    _total_embedding_tokens: int = field(default=0, init=False)
    _trip_reason: Optional[str] = field(default=None, init=False)

    @property
    def tripped(self) -> bool:
        return self._trip_reason is not None

    def check_before_crawl(
        self,
        source_url: str,
        estimated_pages: int,
    ) -> dict:
        """
        Call before starting to crawl a docs source.
        Returns {"approved": True} if the crawl fits within budget, or
        {"approved": False, "reason": ...} to block it.
        """
        if self.tripped:
            return {"approved": False, "reason": self._trip_reason}

        projected_source_pages = self._pages_by_source.get(source_url, 0) + estimated_pages
        projected_total_pages = self._total_pages + estimated_pages
        projected_tokens = self._total_embedding_tokens + estimated_pages * self.avg_tokens_per_page

        if projected_source_pages > self.max_pages_per_source:
            reason = (
                f"docs crawl ceiling for '{source_url}': {projected_source_pages} pages projected "
                f"> {self.max_pages_per_source} per-source limit — "
                f"add a URL filter (startUrl + rootPatterns) to reduce crawl scope in config.json"
            )
            self._trip_reason = reason
            return {"approved": False, "reason": reason}

        if projected_total_pages > self.max_total_pages:
            reason = (
                f"total docs page ceiling: {projected_total_pages} pages projected "
                f"> {self.max_total_pages} across all sources"
            )
            self._trip_reason = reason
            return {"approved": False, "reason": reason}

        if projected_tokens > self.max_embedding_tokens:
            reason = (
                f"embedding token ceiling: ~{projected_tokens:,} tokens projected "
                f"> {self.max_embedding_tokens:,} limit — "
                f"reduce docs sources or switch to local Transformers.js embeddings"
            )
            self._trip_reason = reason
            return {"approved": False, "reason": reason}

        return {"approved": True}

    def record_page(self, source_url: str, page_tokens: int = 0) -> None:
        if self.tripped:
            return
        self._pages_by_source[source_url] = self._pages_by_source.get(source_url, 0) + 1
        self._total_pages += 1
        self._total_embedding_tokens += page_tokens or self.avg_tokens_per_page

    def status(self) -> dict:
        return {
            "total_pages": self._total_pages,
            "pages_by_source": dict(self._pages_by_source),
            "total_embedding_tokens": self._total_embedding_tokens,
            "tripped": self.tripped,
            "trip_reason": self._trip_reason,
        }

The guard's check_before_crawl takes an estimated_pages count, which Continue.dev can compute from the sitemap of the target docs site before starting the crawl. When the trip fires, the guard provides an actionable suggestion: add a startUrl and rootPatterns filter to the docs entry in config.json to scope the crawl to relevant subsections (e.g., only the API reference pages, not the full guide and tutorial content). This is preferable to raising the ceiling, because smaller scoped indexes also improve retrieval precision.

When to apply which guard

The four patterns operate independently and can co-occur within a single Continue.dev session. A developer using @codebase inside an agent task with @docs context and an unguarded autocomplete model can hit all four patterns simultaneously. In that configuration, the guards compose: run CodebaseContextGuard per @codebase retrieval, AutocompleteBudgetGuard per keystroke trigger (with a startup check_model call), AgentIterationGuard per agent session, and DocsIndexGuard per config load when cloud embeddings are configured.