Updated November 3, 2026. The structure is tightened and provider facts are re-checked. Statements about OpenAI, Anthropic, Ollama, Qdrant, LangGraph, and NIST reflect their official documentation as of November 3, 2026; the five-layer structure and the routing and verification principles are a design proposed in this article.
Decide before the model: five layers and routing criteria
When quants add AI, the first question is often “which model is best.” In a working system the longer-lived questions are where the data is allowed to sit, how predictable the job path is, and how you will check the result.
Handing one LLM news ingest, document search, code, backtest notes, and order judgement piles cost, permissions, and failure modes into one place. Splitting the functions into five layers lets you change and verify each one separately.
- Data: collect and normalize market data, documents, code, and logs.
- Retrieval: find the evidence the model is allowed to see.
- Model: pick cloud or local for that job.
- Workflow: control tool order, permissions, and state.
- Evidence: check results with schemas, eval sets, and human review.
Market data ─┐
Documents ─┼─> Normalize ─> Retrieve ─> Route model ─> Run tools ─> Verify
Code & logs ─┘ │ │ │
hybrid search cloud / local approval gateThe LLM does not own the database or the order path. Retrieval supplies evidence, the model drafts and interprets, and the workflow bounds what can run. The last stage splits checks a machine can make from checks a person must make.
| Criterion | When it is low | When it is high |
|---|---|---|
| Data sensitivity | Review a managed cloud API | Prefer local or private environments |
| Task uncertainty | Fixed workflow | Bounded agent loop |
| Repeat rate | Interactive cloud use | Optimize batch cost and throughput |
| Execution risk | Can run automatically | Approval step and split permissions |
The point is not to crown one model. It is to route each job onto the right path.
Model layer: split by data boundary and utilization
Cloud APIs
A cloud LLM lets you try long-document analysis, code review, structured output, and tool calls without running hardware. Swapping and scaling models is relatively easy, so it fits interactive research that changes direction often: comparing assumptions across papers, hunting likely bugs in a data pipeline, pulling candidate windows out of backtest logs, drafting counter-arguments for a research note.
“Not used for training” is not the same as “not stored.” As of November 3, 2026, OpenAI says API data is not used for training by default, while abuse-monitoring logs may be retained for up to 30 days. OpenAI data controls Anthropic offers zero data retention (ZDR) as a per-organization arrangement and states that features which must store data, such as the batch API, are not ZDR-eligible. Anthropic API and data retention Before you send strategy code or positions, re-read the current contract and per-feature retention terms. Cost and retention by call pattern are covered in the Claude API article.
Local runtimes
The main gain of a local LLM is not the weights. It is owning the execution boundary. Internal docs, unpublished factor definitions, and order logs can stay on a private network, and short classification or fixed-form summaries in bulk are also candidates.
Runtimes such as Ollama expose a local API, so wiring an existing app is easy. Local does not automatically mean safe or cheap. As of November 3, 2026, Ollama’s FAQ gives 127.0.0.1 as the default bind address and 4,096 tokens as the default context window. Exposing it on a network or raising the context brings authentication, firewall, and memory design with it. The same runtime also offers cloud-hosted models, so confirm in configuration that the local path really stays on the machine. Ollama FAQ
Model files and embedding stores also need access control. Compare quality and throughput against cloud candidates on the same eval set; the criteria are covered in the local LLM versus cloud API article.
GPU sourcing
Choosing a local model does not force a GPU purchase, and renting a cloud GPU does not erase operations. The deciding questions are utilization and how much downtime you can tolerate.
| Job type | Look at first |
|---|---|
| Occasional large-model experiments | Hourly GPU rental |
| Short parallel batches | Several rental instances plus a job queue |
| Steady daily inference | Compare owned hardware to total API cost |
| Long jobs that must not die | Stable instance plus checkpoints |
| Always-on sensitive data | Internal GPU or private deploy |
Do not compare on hourly price alone. Add model load time, data transfer, volume storage, reruns of failed jobs, and operator time.
RAG: a layer that supplies evidence, not a smarter model
Retrieval-augmented generation does not trust the model’s memory. It searches related documents at request time and puts them in context. For quant work that means papers, strategy specs, data dictionaries, and old experiment logs.
Document → Chunk → Embedding → Index
Question → Retrieve → Rerank → Prompt → Answer + SourceVector search is good at near-meaning and weak on exact tickers, function names, and error codes. Qdrant’s documentation describes hybrid queries that fuse the results of a sparse-vector query and a dense-vector query. Qdrant hybrid queries For finance and code, pair vector search with lexical search and metadata filters.
Score retrieval and generation separately. If you only grade the final sentence, the two error types mix.
- Does the right source document appear in the top hits?
- Do date, market, asset, and document-version filters hold?
- Does the answer match the retrieved evidence?
- When there is no source, does the model stop instead of guessing?
Agents: only for dynamic paths, and only inside bounds
Workflows and agents are different. LangGraph’s documentation describes a workflow as a predetermined code path and an agent as a model that defines its own process and tool use. LangGraph workflows and agents
Daily ingest with a fixed order is simpler and more reproducible as ordinary code.
fetch → validate → transform → store → reportA bounded agent can help when the next step changes with the input, as in “investigate this error, pick the logs you need, and list cause candidates.” Even then, do not hand the model every permission.
- Split read tools from write tools.
- Require a person for orders, deletes, and public posts.
- Cap loops and spend.
- Log every tool input and result.
- Prefer stopping over retrying when a failure condition is met.
The goal is not maximum autonomy. It is handling uncertain work inside an observable box.
Verification: scale the check to the harm of a wrong answer
A plausible paragraph is not verification. NIST’s Generative AI Profile (NIST AI 600-1) incorporates trustworthiness considerations into design, development, use, and evaluation. NIST AI 600-1 In a quant shop, set the strength of the check by the damage a wrong answer can do.
| Check | Examples |
|---|---|
| Automatic | JSON schema and required fields, numeric ranges and units, ticker and date formats, read-only SQL, existence of cited document IDs |
| Eval set | Retrieval recall@k, classification accuracy and per-class errors, grounding in retrieved text, variance on repeated input, regression after a model or prompt change |
| Human approval | Live orders and position changes, public reports, strategy-rule or risk-limit edits, sensitive data leaving the boundary |
Splitting format validation from evidence matching in code is covered in the structured output and evidence validation article. Treat LLM market commentary as a hypothesis to investigate, not a trading signal.
A small start, in order
You do not need a full agent platform on day one. This sequence keeps each layer’s effect visible.
- Pick one frequent job with a checkable answer.
- Build a rule-based baseline with no model.
- Measure a cloud-model quality baseline.
- If sensitivity or cost shows up, compare a local model on the same eval set.
- Add RAG only when you need current documents, and measure retrieval first.
- Add an agent loop only when the path cannot be written down in advance.
- Put an approval step in front of writes.
That order keeps you from debugging the model, the search, and the orchestration at the same time. The aim is not a system that judges for you but one that lets a researcher move faster and still trace which data and which evidence produced the conclusion.
Sources and scope
- OpenAI API data controls: no training by default and abuse-monitoring logs retained up to 30 days.
- Anthropic API and data retention: ZDR arrangement and per-feature retention.
- Ollama FAQ: default bind address, default context length, data handling for cloud models and local runs.
- Ollama API introduction: local API address.
- Qdrant hybrid queries: fusing sparse and dense vector query results.
- LangGraph workflows and agents: predetermined code paths versus dynamic agents.
- NIST AI 600-1: Generative AI Profile: trustworthiness from design through evaluation.
- The five-layer structure, routing table, GPU sourcing table, and verification tiers are a design proposed here. No tool or model performance was measured and no usage survey was conducted.
