Lead Research Source — Options, Costs and Recommendation
Status: DECIDED — Exa, chosen by Empire Teas on 2026-11-10 (Kavindu Anthony, in the progress meeting;
.forge/discovery/meeting-notes/2026-11-10-kavindu-crm-progress-update-demo.md§8; SP-ARCH-004 closed, Rev 11). The person search may go to Exa, behind a switch on the CRM’s Settings page that is on by default (Rev 11), built by F11-i (Rev 12), and on in every environment with no precondition (Rev 13). The status as it stood before: PROPOSAL — the provider is not chosen. The seam is settled (PRD Rev 30); what sits behind it is pending the quality bake-off in §7. Nothing here is a commitment. (Update 2026-10-13: the bake-off has run — §11. AgentCore came last on quality; the choice is now Dinil’s, with Tharinda.) (Update 2026-10-15: the team recommends Exa; Empire Teas heard the comparison on 14 October and asked for it by email, so the choice waits on Kavindu Anthony’s answer. The analysis was reviewed on 15 October — §12.) (Update 2026-10-20: Kavindu asked the team to use a free or trial version for now, so F11-c is built on Exa’s free credit, in an account Altrium holds. The choice stays provisional until Empire answers on Exa itself — §11, Rev 8. One provider only, with no Tavily fallback — Rev 9. Prices, free-credit capacity, rate limits, terms and accounts in one place — §13.) Scope: which external source grounds the AI research pass that precedes lead ranking — F11 (every lead, PRD Rev 30) and F36 (trade-show capture). Closes thesow-cost-annexe.md§6 row that has read “Enrichment data source — still to be selected” since 2026-08-21. All prices verified 2026-09-16 against the sources in §9; the four tested providers’ prices re-verified 2026-10-13 (§3, §11). Search-API pricing moves; re-verify at build time.
1. What is actually needed
One search per lead, synthesised by the model into a grounded snapshot. Not a firmographics purchase.
The shape is already proven in the capture-app POC (../forge-et-pwa-poc, read-only reference):
apps/api/app/adapters/search/— aSearchProviderprotocol withmockandtavilyimplementations.- Its web-enrichment client — one
search()call per lead (max_results=5), results passed to the model, which returnsindustry, size, location, role, recent_signals, certifications, origin, import_export, summaryplus asourcesmap pinning each populated field to the URL that backs it. - Its model scorer — scores from that snapshot, under “Score only from the research provided; do not invent facts.”
That last line is why research is load-bearing rather than decorative: strip the snapshot and the scorer is ranking on a job title. It is also why per-field provenance is a requirement and not a nicety — see §6.
One search per lead is the unit of cost for everything below.
2. The constraint that shapes the choice
AD #19 puts AI on Claude via Amazon Bedrock exclusively — “one vendor relationship and one bill… the client holds no separate AI provider accounts,” with in-account processing as the data-governance answer. Three current facts about giving Claude-on-Bedrock a web search:
- Anthropic’s own
web_searchserver tool is not supported on Bedrock. On the direct Anthropic API it is $10/1,000 searches + tokens, but reaching it means going direct, which AD #19 rules out. - “Web Search on Amazon Bedrock” (GA 4 Aug 2026) does not help us. It is OpenAI GPT-5.x only, via the
bedrock-mantleResponses API. Not available to Claude models. - AgentCore Web Search (GA 17 Jun 2026) does. Exposed over MCP through AgentCore Gateway, model-agnostic
(AWS’s own walkthrough drives it with Claude), $7 per 1,000 queries, available in
us-east-1,eu-west-1andap-northeast-1— the first of which is our region (AD-O-9) — and marketed as “zero data egress from customer’s secured AWS environment.”
Item 3 post-dates AD #19 by two months. It means Claude-on-Bedrock can have web search on the AWS bill, in account, with no new vendor — satisfying AD #19’s letter and its rationale. No AD change is needed; this is an implementation choice inside it.
How the AgentCore call is actually made
Recorded so the implementer does not have to rediscover it. The Gateway is the endpoint — the tool has no public URL of its own, which is why one has to be provisioned before anything can be called.
POST https://<gateway-id>.gateway.bedrock-agentcore.<region>.amazonaws.com/mcp
Authorization: AWS4-HMAC-SHA256 … (SigV4, service "bedrock-agentcore", the task's IAM role)
{"jsonrpc":"2.0","id":1,"method":"tools/call",
"params":{"name":"WebSearch","arguments":{"query":"…","maxResults":5}}}
query is required and capped at 200 characters; maxResults is 1–25, default 10. Optional per-request filters
cover domain include/exclude lists and published-date bounds. The reply is an MCP envelope whose
content[0].text holds a JSON string of {title, url, text, publishedDate} results — one extra unwrap compared
with Tavily, then it maps onto the POC’s existing SearchHit(title, url, content) shape.
(Update 2026-10-13 — run against the dev account’s gateway, et-dev-gateway, from Python with botocore’s
SigV4 signer; §11.) Two corrections to the request above. The tool name is web-search___WebSearch: the
gateway prefixes each tool with its target’s name (the dev target is web-search), so a bare WebSearch is not
what tools/list returns. The request carries Accept: application/json, text/event-stream, as MCP’s
streamable HTTP transport expects. The reply shape is as described. The caller needs
bedrock-agentcore:InvokeGateway on the gateway’s ARN; AmazonBedrockFullAccess does not include it, because
AgentCore is a separate service prefix. No Java has been run yet (§7).
Two capabilities Tavily does not advertise, both relevant here: a knowledge graph for entity facts (who holds a role, when a company was founded — directly useful for lead research) and semantic snippet extraction, which returns the passages bearing on the query rather than raw page text.
A licence term that constrains the UI
AWS’s acceptable-use terms for the tool require that we retain and display the source citations and links with any output shown to end users, and forbid extracting or storing search content in bulk or building a competing index.
This is a requirement on the screen, not just the database. The PRD already has every field carrying its source
URL, and the design system already gives each AI payload a source alongside its confidence — so we are close, but
displaying the link becomes mandatory rather than a design preference. It lands on F11-f, which hosts the AI
panels, and on F11-g, whose “why this priority” text is built from search results. Our shape — a short
synthesised snapshot with citations — is well inside the bulk-storage prohibition; caching raw search results
wholesale for reuse would not be. This term binds only if AgentCore is chosen; Tavily’s terms have not been read.
3. The options, priced
At one search per lead. “Per 150-lead show” is the F36 unit; F11 is ~40/month plus the 200-lead backlog.
(Update 2026-10-13: the four tested providers’ rows — AgentCore, Tavily, Serper, Exa — are re-verified against each vendor’s own pricing page that day; the other rows are as of 2026-09-16. Tavily’s row is corrected: it has pay-as-you-go with no monthly minimum, and the $30 is an optional plan. The yearly comparison is §11.)
| Option | Unit price | 150-lead show | Monthly floor | New vendor? | Notes |
|---|---|---|---|---|---|
| Bedrock AgentCore Web Search | $7 / 1,000 | ~$1.05 | none | No — AWS | us-east-1 (also eu-west-1, ap-northeast-1); MCP via Gateway; in-account. Gateway: $0.005 / 1,000 invocations, $0.025 / 1,000 search-API calls, $0.02 / 100 tools indexed a month — negligible. No free tier |
| Tavily (the POC’s provider) | $0.008/credit (basic = 1, advanced = 2) | ~$1.20 | none — pay-as-you-go; 1,000 credits free every month | Yes | Optional plans: Project $30/mo for 4,000 credits, up to Growth $500/mo for 100,000; its extract endpoint also serves F36 exhibitor lists |
| Brave Search API | $5 / 1,000 | ~$0.75 | ~$5/mo credits (free tier withdrawn early 2026) | Yes | “Answers” tier $4/1,000 + $5/M tokens |
| Serper | $1 / 1,000 in the smallest pack (→ $0.30 at volume) | ~$0.15 | $50 minimum pack; credits expire after 6 months | Yes | Cheapest raw; Google SERP proxy. 2,500 free queries to try. Packs: $50 / 50k, $375 / 500k, $1,250 / 2.5M, $3,750 / 12.5M |
| Exa | $7 / 1,000 (auto/fast, up to 10 results, page text included since 3 Mar 2026) |
~$1.05 | none; $10 of credit free every month | Yes | Semantic, built for RAG. instant $4 / 1,000 (not tested); $1 / 1,000 per result above 10 |
| Perplexity Sonar | tokens + $5–14 / 1,000 | ~$1–2 | — | Yes | Bundles synthesis we already do on Bedrock — pays twice |
Anthropic web_search |
$10 / 1,000 + tokens | ~$1.50 | — | Yes — breaks AD #19 | Not on Bedrock |
| Google Custom Search JSON | $5 / 1,000 | ~$0.75 | — | Yes | Ruled out: closed to new customers, shuts down 1 Jan 2027 |
| People Data Labs | ~$27.70 per company | ~$4,155 | $98/mo | Yes | This is F29. Dropped (PRD Rev 66). |
| Apollo.io | ~$10.88 per company | ~$1,632 | $49/user/mo | Yes | Same. |
4. What the numbers say
Usage cost is not the decision. Six shows a year plus research on every F11 lead is roughly 900 + 680 ≈ 1,580 searches/year ≈ $11/year at AgentCore rates. Against ET-002’s USD 650/month this is indistinguishable from zero. Nobody should argue this choice on unit price.
The monthly floor is the decision. Tavily’s $30/month minimum is $360/year to consume ~$11 of search — a 30× overpay whose only purchase is a second vendor relationship.
(Correction 2026-10-13: the paragraph above no longer holds. Tavily offers pay-as-you-go at $0.008 a search with no monthly minimum, and 1,000 free searches a month; the $30 is an optional plan. At ~1,580 searches a year, AgentCore, Exa and Tavily all cost about $11–13 a year at their paid rates, and Exa’s and Tavily’s monthly free allowances cover the expected volume. The provider with a real floor is Serper — a $50 pack whose credits expire after six months, about $100 a year at our volume. Price does not separate AgentCore, Exa and Tavily; quality and the vendor relationship do. §11.)
The licensed-enrichment tier is not a candidate. PDL and Apollo are three orders of magnitude more expensive per record and are F29, deferred from V1 outright (2026-08-20) and dropped on 2026-10-11 (PRD Rev 66). PRD Rev 30 keeps that boundary explicit in §Integration Points: F29 is buying firmographic records; this is the AI reading public sources.
5. The commercial argument — from ET-002, not from price
Superseded basis (Rev 5, 2026-09-29). SOW ET-002 is not the arrangement in force: Empire Teas holds the AWS account and AWS bills Empire Teas directly (
.forge/client/aws-sow.md, marked superseded). The argument below is kept as it was written, but its premise is gone. On the new basis, AgentCore Web Search is AWS usage on Empire Teas’ own AWS bill — no new vendor, no key, IAM-scoped and CloudTrail-audited, as before. Tavily would need its own vendor account, with a card and a key in Secrets Manager. Whose card that would be, and who holds that account, is not decided: Dinil answers it after the SP-ARCH-004 stage-2 comparison of AgentCore and Tavily, together with the provider choice. The “Who pays” paragraph below no longer applies.
This is the heaviest consideration and it is contractual rather than financial.
SOW ET-002 §: “The monthly fee is all-inclusive… covers the managed infrastructure services in this SOW together with the associated AWS and approved third-party platform usage.” And: “Material additions or changes outside the approved architecture… including additional environments, regions or third-party services—require prior written approval through Change Control.”
- A search API (Tavily / Brave / Serper) is a third-party service never named in the approved baseline. Taking one is at minimum a baseline amendment and arguably a Change Control item — plus a card, a key in Secrets Manager, a vendor ToS, a DPA, and one more thing that can fail.
- AgentCore Web Search is AWS usage. It lands inside the fee with no Change Control question, no new vendor, no key to rotate, IAM-scoped and CloudTrail-audited.
Who pays, stated plainly: under ET-002 Empire pays a fixed USD 650/month and the annexe marks this line “Altrium-side.” Whatever is chosen comes out of Altrium’s margin, not a client invoice. The question is not “don’t spend the client’s money” — it is “don’t add a vendor and a $360/year floor for $11 of search.”
6. Compliance — and a date
The POC’s ADR-0007 (“Licensed enrichment APIs; no LinkedIn scraping”) already decided the posture: licensed
APIs plus public company web, no LinkedIn scraping — citing Proxycurl’s shutdown in July 2025 after a LinkedIn
suit — with data minimisation, per-field provenance, and a deletion path, built to GDPR grade. That posture
ports; the ADR itself is input material, not a decision of this project (.claude/rules/source-trust.md).
Sri Lanka’s PDPA Parts I and III take effect 1 January 2027 (Gazette Extraordinary No. 2498/16, 22 July 2026). V1 sign-off targets 15 December 2026, so Empire’s controller obligations commence about two weeks after go-live, (Rev 5, 2026-09-29: sign-off now targets Fri 29 January 2027, so the obligations commence about four weeks before go-live, while the parallel run and final migration are under way — the three requirements below matter from the Phase 1 UAT data onward) over a system processing researched third-party personal data. Not a blocker — but it makes ADR-0007’s three requirements V1 requirements rather than good intentions, and the POC already implements per-field provenance, so keep it when porting.
This also favours search-and-synthesise over buying a dataset: we process public web content about a company that contacted us or visited our stand, rather than ingesting a purchased personal-data record whose consent chain we would have to stand behind.
7. Recommendation — and the spike that must precede it
Update 2026-10-13 — stage 2 has run (§11), and AgentCore came last on quality. Judged by Claude, it found the right company for 5 of 25 Empire customers, against Exa 13, Serper 11 and Tavily 8. The recommendation below is kept as written, but its quality premise (“if AgentCore wins or ties on quality”) did not hold, and its cost argument against Tavily rested on a $30 floor that no longer exists (§4). The provider is not chosen: Dinil decides, with Tharinda, between Exa (best results), Tavily (the pre-decided fallback) and AgentCore (inside AWS, weakest results) — §11 sets out the three. The standalone Java check below is owed only if AgentCore stays in the running.
Update 2026-10-20 — no fallback provider. F11-c is built on Exa alone; Tavily is not kept as a fallback (§11, Rev 9).
Primary: Bedrock AgentCore Web Search, behind the ported SearchProvider seam. In-account, no new vendor, no
Change Control exposure, no monthly floor, us-east-1, ~$11/year.
Fallback, pre-decided: Tavily on the same seam. The POC adapter exists and works, so falling back costs about a day.
The spike that settles this is SP-ARCH-004 in tracker.yaml.
The integration question is answered on the documentation, and still owes a practical check
(Rev 3). On paper a plain Spring Boot backend can call the tool directly: MCP over streamable HTTP is JSON-RPC in
a POST body, so it is one SigV4-signed request built with RestClient and the AWS SDK’s signer — no MCP library,
no Spring AI, no agent framework. AWS states the tool is usable by “any MCP-compatible client”, and a first-party
Spring AI SDK for AgentCore exists as well, though we do not need it. What remains is one-time provisioning: a
Gateway with a connectorId: "web-search" target and a service role, which is Terraform in the FS-1 cloud half.
None of that has been run. It is a reading of vendor documentation, not a result, and this document has already been wrong once by trusting a measurement nobody validated (§10). It matters more than usual here because AgentCore is the recommendation and the case for it — no new vendor, no Change Control, data inside our own account — assumes calling it is straightforward. Owed when the AWS account lands: about thirty lines of standalone Java that sign one request and read the response, run once and discarded, before anyone builds the real adapter. An hour, and it runs alongside the stage-2 search comparison because both wait on the same account.
The language of the stage-1 comparison script is deliberately not Java: it measures what the search services return, which is identical whoever asks. Java only matters for the question above, which is a different one.
The quality question is open, and it is the one that decides the choice. Research grounds F11’s priority score (Rev 30), so coverage matters more than price, and no provider here has been tested. Amazon’s index is new and there is no evidence of how it performs on the companies we actually care about — a mid-sized Polish importer, a German private-label house, a Gulf distributor.
Test method (revised, Rev 1). Three corrections to what this section first said:
- The population is CRM leads first, not exhibitors. F11-c is Phase 1; F36-c was Phase 2 when this was written, and is Phase 1 since 2026-11-11 (PRD Rev 91), still built after F11-c. An exhibitor list over-represents large retailers because trade shows attract them. The right population is the companies that write in — and the manual-add path (F11-b) is harder still, because a rep may type only a name and a country.
sample-exhibitors.csvcannot answer the question. Opened after this doc was first written: its twenty rows are Aldi, Lidl, Costco, Trader Joe’s, Tesco and similar. Every provider would score near 100% and we would learn nothing. Do not use it.- Test three input tiers, not one, because all three occur in production: name only (manual add), name + country (the common manual add), and name + website domain (inbound email and card scans, where the domain comes free from the sender address). A provider that fails tier 1 and passes tier 3 is not unusable — it means always pass the domain, and expect weaker research on manual leads.
Source of names, and its bias. Best is a slice of Empire’s own lead or client list — a small, specific ask, and not the same thing as the full Pipedrive audit. Whoever builds the list must keep the companies they could not find, with the domain blank: those rows are the hard cases and dropping them silently rigs the result. Any list assembled by searching is biased toward findable companies, and the write-up says so rather than presenting the score as clean. Client names are confidential — they stay out of the repo, and only aggregate scores come back here.
Score per company, per provider, per tier: fields populated out of the nine in the snapshot; whether each populated field is backed by a real source URL; whether it is the right company (a confident answer about a same-named firm elsewhere is worse than an empty one); and whether tea-relevant signals appear at all (certifications, import/export posture, origin).
If AgentCore wins or ties on quality, take it for the §5 reasons. If it loses materially on the long tail, Tavily at $360/year is cheap insurance for the feature’s core signal — an easy argument internally, where “$30/month for a floor” was not.
Ruled out now: Google CSE (shutting down), PDL/Apollo (F29, dropped, ~1000× the cost), Anthropic direct
web_search (breaks AD #19), Perplexity Sonar (pays twice for synthesis).
8. What this does not settle
- The provider. Pending §7.
- Page rendering has no AgentCore equivalent — this is now a known gap, not an open question (Rev 1). The POC
uses Tavily’s
extractwithadvanceddepth to render client-side exhibitor directories. The AgentCore web-search connector has no fetch operation at all; it returns semantically selected snippets instead. If we pick AgentCore and something later needs a rendered page, that capability has to come from somewhere else. It does not affect F11-c, which only needs search. It is F36-c’s to confirm — and note that no Gate 3 feature appears to cover exhibitor-list import, so the capability may have no V1 consumer at all. Confirm rather than assume. (Update 2026-11-12, PRD Rev 96: exhibitor planning is now in Phase 1, in F36-a and F36-c, but the list is uploaded as a CSV or Excel file and read without AI, not fetched from a show’s web directory, so the rendered-page fetch still has no V1 consumer; F36-c’s spec confirms it for the exhibitor research.) - Two questions F11-c’s spec must answer, found while working through the lead entry paths and recorded here
because no spec owns them yet. (a) What triggers the research job? A lead can arrive four ways — inbound email
(F12/F20), Pipedrive migration (F4-b), manual add (F11-b) and trade-show capture (F36-a). PRD Rev 30 says
every lead is researched, so the trigger is lead creation on all four paths, but the spec should name it rather
than leave it to the implementer. (b) Does editing re-trigger it? If a rep corrects a mistyped company name,
the stored research is now about the wrong company, and nobody has decided whether an edit re-runs it.
Sequencing note: F11-b ships before F11-c, so manually added leads will exist before research does, and the queue
needs a sensible state for a lead that is not yet researched. (Answered 2026-10-21 by the F11-c spec: (a) research
is queued when the intake creates a lead, on every path, except Pipedrive leads, which the paced hourly sweep
takes (decisions 3 and 9); (b) a change to the company name, website, country or contact name re-runs it
(FR-S2); leads created before F11-c reached
main(2026-10-22) are picked up by the same sweep (decision 10).) - Empire-side research sources. Trade-data sources (import/export records) were not evaluated; they are a post-V1 question (close to F29, dropped in PRD Rev 66).
9. Sources
Verified 2026-09-16. Prices and regional availability move — re-verify before locking.
- Bedrock Web Search, and its OpenAI-models-only limitation: the Bedrock User Guide’s Web Search page (https://docs.aws.amazon.com/bedrock/latest/userguide/web-search.html)
- AgentCore Web Search — GA 17 Jun 2026, $7/1,000,
us-east-1, zero-egress claim: the AWS News Blog announcement and the AWS Machine Learning Blog introduction - Anthropic web search on the Anthropic API ($10/1,000 + tokens): Anthropic’s web search on the Anthropic API announcement (https://claude.com/blog/web-search-api)
- Tavily, Brave, Exa, Serper, SerpAPI, Perplexity Sonar pricing: each vendor’s own pricing page, cross-checked against independent 2026 search-API pricing comparisons
- Google Custom Search JSON API shutdown (1 Jan 2027; already closed to new customers)
- People Data Labs / Apollo per-company enrichment rates: published self-serve benchmark comparisons
- Sri Lanka PDPA commencement — Gazette Extraordinary No. 2498/16, 22 Jul 2026: the Data Protection Authority’s dates-of-operation page (https://www.dpa.gov.lk/est.php)
- POC implementation, in the read-only sibling repo: the Tavily search adapter, the web-enrichment client and
the model scorer under that repo’s
apps/apitree, plus its ADR-0007 on licensed enrichment and no LinkedIn scraping. See the Reference Map row for the capture-app POC - Commercial basis: the AWS infrastructure SOW ET-002 (all-inclusive fee + the Change Control clause on
additional third-party services) and the running-cost annexe §6 — both in
.forge/client/
Re-verified 2026-10-13, for the four tested providers (§3, §11), each from the vendor’s own page:
- AgentCore Web Search and Gateway prices: https://aws.amazon.com/bedrock/agentcore/pricing/
- Exa: https://exa.ai/pricing, and its 3 Mar 2026 pricing changelog (page text included in the search price): https://exa.ai/docs/changelog/pricing-update
- Tavily: https://www.tavily.com/pricing and its credit costs, https://docs.tavily.com/documentation/api-credits
- Serper: https://serper.dev/ (packs, the six-month credit expiry, the free trial queries)
- Claude Haiku 4.5, for the cost of the stage-2 judging: Anthropic’s pricing page (https://platform.claude.com/docs/en/about-claude/pricing) — $1 / $5 per million input / output tokens, with the 10% premium it states for regional and geographic endpoints; not read from the Bedrock price page itself
Re-verified 2026-10-20, for Exa’s free credit (§11):
- Exa’s prices and free credit: https://exa.ai/pricing
- Exa’s terms of service (free services, the API licence, the licence over search text, prohibited uses): https://exa.ai/terms
- Exa’s rate limits and the
429/402responses (read 2026-10-21): https://exa.ai/docs/reference/rate-limits - Tavily’s free tier (1,000 credits a month, no card), unchanged: https://www.tavily.com/pricing
10. Stage-1 run, 2026-09-18 — one firm result and a method that did not work
Tavily, Serper and Exa were run against 25 real Empire customers taken from the weekly pricing workbook, at four query shapes, about 250 searches, no errors. AgentCore was not included: it needs an AWS account that does not exist yet.
The artifacts are not in this repository. They contain Empire customer names, so they live on
the developer’s machine at Desktop/empire-search-test/ with a README.md explaining how to
finish the test. Every raw response is saved, so adding AgentCore later runs one provider and
compares against the same 25 companies rather than repeating this work.
The result that holds: do not add context to the search query
Adding words to the query makes the search worse, and the effect is large. Two measurements show it, neither depending on the scoring problem described below:
- The results are different pages. Comparing “company name” against “company name + country + tea importer”, 14 of 19 companies had no result site in common between the two.
- The replacements are directories, not companies. One tea directory came back 31 times
across different companies; trade-data sites (
trademo.com,exportgenius.in,oec.world) and other directories dominated the rest.
Adding “tea importer” makes the search return the tea industry instead of the business. The danger is that it looks like an improvement: every naive relevance check scores those results highly because they are full of tea content. They are about the wrong companies.
Consequence for F11-c: build the company query from the company name plus the website domain when known, and nothing else. (The person is researched too — PRD Rev 56, 2026-10-08 — by a lookup F11-c designs separately; this finding is about the company query.) No country, no industry words. This is the opposite of what a developer would reasonably assume, and the prototype’s own query builder would have been extended the wrong way.
The result that does not hold: which provider is best
Stage 1 scored “did it find the company?” by checking whether the company’s name appeared in a returned website address. That measure is invalid, in both directions, and two cases prove it:
- A false success. A four-letter customer name matched a US car dealership’s site and was counted as found.
- A false failure. One customer was counted as not found although all three providers had returned its real website. It is a publicly listed company — a stock-exchange listing, a ninety-year history, and the largest business-to-business tea supplier in its market. Its web address is an abbreviation that shares almost no letters with its registered name, so the check discarded the correct answer.
So the per-provider ranking, and the apparent strength of the name-plus-domain query, are both withdrawn. Exa led at every tier, which is a hint worth carrying into stage 2, not a result.
What a correct test needs, and why it waits
The measure has to be Claude reading the results and judging whether they describe the right company — which is also how the finished feature works. The search tool only has to surface the page; the model decides what it is. Scoring the search tool with a text match measures something the product will never do, and understates every provider.
That needs a way to call Claude, which means Bedrock — and Bedrock authenticates with ordinary AWS credentials, not a separate API key. So stage 2 has one blocker, not two: the AWS account. It supplies both halves — Claude for the judging, and the Gateway for AgentCore. The only extra step is enabling access to the pinned models in the account, which belongs to standing the account up rather than to this spike. When it lands, the saved responses are re-scored at no additional search cost.
A design point for F11-c that came out of this: research quality rests on the model’s reading at least as much as on the provider. One search, five results, no second look is what the prototype does; a lead whose first search returns nothing useful gets no second attempt. Whether the job should re-query on a poor first result is a question the spec should answer.
11. Stage-2 run, 2026-10-13 — AgentCore added, all four judged by Claude
AgentCore was run on the same 25 Empire customers as stage 1, and Claude Haiku 4.5 on Bedrock then
judged every saved response from all four providers. This is the comparison §10 said a correct test
needs. As in stage 1, the company list, the per-company results and the judgements stay on the
developer’s machine at Desktop/empire-search-test/ (bakeoff.py runs the searches, judge.py
the judging); only totals are recorded here.
Access, as it was set up
- Login: the
svc-harnessIAM user in the dev account (570417736903), used through an access key on the developer’s laptop (environments-and-identity.md§4 item 7). - Gateway:
et-dev-gatewayinus-east-1, built by Shyamin Ayesh inforge-et-iaac(base/agentcore/, used byoverlays/dev/). Checked read-only before use: status READY, protocol MCP, inbound authAWS_IAM(SigV4), one targetweb-searchon the managed connectorweb-searchv1.2.0, and a service role that only AgentCore gateways in this account can assume, allowingInvokeWebSearch. - Permission:
svc-harnesscould not call the gateway —AmazonBedrockFullAccesscoversbedrock:*, notbedrock-agentcore:*. With Shyamin Ayesh’s agreement, an inline policyspike-invoke-gatewaywas added tosvc-harness, allowingbedrock-agentcore:InvokeGatewayon this one gateway only. It comes off when the spike closes (owed.md). - The call: one signed
tools/listand one test search returned results, which proved the path end to end. The request details are in §2.
Method
- The same 25 companies as stage 1, unchanged, so the stage-1 responses of Tavily, Serper and Exa are compared with AgentCore’s directly. Those three were not searched again.
- The same four query shapes (§10): name; name + country; name + website domain; name + country
- “tea importer”. A shape is skipped where the company has no country or no domain, which gives 82 searches per provider. Each search asked for 5 results. AgentCore’s 82 searches returned no errors.
- The judge. For each of the 328 saved responses, Claude Haiku 4.5
(
us.anthropic.claude-haiku-4-5-20251001-v1:0, temperature 0) was given the company’s name, country and recorded website, and the five results, and asked: (1) is at least one result about this exact company — not a similar name, not a firm in another country (yes / no / unclear); and (2) which of the nine snapshot fields (§1) a result about the right company states, each tied to one result, with no guessing. The judge was not told which provider produced the results. - One correction before the full run. A first trial presented the recorded website as fact and assumed every customer is a tea company. The prompt was changed to say the website was found by a separate search and may belong to another firm, and that a customer buys tea but may be a supermarket, a hotel or catering group or a general food trader. The trial’s judgements were discarded and everything was judged with the corrected prompt.
- Hand checks. The judge rejected the customer whose recorded website is a US car dealer’s (the stage-1 false success) and found the real customer through import-trade records. In four of the 17 searches where AgentCore was judged “no” and Exa or Serper “yes”, the AgentCore results were checked by hand and were indeed unrelated: a clothing retailer’s pages, Wikipedia articles about the customer’s country, product-marketplace listings — one of them with the company’s own website in the query.
Results
| Provider | Companies found, of 25 (any query shape) | Searches that found the right company, of 82 | Found by this provider only |
|---|---|---|---|
| Exa | 13 | 33 | 3 |
| Serper (Google) | 11 | 28 | 0 |
| Tavily | 8 | 22 | 0 |
| AgentCore | 5 | 14 | 0 |
“Companies found” counts a customer when at least one search found the right company. “Searches” counts each search on its own; since each lead gets one search in production, it is the day-to-day success rate. 14 of the 25 customers were found by at least one provider; 11 were found by none.
By query shape — searches that found the right company, and the average number of the nine fields a search could fill (a search about the wrong company scores 0):
| Query shape | Exa | Serper | Tavily | AgentCore |
|---|---|---|---|---|
| Name (of 25) | 7 · 1.8 | 6 · 1.2 | 5 · 1.0 | 4 · 0.8 |
| Name + country (of 20) | 9 · 2.3 | 8 · 1.6 | 6 · 1.7 | 3 · 0.8 |
| Name + domain (of 17) | 8 · 2.6 | 9 · 2.1 | 7 · 2.1 | 3 · 0.9 |
| Name + country + “tea importer” (of 20) | 9 · 3.2 | 5 · 1.4 | 4 · 1.2 | 4 · 1.3 |
“Unclear” verdicts (between 1 and 5 per cell) count as not found. The stage-1 automated measure (the company’s domain in a result address), run again with AgentCore added, ranks the providers the same way, but §10 explains why it is not used as evidence.
What the results mean
- AgentCore finds the right company least often — about half as often as Serper, well under half as often as Exa, and clearly behind Tavily (14 searches against 22). Its index covers the small importers and distributors Empire deals with poorly. §5’s reasons for it (no new vendor, one bill, data inside AWS) still hold; the search quality does not support them.
- Exa is the strongest, by companies found, by searches and by fields filled. Part of its lead on fields comes from returning up to 2,000 characters of page text per result, where Tavily’s basic search and Serper return short snippets. That does not explain AgentCore’s result: AgentCore also returns long text.
- Tavily, the pre-decided fallback, is third, behind Serper and clearly behind Exa.
- Even the best provider misses about half the customers. F11-c has to treat “nothing found” as a normal outcome with its own state, not as a weak lead.
- The stage-1 query rule mostly holds. Adding “tea importer” to name + country lowered the right-company count for Tavily and Serper, left Exa’s the same and raised AgentCore’s by one. Fields filled fell for Tavily and Serper and rose for Exa and AgentCore; where they rose, tea words pulled in tea content, which is the trap §10 describes. F11-c keeps the company query to the name plus the website domain when known.
Limits
- Small sample: 25 companies, one run per search, one judge model. The judge was checked by hand on a handful of cases, not on all 328.
- Existing customers, not cold leads. A company that already buys from Empire is on average more established than a new inquiry, so production results will likely be somewhat worse for every provider.
- Some recorded websites are wrong, because they were found by searching (§10). That affects every provider’s name + domain searches alike.
- Modes not tested: Tavily’s advanced search (twice the price, more text), Exa’s
instantsearch (cheaper), and more than five results per search.
Price comparison, verified 2026-10-13
At the expected volume of about 1,580 searches a year (§4: six shows of ~150 leads, plus ~40 inbound leads a month and the one-off 200-lead backlog), about 130 a month:
| Provider | Price as tested | Paid cost a year | Free allowance | Realistic cost a year | Billing |
|---|---|---|---|---|---|
| AgentCore | $7 / 1,000, plus negligible gateway fees | ~$11 | none | ~$11 | Empire Teas’ AWS bill; no new vendor |
| Exa | $7 / 1,000 (auto, up to 10 results, page text included) |
~$11 | $10 of credit a month (~1,400 searches) | $0 while the free credit lasts; ~$11 without it | Exa account; a card only above the free credit |
| Tavily | $0.008 a search (basic) = $8 / 1,000 | ~$13 | 1,000 searches a month | $0 while the free tier lasts; ~$13 without it | Tavily account; pay-as-you-go, no minimum |
| Serper | $1 / 1,000 in the $50 pack | ~$1.60 | 2,500 trial queries | ~$100 — a $50 pack every six months, mostly expiring unused | Serper account; prepaid packs |
Even a busy show month (~190 searches) sits inside Exa’s and Tavily’s free allowances. Budget at the paid rate: free terms change (Exa’s changed more than once in 2026), and whether each vendor’s free tier may be used for a production system has not been checked. Claude’s cost of reading the results is separate and similar whichever provider is chosen; Exa’s longer text adds a little per lead — an estimate of a few dollars a year at most, not measured.
The test’s own cost, on the dev account: about 87 AgentCore searches (the run, a one-company check and one test search), about $0.61; and about 766,000 input and 52,000 output tokens of Haiku 4.5, about $1.13 at the rates in §9. About $1.75 in all.
The decision, and what it carries
Update 2026-10-14: the lead leans towards Exa, pending Thilina’s approval and Empire Teas’ (Kavindu Anthony), since Exa is billed outside AWS and Empire would add its card there. If AgentCore is not chosen, the dev gateway (forge-et-iaac base/agentcore) and svc-harness’s spike-invoke-gateway policy are removed; the gateway has no standing charge meanwhile (.forge/owed.md).
Update 2026-10-15: Tharinda and Dinil took the comparison to Kavindu Anthony on 14 October
(.forge/discovery/meeting-notes/2026-10-14-kavindu-search-provider.md). He would rather stay on AWS, since
another vendor means another bill and card on Empire’s side, but did not refuse Exa; he asked for the findings
and the recommendation by email, with Sajeev De Silva included. Thilina Abeyrathna has approved Exa (reported
by Tharinda, 2026-10-15), so only Empire’s approval is outstanding. The provider is chosen when Empire answers
that email. The email recommends Exa, with AgentCore as the alternative if Empire will not take an outside
vendor.
Update 2026-10-20: asked by Tharinda on WhatsApp whether there was news on the search tool, Kavindu Anthony answered: “For the moment, can we use the free or a trial version?” Tharinda agreed. The lead’s decision (2026-10-20):
-
Build F11-c on Exa, using its free credit. Exa’s pricing page, read 2026-10-20: $10 of credit on sign-up, reset to $10 on the first of every month, no payment method required; $7 / 1,000 for
autoandfast, $4 forinstant(unchanged since §12). $10 a month is about 1,400 searches at $7 / 1,000, against about 115 to 230 a month in normal use (§12). Exa’s terms of service say “certain of our Services are free” and put no production or evaluation-only limit on them; they also keep Exa’s right to end API access at any time, and the licence over search text that §12 describes applies to the free credit too. -
SP-ARCH-004 stays open. Empire has not answered on Exa itself, so the choice is provisional until it does. If Empire declines an outside vendor, F11-c’s
SearchProviderseam is what lets AgentCore replace Exa without rewriting the feature. -
Altrium holds the Exa account for now. Exa’s API licence is non-transferable, so the account does not move to Empire: before production, Empire signs up for its own account and the backend’s key is replaced (
.forge/owed.md). -
One provider, no fallback (the lead, 2026-10-20). F11-c builds one
SearchProviderimplementation, Exa, and no second one: Tavily is no longer “the pre-decided fallback” (§7), the POC’s Tavily adapter is not ported, and no Tavily account is opened. When Exa fails or refuses a search, the lead stays unresearched and the job retries later, the same posture as the AI layer’s outage path (AD #19). The seam stays: thelocalandtestprofiles use its stub, and a later change of provider is one new implementation.
Is the free credit enough for dev and production? (worked 2026-10-20, from §12’s volumes.) $10 a month is about
1,430 searches at $7 / 1,000 (auto, the mode tested), or 2,500 at instant’s $4, which was not tested:
| Workload | Searches in a month | Within $10? |
|---|---|---|
| Production, a normal month (~40 inbound leads, plus shows averaged over the year; one or two searches a lead) | about 115 to 230 | yes, with about 1,200 to spare |
| Production, a trade-show month (~150 show leads plus ~40 inbound) | about 190 to 380 | yes |
| Production, the month the migrated backlog is researched (692 leads, plus the normal month) | about 810 to 1,610 | yes at one search a lead; up to ~180 over at two, so the batch is spread over two months |
Dev (leads created by the team and Empire’s testers on dev; local and test use the stub and make no calls) |
tens to low hundreds | yes, but it spends the same credit if dev and production share an account |
So the free credit carries both, on two conditions. Dev and production do not share one month’s credit:
dev uses Altrium’s account and production uses Empire’s own (the move .forge/owed.md already requires), so
each environment has its own $10, and a temporary UAT environment uses dev’s key unless Empire asks otherwise.
F11-c does not multiply searches per lead: a re-query on a poor first result (§10) or more than two
searches a lead would eat the margin, and the backlog figure above already needs spreading at two. Exa’s
rate-limit page gives the free plan 10 searches a second (§13); the research job runs from a queue, so a limit
would slow it rather than fail it. Exa has changed its free terms more than once in 2026 (§11), so budget at the paid
rate, about $10 to $19 a year plus $5 to $10 once for the backlog (§12), and treat the free credit as a saving,
not a plan.
Three inputs for the F11-c spec follow from running on the free credit:
- With no card on the account, a search after the month’s credit is used up is refused until the first of the next month. F11-c treats that as the provider being unavailable: the lead stays unresearched and is retried, never shown as “nothing found”.
- The migrated backlog (692 leads, one or two searches each, §12) can take more than one month’s credit, so the batch is spread across months or waits for a paid plan.
- Dev testing spends the same kind of credit. Give dev and production separate accounts (dev on Altrium’s, production on Empire’s — see the paragraph above and §13), so testing cannot use up production’s month.
The provider is not chosen. Dinil decides, with Tharinda:
| Option | For | Against |
|---|---|---|
| Exa | Best results by a clear margin; same price as AgentCore | A new outside vendor; not the pre-decided fallback |
| Tavily | The pre-decided fallback; the POC’s adapter exists | Clearly weaker than Exa here |
| AgentCore | Inside AWS: no new vendor, one bill, the simplest compliance position (§6) | Finds the right company about one search in six |
If an outside vendor is chosen, three questions go with it (Rev 5): who holds the vendor account,
whose card pays, and whether sending lead company names and websites outside AWS is acceptable under
§6’s posture. Because F11-c builds the SearchProvider seam, the provider can be changed later without
rewriting the feature.
Two inputs for the F11-c spec, from this run:
- Discard results about a different company before the snapshot is written. The POC’s prompt says only “do not invent facts”; it does not tell the model to first set aside results about a same-named or unrelated firm. Without that step a lead can get a confident snapshot of a car dealer. The judge’s instruction in this run is a starting point.
- “Nothing found” is a normal outcome (11 of 25 here), and needs its own state on the lead, so an unresearched or unfindable lead is not ranked as a weak one.
12. Review of the analysis, 2026-10-15
Before the recommendation went to Empire Teas, §3, §4 and §11 were checked again. What was checked, and what was found:
Prices — unchanged. Each of the four tested providers’ own pages was read again on 2026-10-15
(the §9 links): AgentCore $7 / 1,000 and the gateway fees as in §3; Exa auto and fast $7 / 1,000 for up to
10 results with page text included (its 3 Mar 2026 changelog), instant $4, and a $10 free credit that
resets on the first of each month, with no subscription or minimum; Tavily $0.008 a credit pay-as-you-go,
basic search 1 credit and advanced 2, 1,000 free credits a month (the Project plan’s price did not render on
the page, so the $30 in §3 was not re-read); Serper’s four packs, from $50 for 50,000 to $3,750 for 12.5
million, each expiring after six months, and 2,500 free trial queries.
The test’s cost — confirmed by the AWS bill. Cost Explorer on the dev account shows, for 13 October, $0.61 for Amazon Bedrock AgentCore and $1.13 for Claude Haiku 4.5, the two figures §11 gives. That matches about 87 AgentCore searches at $7 / 1,000, and is independent evidence that the AgentCore run and the judging took place at the scale §11 describes.
The results tables — consistent. Each provider’s searches by query shape add up to its total (Exa 7+9+8+9 = 33, Serper 28, Tavily 22, AgentCore 14), each column covers 82 searches (25+20+17+20), and 82 searches × 4 providers gives the 328 judged responses. “Found by this provider only” agrees with 14 customers found by at least one provider. §11’s statements in words (half as often as Serper, one search in six, the effect of “tea importer”) match the tables.
Not checked: the per-company judgements and the raw responses. They hold Empire customer names and stay on the developer’s machine (§10, §11), so this review relied on the totals. The limits §11 states — 25 companies, one judge model, existing customers rather than cold leads — still apply.
Two corrections to the volume basis. §4 and §11 count one search per lead and a 200-lead backlog. Two later facts change that:
- The backlog is about 692 leads, the open leads in the 2026-09-13 Pipedrive export that F4-b migrates and F11-c researches in a batch (F4-b spec); “200+” was the PRD’s discovery-time figure.
- The person is researched as well as the company (PRD Rev 56, 2026-10-08), which may add a second search per lead; F11-c designs that lookup.
At one to two searches per lead: about 1,380 to 2,760 searches a year in normal use (six shows of ~150 leads and ~40 inbound leads a month), about 115 to 230 a month, plus 692 to 1,384 once for the migrated backlog. At $7 / 1,000 that is about $10 to $19 a year in normal use and $5 to $10 once for the backlog — still small enough that price does not decide the provider. In the month the backlog is researched, the total (about 800 to 1,600 searches) can pass Exa’s free credit of about 1,400 searches, by a dollar or two at most. A re-query on a poor first result (§10), if F11-c adds one, raises these figures in proportion.
One new point on Exa — what it may do with our searches. Exa’s standard terms of service give Exa a perpetual licence to use what customers send it (the search text) and what it returns, to provide and improve its services; zero data retention is listed only as an Enterprise feature, and the terms point to a data processing agreement where data-protection law applies. AgentCore keeps the search inside Empire Teas’ AWS account (§2). What this means here:
- Company names and website addresses are business identifiers, mostly public already; sending them under these terms is a small concern, and Empire is told in the recommendation email.
- A person’s name is personal data. If F11-c’s person lookup (PRD Rev 56) goes through Exa, its privacy position (the F11-c stub, §6 and the PDPA from 1 January 2027) has to cover this licence: keep person searches off Exa, accept the terms with Exa’s data processing agreement, or ask Exa for Enterprise terms. That is F11-c’s to decide, and it does not change the provider recommendation.
13. Running on Exa’s free credit — the reference, 2026-10-21
One place for the facts F11-c, the DevOps team and whoever holds the Exa accounts need. The decisions behind it are in §11 (the 2026-10-20 notes, Rev 8 and Rev 9). Read from Exa’s own pages on 2026-10-20 and 2026-10-21 (§9); re-read them before building and before production, because Exa’s prices and free terms have changed more than once in 2026.
Prices and the free credit
| Item | Value |
|---|---|
| Free credit | $10 a month, reset to $10 on the first of every month; no payment method required |
| Sign-up bonus | a one-time $10 for the first team that completes onboarding in the dashboard |
auto and fast search (auto is the mode tested in §11) |
$7 / 1,000, up to 10 results with page text included |
instant search (not tested) |
$4 / 1,000 (“up to 2,500 Lite searches” a month on the free credit) |
| Searches the free credit buys | about 1,430 a month at auto; about 2,500 at instant |
| Paid use | pay-as-you-go once a card is added; no subscription or minimum (§12) |
Expected use, and whether the free credit covers it
The table in §11 (2026-10-20 notes) gives the figures: production uses about 115 to 230 searches in a normal
month and up to about 380 in a trade-show month; the one-off research of the 692 migrated leads takes about 690
to 1,380 more; dev uses tens to low hundreds; local and test use the stub and make none. The free credit
covers each environment on its own account, provided F11-c makes at most two searches a lead and spreads the
backlog over two months if it makes two. The budget stays at the paid rate: about $10 to $19 a year in
production plus $5 to $10 once for the backlog (§12). The running-cost annexe’s line for lead-research search
(.forge/client/sow-cost-annexe.md, Exa on its free credit, ≈ $0–2 a month) still holds at these figures.
Rate limits, and what a refused search returns
From Exa’s rate-limit page (read 2026-10-21):
| Endpoint | Limit, per account (shared by all its API keys) |
|---|---|
/search |
10 queries a second — the free plan’s limit is the same as pay-as-you-go’s default |
/search with a deep type |
5 a second |
/contents |
100 a second |
- Over the rate limit:
429 Too Many Requests, withRetry-Afterwhen present. - Credit used up, or a key’s own budget reached:
402 Payment Required. With no card, a402lasts until the first of the next month.
For F11-c: the research job runs from the queue, so it throttles itself well under 10 a second (the backlog
batch included), backs off on a 429 as Retry-After says, and treats a 402 as the provider being
unavailable — the lead stays unresearched and is retried, never shown as “nothing found” (§11). A 402 on dev
more often means a bug (a job searching in a loop) than real use, so it is investigated, not worked around.
Exa lets a budget be set on each API key; a budget on the dev key would stop a runaway loop early.
Exa’s terms that matter here
From Exa’s terms of service (read 2026-10-20 and 2026-10-21):
- Free services are allowed in production. The terms say “certain of our Services are free” and limit them to neither evaluation nor non-production use.
- Exa can end API access at any time (the API licence is “revocable”), and can suspend or delete an
account on a breach. With no fallback provider (§11, Rev 9), the outage path is the lead staying
unresearched; a lasting loss means moving to another provider through the
SearchProviderseam. - The API licence is non-transferable and non-sublicensable, and signing up on behalf of a company binds
that company. So an account cannot be handed from Altrium to Empire: Empire opens its own before production
(
.forge/owed.md). - No reselling or sublicensing the service to a third party without Exa’s consent. A further reason for production to run on Empire’s own account rather than on Altrium’s.
- No circumventing Exa’s technical measures (the prohibited-uses list). This bears on the spare-accounts question below.
- Exa may use what is sent to it and what it returns to provide and improve its services, under a
perpetual licence; zero data retention is an Enterprise feature (§12). Company names and websites are a small
concern. A person’s name is not sent: F11-c’s privacy position (its spec’s decision 5) keeps the person search
off until Empire agrees (
.forge/owed.md). - Exa may change the terms, by updating their “Last updated” date.
The accounts
| Environment | Account | Key |
|---|---|---|
local, test |
none — the stub | none |
| dev, and a temporary UAT environment unless Empire asks otherwise | Altrium’s free account, held for now | the dev key, in dev’s Secrets Manager |
| production | Empire’s own account, opened before production (.forge/owed.md) |
production’s key, in production’s Secrets Manager |
Changing the key is a configuration change with no code change, whichever account it comes from.
Spare accounts to switch to — assessed 2026-10-21, not recommended
The lead asked whether Altrium could open extra Exa accounts and switch dev’s key to another one if a month’s credit ran out (a key change only, no fallback in code). The assessment, given to the lead:
- Dev does not need it: tens to low hundreds of searches a month against about 1,430.
- It puts the account we depend on at risk. Opening accounts to multiply a free allowance works against the intent of the free credit and is close to the terms’ ban on circumventing Exa’s technical measures; Exa can end API access at any time, and if it links the accounts it may close all of them, the main one with them.
- It hides the usual cause: dev running out more often means a loop than real use.
- The fix that is allowed costs little: add a card to the dev account (a few dollars in a heavy month at $7 / 1,000) or wait for the reset on the first. Dev keeps working meanwhile; leads stay unresearched and are retried.
So the plan stays one account per environment, as Rev 9 recorded. If the lead decides otherwise, record it here as a new revision.
Revisions
Rev 0 — 2026-09-16
- Created alongside PRD Rev 30, which made research a requirement on every F11 lead and so turned the provider question from an F36-only item into one serving both features. Carries the options, verified pricing, the ET-002 argument, the compliance position and the two open spike items.
Rev 1 — 2026-09-18
Corrections and additions from reading AWS’s own documentation for the tool. Rev 0 was written from search results and secondary sources, and four of its statements were wrong.
- The integration question is answered on the documentation (narrowed further by Rev 3, which records that it
has still not been run). Rev 0 called the JVM path “unproven” on the
basis that every worked example was Python or TypeScript. AWS states the tool is usable by “any MCP-compatible
client”, and MCP over streamable HTTP is JSON-RPC in a POST body — so it is one SigV4-signed
RestClientcall, no MCP library and no agent framework. The spike drops from two questions to one. What remains is provisioning, not programming: a Gateway must exist before any call, so the AWS arm of the comparison cannot run until the AWS account does. Rev 0’s “a few dollars, one adapter swap” holds only for Tavily and Serper. - Region corrected. Rev 0 said
us-east-1only, in two places. The tool is also ineu-west-1andap-northeast-1. No effect on us —us-east-1is our region (AD-O-9) — but the statement was untrue. - Page rendering corrected from an open question to a known gap. Rev 0 said “AgentCore’s Fetch may or may not
cover it”. The connector has no fetch operation; it does semantic snippet extraction. Tavily’s
extracthas no counterpart. - The test method was wrong in three ways and is rewritten. Rev 0 pointed at the POC’s
sample-exhibitors.csvwithout opening it; it is twenty global retailers (Aldi, Lidl, Tesco) and cannot test long-tail coverage. The population is also CRM leads before exhibitors — F11-c is Phase 1, F36-c was Phase 2 then (Phase 1 since 2026-11-11) — and the manual-add path means the test needs three input tiers rather than one. Bias in list construction, and the confidentiality of client names, are now stated conditions of the method rather than afterthoughts. - Added: how the call is actually made (endpoint, SigV4, the
tools/callbody, the input limits, the response shape), so the implementer does not repeat the research; and the knowledge graph and semantic snippet extraction capabilities, which bear on the quality comparison. - Added: the acceptable-use term requiring source links to be displayed, not merely stored — a constraint on F11-f and F11-g, absent from Rev 0 entirely.
- Added: two questions F11-c’s spec must answer (what triggers research across the four lead entry paths; whether editing a company name re-runs it), recorded in §8 because no spec owns them yet.
- The spike now has an id and a home in the tracker: SP-ARCH-004. Rev 0 used the word “spike” informally; it was not tracked anywhere, had no owner field, no estimate and no settle-by.
Rev 2 — 2026-09-18
Stage 1 of SP-ARCH-004 ran: Tavily, Serper and Exa against 25 real Empire customers, four query shapes, about 250 searches, no errors. Recorded in the new §10.
- One firm result — do not add context to the search query. Adding the country, and adding “tea importer”, made every provider return tea-industry directory pages instead of the company: 14 of 19 companies shared no result site between the plain and the contextual query, and one directory appeared 31 times across different companies. F11-c builds the company query from the company name plus the website domain when known, and nothing else (the person lookup is separate, PRD Rev 56). The trap is that the wrong approach scores better on any keyword-based relevance check.
- The provider ranking is withdrawn as unproven. Stage 1 scored “found the company” by whether the name appeared in a result’s web address. That is invalid both ways: a four-letter customer name matched a US car dealership and counted as success, while a publicly listed tea supplier counted as a failure although all three providers had returned its real site — its web address is an abbreviation sharing almost no letters with its name. Exa led at every tier; that is a hint for stage 2, not a finding.
- The correct measure is the model reading the results, which is also how the feature works in production: the search surfaces a page, Claude decides whether it is the right company. That needs Bedrock, so it waits for the AWS account and runs together with the AgentCore arm. The saved responses are re-scored then at no further search cost. (Corrected by Rev 4: Rev 2 called this a separate “model key”. Bedrock uses AWS credentials — the account is the only blocker.)
- A design question for F11-c, raised by the run: the prototype issues one search, takes five results and stops. A lead whose first search returns nothing useful gets no second attempt. Whether the job re-queries on a poor result is the spec’s to answer.
- Artifacts are outside this repository — they carry Empire customer names. They are on the
developer’s machine at
Desktop/empire-search-test/, with a README explaining how to add AgentCore later against the same 25 companies.
Rev 3 — 2026-09-18
- “The integration question is closed” is narrowed to “answered on the documentation, not yet run.” Rev 1 said
a Spring Boot backend can call AgentCore with one SigV4-signed
RestClientrequest and treated that as settled. It came from reading AWS’s documentation; no Java has been executed against the tool. Rev 2 had just finished recording a case where this document trusted an unvalidated measurement and was wrong (§10), so the same standard applies here. It matters because AgentCore is the recommendation and its whole case assumes calling it is easy. - Owed when the AWS account lands: roughly thirty lines of standalone Java that sign one request and read the response — run once, discarded, before the real adapter is built. About an hour, and it runs alongside the stage-2 search comparison since both wait on the same account.
- Recorded for the avoidance of doubt: the stage-1 comparison script is Python on purpose. It measures what the search services return, which is identical whichever language asks. Java is only relevant to the integration question above.
Rev 4 — 2026-09-18
- Corrects Rev 2 and Rev 3: stage 2 has ONE blocker, not two. Those revisions said it waited on “an AWS account and a model key”. Bedrock authenticates with ordinary AWS credentials — there is no separate API key to obtain. The AWS account supplies both halves of stage 2: the Gateway that AgentCore needs, and Claude for the scoring. The only additional step is enabling access to the pinned models within the account, which belongs to standing the account up rather than to this spike.
- No change to what stage 2 does or to any finding. It removes a dependency that never existed and would have had someone looking for a key nobody needs to buy.
Rev 5 — 2026-09-29
- §5’s premise is superseded. SOW ET-002 is not in force: the AWS account is Empire Teas’ own and AWS bills Empire Teas directly. A note at the top of §5 restates the argument on that basis and leaves §5’s text as written.
- One question added to the spike’s output: if Tavily is chosen, whose card pays for it and who holds the vendor account. Dinil answers it after stage 2, with the provider choice. No finding and no stage-2 step changes.
- Stage 2 can now start once Shyamin Ayesh has Bedrock access and the AgentCore Gateway in Empire Teas’ account.
- §6’s PDPA timing is annotated: with sign-off now Fri 29 January 2027, Sri Lanka’s PDPA Parts I and III (1 January 2027) take effect before go-live rather than after it.
Rev 6 — 2026-10-13
SP-ARCH-004 stage 2 ran in the dev account. Recorded in the new §11.
- AgentCore came last on quality. Judged by Claude Haiku 4.5, blind to the provider, AgentCore found the right company for 5 of the 25 customers (14 of 82 searches), against Exa 13 (33), Serper 11 (28) and Tavily 8 (22). 11 customers were found by no provider. §7 carries a dated note; its recommendation text is kept as written.
- The provider is not chosen. Dinil decides with Tharinda between Exa, Tavily and AgentCore (§11), and answers Rev 5’s vendor-account and card questions if an outside vendor is chosen. The standalone Java check (Rev 3) is owed only if AgentCore stays in the running.
- Prices re-verified for the four tested providers (§3, §9, §11). §4’s “$30/month Tavily floor” argument is corrected: Tavily has pay-as-you-go with no minimum. AgentCore, Exa and Tavily all cost about $11–13 a year at their paid rates, and Exa’s and Tavily’s free allowances cover the expected volume; Serper’s $50 pack with six-month expiry makes it about $100 a year.
- §2’s call details corrected from a real call: the tool is
web-search___WebSearchthrough the gateway, the request carries anAcceptheader for MCP’s streamable HTTP, and the caller needsbedrock-agentcore:InvokeGateway, whichAmazonBedrockFullAccessdoes not grant. - Two inputs for the F11-c spec (§11): discard results about a different company before writing the snapshot, and give “nothing found” its own state.
Rev 7 — 2026-10-15
The comparison went to Empire Teas, and the analysis was reviewed before the recommendation was sent (§12).
- Empire heard the comparison on 14 October and asked for the findings and the recommendation by email
(
.forge/discovery/meeting-notes/2026-10-14-kavindu-search-provider.md). Kavindu Anthony would rather stay on AWS but did not refuse Exa. The recommendation is Exa, with AgentCore as the alternative; the provider is chosen when Empire answers. Notes added to the header and §11. - Review (§12): the four providers’ prices are unchanged on 2026-10-15; the AWS bill confirms the test’s $0.61 of AgentCore and $1.13 of Haiku 4.5; the results tables are internally consistent. The per-company judgements were not re-checked (they stay off the repository).
- Volume basis corrected: about 692 backlog leads, not 200, and possibly a second search per lead for the person (PRD Rev 56). Cost stays about $10–19 a year plus $5–10 once.
- New: Exa’s standard terms let it use the search text to improve its services; zero data retention is Enterprise-only. A concern for the person lookup, which F11-c’s privacy position has to cover, not for company names.
Rev 8 — 2026-10-20
Empire answered for now: use a free or trial version (Kavindu Anthony on WhatsApp, 2026-10-20). Notes added to the header and §11; Exa’s and Tavily’s pages added to §9.
- The lead’s decision: F11-c is built on Exa’s free credit ($10 a month, reset on the first, no card — re-read 2026-10-20), about 1,400 searches a month against 115 to 230 needed. SP-ARCH-004 stays open until Empire answers on Exa itself. Altrium holds the Exa account for now; Exa’s API licence is non-transferable, so before production Empire signs up for its own account and the key is replaced.
- Exa’s terms put no production limit on its free services, keep Exa’s right to end API access at any time, and keep the licence over search text from §12.
- Three inputs for the F11-c spec: a search refused because the month’s credit is used up is the provider being unavailable, not “nothing found”; the 692-lead backlog may need more than one month’s credit; dev and production use separate keys.
Rev 9 — 2026-10-20
Two more points settled the same day (§7 and §11 notes, header).
- One provider, no fallback (the lead’s decision). F11-c builds the Exa implementation only. Tavily is no
longer the pre-decided fallback, its POC adapter is not ported, and no Tavily account is opened. A failed or
refused search leaves the lead unresearched and the job retries, as the AI layer degrades to manual (AD #19).
The
SearchProviderseam stays for thelocal/teststub and any later change of provider. - The free credit covers dev and production (table in §11): about 1,430 searches a month against 115 to 380 in production and tens to low hundreds on dev, provided each environment has its own account (dev on Altrium’s, production on Empire’s) and F11-c makes at most two searches a lead. The backlog month needs the batch spread over two months at two searches a lead. Budget stays at the paid rate.
Rev 10 — 2026-10-21
- New §13, one reference for running on Exa’s free credit: prices and the free credit (with the one-time
$10 sign-up bonus), expected use against it, rate limits and the
429/402responses, the terms that matter here, the accounts per environment, and the assessment of spare accounts. - Rate limits read from Exa’s rate-limit page: the free plan allows 10
/searchqueries a second, shared by all of an account’s keys;429over the limit,402when the credit or a key’s budget is spent. This corrects the §11 note of 2026-10-20 that no free-tier rate limit is published. - Terms added: the licence is revocable, non-transferable and non-sublicensable; signing up binds the company; no reselling or sublicensing to a third party without consent; no circumventing Exa’s technical measures; the terms may change.
- Spare Altrium accounts to switch dev’s key to (raised by the lead, 2026-10-21): assessed and not recommended (§13); the plan stays one account per environment.
- §11’s third free-credit input corrected: dev and production are separated by account, not by Exa team.
- Checked: the running-cost annexe’s lead-research line (≈ $0–2 a month, provider pending) still holds.
Rev 11 — 2026-11-11
- Exa is chosen. Kavindu Anthony confirmed Exa for Empire Teas in the progress meeting of 10 November 2026
(meeting note §8, recorded from Dinil’s account; not in the transcript). The provisional choice of Rev 8 is
now the decision, and SP-ARCH-004 is closed in
tracker.yaml. AgentCore is not chosen, so the dev gateway and thespike-invoke-gatewaypolicy can go. (Update 2026-11-19: forge-et-iaac#28, open, removes the gateway, its target and its role fromdev;svc-harness’sspike-invoke-gatewaypolicy is not Terraform and is removed by hand once #28 is applied.) - The person search may go to Exa, under Empire’s control. The team proposed a switch on the CRM’s Settings
page with which an administrator turns the person search on or off, on by default, and Kavindu accepted it.
This replaces the F11-c position that the person search is a server setting, off until Empire agrees: F11-c
is revised to build the switch (
owed.md). Before the switch is on in an environment, F11-c FR-P2’s conditions still apply there: Empire’s agreement in writing and Exa’s data processing agreement for that environment’s account (§12, §13). - Unchanged: one account per environment, Altrium’s for dev and Empire’s own before production (§13), and the running-cost line. §11 and §13 already count one or two searches a lead, so the person search fits the free credit.
Rev 12 — 2026-11-17
- The switch is F11-i (
.forge/specs/F11-i-person-search-switch-spec.md, approved 2026-11-17 under Dinil’s overnight delegation; F11-c Rev 9; PRD Rev 99): a stored switch on the Lead research Settings page, created on in every environment, under its own keysettings.lead_research.manage; the server settinget.research.person-search.enabledis retired. - Rev 11’s condition becomes a rule about real leads (F11-i decision 8): the switch ships on, so before
Empire’s real leads enter an environment — a migration run of Empire’s export on dev, real business cards
captured on dev or UAT, a UAT round, production — either Empire’s written agreement and Exa’s data processing
agreement cover that environment’s account (§12, §13), or its administrator turns the switch off first
(
owed.md).
Rev 13 — 2026-11-18
- The rule about real leads is withdrawn (F11-i spec Rev 3, decision 8). Dinil decided on 2026-11-18 that
the switch is on in every environment, with no precondition: Empire agreed to this approach on 2026-11-10
(Rev 11). Rev 11’s condition and Rev 12’s rule no longer apply; Empire’s agreement in writing and Exa’s data
processing agreement stay owed (
owed.md), not as a gate.