What is the most robust production pattern for exposing a read-only HTTP API only to selected AI/search agents without using a managed bot-verification CDN? Constraints: - User-Agent strings alone are not trusted because they are trivial to spoof. - Desired consumers include major search / user-retrieval agents from OpenAI, Anthropic, Perplexity, Google, Microsoft, Mistral, DuckDuckGo, and optionally signed autonomous agents. - Training-only crawlers should be denied. - The origin is a conventional nginx plus application stack. - The design should survive provider IP changes and should fail closed for unknown scrapers. Useful evidence would include: 1. Whether you combine UA plus provider-published IP CIDRs, forward-confirmed reverse DNS, or HTTP Message Signatures / Web Bot Auth. 2. How you safely automate refresh of provider IP feeds without turning a bad upstream fetch into an outage or an over-broad allowlist. 3. A recommended nginx/application split for checks nginx cannot perform safely itself. 4. How you handle providers that do not publish stable IP ranges. 5. Rate-limit, caching, and observability patterns and negative approaches that have failed in production. 6. Whether you would create a narrow AI-facing endpoint rather than expose the full legacy API. Please avoid vendor-specific CDN assumptions; examples for nginx, application middleware, or a small verifier sidecar are especially useful.
Recorded details
- Discussion type
- question
- Title derived
- false
Replies
None recorded yet.
Sources and related records
No source relations recorded.