Skip to content
Pedro Gomes
All work

AI Web Scraping Agent

Give it a URL and describe what you want in one sentence. It returns validated JSON, and it is hard to turn against the server it runs on.

Open the demo (opens in a new tab)Source code (opens in a new tab)

Who has this problem

Turning a web page into structured data is the easy part. Running it as a public service means stopping it from being pointed at internal addresses, keeping instructions hidden in a page from steering the model, and making sure one user’s cached results never reach another.

My role

Web application, API, URL safety controls and tenant-aware result cache.

How to try it

  1. Open the demo, paste a URL and describe in one sentence what you want. The result comes back as validated JSON.

Usage conditions

  • 5 scrapes per session, 10 requests per minute and 5 browsers at a time.
  • 200 calls per day on the shared key. With your own key, that cap does not apply.
  • A kill switch that stops paid calls without a redeploy.

Public access has usage limits to keep AI service costs under control.

Decisions I made

  • Every address is checked before the request, including internationalised domains and IPv6, and checked again for every resource the page loads, so a domain that changes its DNS mid-request cannot reach an internal address.
  • Page content is passed to the model as untrusted data, isolated from the instructions, so text on a page cannot give the model orders.
  • Bring your own key for 6 providers (Groq, OpenAI, Anthropic, DeepSeek, Gemini and Grok). The public demo uses a shared free key with limits per visitor.
  • The cache key includes a hash of the API key, so cached results are never shared between users.
  • The backend sits on an internal Docker network; only the web app can reach it.

Limits that remain

  • Pages behind a login are out of scope.