What you’ll need

  • Python 3.10+ and pip — the crawler runs a real browser, so budget roughly 2 GB of disk for the browser download on first setup.
  • No API keys. Every step in this tutorial runs locally with zero LLM calls — that’s the whole point.

Step 1 — install the crawler

Two commands. The second one downloads the browser, once:

pip install -U crawl4ai
crawl4ai-setup

Step 2 — scrape your first page

The core loop is three lines: open the crawler, run it on a URL, read result.markdown. Save this as scrape.py:

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://news.ycombinator.com")
        print(result.markdown)

asyncio.run(main())

Run it with python scrape.py. What you get is not HTML soup: it’s smart Markdown with headings, lists, tables, and code blocks in a structure an LLM reads well, and every page link becomes a numbered citation. Print just the first 2,000 characters and you’ll see why this output feeds a chunker better than anything you’d regex together yourself.

An abstract illustration of a web page dissolving into a clean markdown document with an arrow between them
Page to Markdown: Crawl4AI’s core job. Illustration generated for this tutorial.

Step 3 — do it from the terminal with crwl

The same engine ships as a CLI for quick jobs. One page as Markdown:

crwl https://news.ycombinator.com -o markdown

Deep crawl, breadth first, at most ten pages — useful when you want a whole docs site, not one page:

crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10

There’s also a -q mode that answers a question about a page (crwl https://www.example.com/products -q "Extract all product prices"), but that one needs an LLM key configured via crwl config — skip it for now, because the next step gets you structured data without any model at all.

Step 4 — pull structured JSON with a CSS schema

Markdown is for reading. For pipelines you want typed records. Crawl4AI’s JsonCssExtractionStrategy extracts them with CSS selectors — no LLM, no tokens. This example is straight from the project’s own docs and runs against quotes.toscrape.com, the canonical scraping test site:

import asyncio, json
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, JsonCssExtractionStrategy

schema = {
    "name": "Quotes",
    "baseSelector": "div.quote",
    "fields": [
        {"name": "text", "selector": "span.text", "type": "text"},
        {"name": "author", "selector": "small.author", "type": "text"},
        {"name": "tags", "selector": "a.tag", "type": "list", "fields": [{"name": "tag", "type": "text"}]},
    ],
}

async def main():
    run_config = CrawlerRunConfig(
        extraction_strategy=JsonCssExtractionStrategy(schema),
        scan_full_page=True,   # scroll to the end, so the page loads every quote
        scroll_delay=0.5,
        cache_mode=CacheMode.BYPASS,
    )
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun(url="https://quotes.toscrape.com/scroll", config=run_config)
        quotes = json.loads(result.extracted_content)
        print(f"Extracted {len(quotes)} quotes")
        print(json.dumps(quotes[0], indent=2))

asyncio.run(main())

result.extracted_content comes back as a JSON string of typed records — text, author, tags — ready to load into a database or a vector store. Note scan_full_page=True: it scrolls infinite-scroll pages to the end before extracting. For your own targets, swap the selectors; the pattern is identical.

An abstract branching tree of crawled web pages spreading from a central node, each leaf turning into a tidy JSON data card
From a page tree to typed JSON records. Illustration generated for this tutorial.

What you built

A local, reproducible pipeline with no keys and no cloud bill: any URL goes in, clean LLM-ready Markdown and structured JSON come out. Point the Step 2 script at your docs, the Step 3 deep crawl at a knowledge base, the Step 4 schema at a listing page — the outputs drop straight into a RAG indexer.

Honest limitations

  • It runs a real browser. Expect real RAM, CPU, and disk usage — this is not a curl replacement for every job.
  • Bot walls exist. The README documents enable_stealth, proxies with authentication and rotation, persistent profiles with saved logins, and full header/cookie control — those are controls for sites that allow automation, not a pass to ignore a site’s terms. Check the terms of anything you crawl at volume.
  • Heavy dynamic pages need configuration. Waits, scrolling, and sessions handle most of it; the browser-control docs are where you go when the defaults don’t.
  • You don’t have to run browsers at all. The same scrape/extract/search endpoints exist as a hosted product (Crawl4AI Cloud, free credit to start, pay as you go). The library itself is free, forever — this tutorial never touched the cloud.