NativeAIHub

LLM Integration Patterns

All plans2 min read

RAG Pipelines

The most common Firecrawl integration pattern is feeding web content into retrieval augmented generation systems. Firecrawl's clean markdown output eliminates custom HTML parsing and produces consistently structured content that chunks well for vector databases.

1

Map

Discover all URLs on the target site using sitemap scanning

2

Batch Scrape

Scrape relevant pages and get clean markdown output

3

Chunk

Split markdown by heading hierarchy into semantic sections

4

Embed

Store chunks in Pinecone, Weaviate, or Chroma

5

Retrieve

Query relevant chunks when the LLM needs context

MCP setup for AI coding agents

One command gives Claude Code, Cursor, and Windsurf full web scraping capability: npx -y firecrawl-cli@latest init --all --browser. The MCP server exposes all endpoints as tools (firecrawl_scrape, firecrawl_map, firecrawl_search, firecrawl_crawl, firecrawl_extract, firecrawl_agent). Remote hosted URL: https://mcp.firecrawl.dev/{API_KEY}/v2/mcp

Framework integrations

🔗
LangChainFirecrawlLoader document loader for seamless integration with LangChain RAG chains. Drop in replacement for other document loaders.
🦙
LlamaIndexFirecrawlWebReader for ingesting web content into LlamaIndex indices. Works with any LlamaIndex pipeline out of the box.
🤖
OpenAIFunction calling compatible responses for building OpenAI tool use agents. Returns structured data ready for function schemas.

Real time context for AI assistants

Use the Search endpoint to give AI assistants access to current web information. When a user asks a question requiring up to date data, search via Firecrawl and include the scraped content in the LLM context. This is how many "search grounded" AI assistants work under the hood.