Repository navigation
feat(examples): add with-firecrawl-search #1433
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
JuampiHernandez
wants to merge
4
commits into
VoltAgent:main
Choose a base branch
from
JuampiHernandez:feat/examples-with-firecrawl-search
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
e6344d3
feat(examples): add with-firecrawl-search
JuampiHernandez a8873db
fix(examples): use .js import and surface invalid JSON in with-firecr…
JuampiHernandez c18202a
fix(examples): add request timeout and NodeNext config to with-firecr…
JuampiHernandez 4770d22
fix(examples): use safeStringify in with-firecrawl-search
JuampiHernandez File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # OpenAI API Key (required) | ||
| OPENAI_API_KEY=your_openai_api_key_here | ||
|
|
||
| # Firecrawl API Key (required) | ||
| # Get one at https://www.firecrawl.dev/signin?utm_source=voltagent&utm_medium=integration&redirect=%2Fapp%2Fapi-keys | ||
| FIRECRAWL_API_KEY=your_firecrawl_api_key_here | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| # VoltAgent with Firecrawl Search | ||
|
|
||
| This example shows how to give a VoltAgent agent web search and page reading with Firecrawl. [Firecrawl search](https://www.firecrawl.dev/search?utm_source=voltagent&utm_medium=integration) returns web results and can include each page's content as Markdown in the same call. A second tool fetches any URL as Markdown. | ||
|
|
||
| ## Features | ||
|
|
||
| - **Web Search**: Find current pages on any topic, with titles, URLs and descriptions | ||
| - **Page Content in Search**: Optionally get each result's page content as Markdown in the same call | ||
| - **Page Reading**: Fetch any URL as clean Markdown | ||
| - **Cited Answers**: The agent searches first, reads the most relevant pages when needed, and cites the URLs it used | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| 1. **Firecrawl API Key**: Get your API key from the [Firecrawl dashboard](https://www.firecrawl.dev/signin?utm_source=voltagent&utm_medium=integration&redirect=%2Fapp%2Fapi-keys) | ||
| 2. **OpenAI API Key**: For the AI model integration | ||
|
|
||
| ## Setup | ||
|
|
||
| 1. **Install dependencies**: | ||
|
|
||
| ```bash | ||
| pnpm install | ||
| ``` | ||
|
|
||
| 2. **Configure environment variables**: | ||
| Copy the example environment file and add your API keys: | ||
|
|
||
| ```bash | ||
| cp .env.example .env | ||
| ``` | ||
|
|
||
| Then edit the `.env` file with your actual API keys: | ||
|
|
||
| ```env | ||
| OPENAI_API_KEY=your_actual_openai_api_key | ||
| FIRECRAWL_API_KEY=your_actual_firecrawl_api_key | ||
| ``` | ||
|
|
||
| 3. **Run the example**: | ||
| ```bash | ||
| pnpm dev | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| The agent will be available at `http://localhost:3141`. Try queries like: | ||
|
|
||
| - "What changed in the latest Node.js LTS release?" | ||
| - "Find the VoltAgent docs page about tools and summarize how to create one." | ||
| - "Read https://voltagent.dev/docs/ and list the main sections." | ||
|
|
||
| ## Tools Available | ||
|
|
||
| ### 1. Firecrawl Search Tool | ||
|
|
||
| - **Purpose**: Search the web, optionally with page content | ||
| - **Endpoint**: `POST https://api.firecrawl.dev/v2/search` | ||
| - **Parameters**: | ||
| - `query`: Search query string | ||
| - `limit`: Number of results to return (default: 5, max: 10) | ||
| - `scrapeContent`: Include each result's page content as Markdown (default: false) | ||
|
|
||
| ### 2. Firecrawl Scrape Tool | ||
|
|
||
| - **Purpose**: Fetch a single URL as Markdown | ||
| - **Endpoint**: `POST https://api.firecrawl.dev/v2/scrape` | ||
| - **Parameters**: | ||
| - `url`: URL of the page to fetch | ||
| - `onlyMainContent`: Drop navigation, headers and footers (default: true) | ||
|
|
||
| Page content is truncated so it fits in the model's context: 5,000 characters per search result and 20,000 characters for a scraped page. You can adjust both limits with `MAX_SEARCH_MARKDOWN_LENGTH` and `MAX_SCRAPE_MARKDOWN_LENGTH` in `src/tools.ts`. Returning page content with search uses more Firecrawl credits per result. | ||
|
|
||
| See the [Firecrawl search docs](https://docs.firecrawl.dev/features/search?utm_source=voltagent&utm_medium=integration) for more options, such as `sources`, `location` and `tbs`. | ||
|
|
||
| ## Configuration | ||
|
|
||
| The agent is configured with: | ||
|
|
||
| - **Model**: GPT-4o-mini | ||
| - **Port**: 3141 (configurable) | ||
| - **Logging**: Pino logger with info level | ||
| - **Memory**: LibSQL | ||
| - **Tools**: Firecrawl search and scrape tools | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| - **API Key Issues**: Ensure both `OPENAI_API_KEY` and `FIRECRAWL_API_KEY` are set | ||
| - **401 errors from Firecrawl**: Check that the key in `.env` is correct and active | ||
| - **402 or 429 errors from Firecrawl**: Your account is out of credits or over its rate limit; check your usage in the Firecrawl dashboard | ||
| - **Port Conflicts**: Change the port in the server configuration if needed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| { | ||
| "name": "voltagent-example-with-firecrawl-search", | ||
| "author": "", | ||
| "dependencies": { | ||
| "@voltagent/cli": "^0.1.21", | ||
| "@voltagent/core": "^2.11.0", | ||
| "@voltagent/internal": "^1.0.3", | ||
| "@voltagent/libsql": "^2.1.2", | ||
| "@voltagent/logger": "^2.0.2", | ||
| "@voltagent/server-hono": "^2.0.14", | ||
| "ai": "^6.0.0", | ||
| "zod": "^3.25.76" | ||
| }, | ||
| "devDependencies": { | ||
| "@types/node": "^24.2.1", | ||
| "tsx": "^4.21.0", | ||
| "typescript": "^5.8.2" | ||
| }, | ||
| "keywords": [ | ||
| "agent", | ||
| "ai", | ||
| "voltagent" | ||
| ], | ||
| "license": "MIT", | ||
| "private": true, | ||
| "repository": { | ||
| "type": "git", | ||
| "url": "https://github.com/VoltAgent/voltagent.git", | ||
| "directory": "examples/with-firecrawl-search" | ||
| }, | ||
| "scripts": { | ||
| "build": "tsc", | ||
| "dev": "tsx watch --env-file=.env ./src", | ||
| "start": "node dist/index.js", | ||
| "volt": "volt" | ||
| }, | ||
| "type": "module" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| import { Agent, Memory, VoltAgent } from "@voltagent/core"; | ||
| import { LibSQLMemoryAdapter } from "@voltagent/libsql"; | ||
| import { createPinoLogger } from "@voltagent/logger"; | ||
| import { honoServer } from "@voltagent/server-hono"; | ||
| import { firecrawlScrapeTool, firecrawlSearchTool } from "./tools.js"; | ||
|
|
||
| // Create logger | ||
| const logger = createPinoLogger({ | ||
| name: "firecrawl-search-agent", | ||
| level: "info", | ||
| }); | ||
|
|
||
| // Create Memory instance backed by LibSQL | ||
| const memory = new Memory({ | ||
| storage: new LibSQLMemoryAdapter(), | ||
| }); | ||
|
|
||
| // Create the research agent with Firecrawl tools | ||
| const researchAgent = new Agent({ | ||
| name: "Web Research Agent", | ||
| instructions: `You answer questions with current information from the web. | ||
|
|
||
| 1. Use firecrawlSearch to find relevant pages. | ||
| 2. If the search descriptions are not enough to answer, use firecrawlScrape on the one or two most relevant URLs to read the full page. | ||
| 3. Answer from what you found and cite the URLs you used. | ||
|
|
||
| If a tool returns an error, tell the user what went wrong.`, | ||
| model: "openai/gpt-4o-mini", | ||
| tools: [firecrawlSearchTool, firecrawlScrapeTool], | ||
| memory, | ||
| }); | ||
|
|
||
| // Initialize the VoltAgent with the research agent and server | ||
| new VoltAgent({ | ||
| agents: { | ||
| researchAgent, | ||
| }, | ||
| logger, | ||
| server: honoServer(), | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| import { createTool } from "@voltagent/core"; | ||
| import { safeStringify } from "@voltagent/internal"; | ||
| import { z } from "zod"; | ||
|
|
||
| const FIRECRAWL_API_URL = "https://api.firecrawl.dev/v2"; | ||
| const REQUEST_TIMEOUT_MS = 120_000; | ||
|
|
||
| // Keep page content small enough for the model's context window | ||
| const MAX_SEARCH_MARKDOWN_LENGTH = 5_000; | ||
| const MAX_SCRAPE_MARKDOWN_LENGTH = 20_000; | ||
|
|
||
| const missingApiKeyResult = { | ||
| success: false, | ||
| error: "Firecrawl API key not configured", | ||
| message: "Firecrawl API key is required. Please set FIRECRAWL_API_KEY environment variable.", | ||
| }; | ||
|
|
||
| const truncateMarkdown = (markdown: string, maxLength: number) => | ||
| markdown.length > maxLength ? `${markdown.slice(0, maxLength)}\n\n[Content truncated]` : markdown; | ||
|
|
||
| const callFirecrawl = async (path: string, apiKey: string, body: Record<string, unknown>) => { | ||
| const response = await fetch(`${FIRECRAWL_API_URL}${path}`, { | ||
| method: "POST", | ||
| headers: { | ||
| "Content-Type": "application/json", | ||
| Authorization: `Bearer ${apiKey}`, | ||
| }, | ||
| body: safeStringify({ ...body, origin: "voltagent" }), | ||
| signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS), | ||
| }); | ||
|
cubic-dev-ai[bot] marked this conversation as resolved.
|
||
|
|
||
| // Keep the HTTP status error for failed responses, but surface invalid JSON on success | ||
| const data = await response.json().catch((error) => { | ||
| if (response.ok) throw error; | ||
| return {}; | ||
| }); | ||
|
|
||
| if (!response.ok) { | ||
| const detail = typeof data.error === "string" ? `: ${data.error}` : ""; | ||
| throw new Error(`Firecrawl API error ${response.status}${detail}`); | ||
| } | ||
|
|
||
| return data; | ||
| }; | ||
|
|
||
| export const firecrawlSearchTool = createTool({ | ||
| name: "firecrawlSearch", | ||
| description: | ||
| "Search the web with Firecrawl and return titles, URLs and descriptions, optionally with the page content as Markdown.", | ||
| parameters: z.object({ | ||
| query: z.string().describe("Search query (e.g., 'latest TypeScript release notes')"), | ||
| limit: z | ||
| .number() | ||
| .int() | ||
| .min(1) | ||
| .max(10) | ||
| .optional() | ||
| .describe("Number of results to return (default: 5, max: 10)"), | ||
| scrapeContent: z | ||
| .boolean() | ||
| .optional() | ||
| .describe("Also return each result's page content as Markdown (default: false)"), | ||
| }), | ||
| execute: async ({ query, limit = 5, scrapeContent = false }) => { | ||
| try { | ||
| console.log("Firecrawl search:", query); | ||
|
|
||
| const apiKey = process.env.FIRECRAWL_API_KEY; | ||
| if (!apiKey) { | ||
| return missingApiKeyResult; | ||
| } | ||
|
|
||
| const data = await callFirecrawl("/search", apiKey, { | ||
| query, | ||
| limit, | ||
| ...(scrapeContent ? { scrapeOptions: { formats: ["markdown"] } } : {}), | ||
| }); | ||
|
|
||
| const results = (data.data?.web ?? []).map((item: any) => ({ | ||
| title: item.title || "No Title", | ||
| url: item.url || "", | ||
| description: item.description || "", | ||
| ...(item.markdown | ||
| ? { markdown: truncateMarkdown(item.markdown, MAX_SEARCH_MARKDOWN_LENGTH) } | ||
| : {}), | ||
| })); | ||
|
|
||
| return { | ||
| success: true, | ||
| query, | ||
| results, | ||
| totalResults: results.length, | ||
| message: | ||
| results.length > 0 | ||
| ? `Found ${results.length} web results for "${query}".` | ||
| : `No web results found for "${query}". Try a different search query.`, | ||
| }; | ||
| } catch (error) { | ||
| console.error("Firecrawl search error:", error); | ||
| return { | ||
| success: false, | ||
| error: error instanceof Error ? error.message : "Firecrawl search failed", | ||
| message: `Firecrawl search failed: ${error instanceof Error ? error.message : "Unknown error"}`, | ||
| }; | ||
| } | ||
| }, | ||
| }); | ||
|
|
||
| export const firecrawlScrapeTool = createTool({ | ||
| name: "firecrawlScrape", | ||
| description: "Fetch a URL with Firecrawl and return the page as clean Markdown.", | ||
| parameters: z.object({ | ||
| url: z.string().describe("URL of the page to fetch"), | ||
| onlyMainContent: z | ||
| .boolean() | ||
| .optional() | ||
| .describe("Drop navigation, headers and footers (default: true)"), | ||
| }), | ||
| execute: async ({ url, onlyMainContent = true }) => { | ||
| try { | ||
| console.log("Firecrawl scrape:", url); | ||
|
|
||
| const apiKey = process.env.FIRECRAWL_API_KEY; | ||
| if (!apiKey) { | ||
| return missingApiKeyResult; | ||
| } | ||
|
|
||
| const data = await callFirecrawl("/scrape", apiKey, { | ||
| url, | ||
| formats: ["markdown"], | ||
| onlyMainContent, | ||
| }); | ||
|
|
||
| const markdown = data.data?.markdown; | ||
| if (!markdown) { | ||
| return { | ||
| success: false, | ||
| error: "No content returned", | ||
| message: `No content could be fetched from ${url}`, | ||
| }; | ||
| } | ||
|
|
||
| return { | ||
| success: true, | ||
| url, | ||
| title: data.data.metadata?.title || "No Title", | ||
| markdown: truncateMarkdown(markdown, MAX_SCRAPE_MARKDOWN_LENGTH), | ||
| message: `Fetched ${url} as Markdown.`, | ||
| }; | ||
| } catch (error) { | ||
| console.error("Firecrawl scrape error:", error); | ||
| return { | ||
| success: false, | ||
| error: error instanceof Error ? error.message : "Firecrawl scrape failed", | ||
| message: `Firecrawl scrape failed: ${error instanceof Error ? error.message : "Unknown error"}`, | ||
| }; | ||
| } | ||
| }, | ||
| }); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "compilerOptions": { | ||
| "target": "ES2022", | ||
| "module": "NodeNext", | ||
| "moduleResolution": "NodeNext", | ||
| "outDir": "dist", | ||
| "esModuleInterop": true, | ||
|
cubic-dev-ai[bot] marked this conversation as resolved.
|
||
| "forceConsistentCasingInFileNames": true, | ||
| "allowSyntheticDefaultImports": true, | ||
| "strict": true, | ||
| "skipLibCheck": true, | ||
| "declaration": true, | ||
| "declarationMap": true, | ||
| "sourceMap": true | ||
| }, | ||
| "include": ["src/**/*"] | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.