Getting started with Vantyx Crawl

Vantyx Crawl is a Firecrawl-compatible web data API that lets you scrape pages, crawl sites, search the web, and extract structured data for AI applications. This guide walks you through your first steps so you can start pulling data in minutes.

1. Create an account

Sign up at Vantyx Crawl to get your API key. The dashboard gives you access to usage metrics, project settings, and documentation references all in one place.

2. Set up authentication

Vantyx Crawl uses a simple API key pattern. Include the key as an Authorization: Bearer <key> header on every request:

Scrape endpoint: /v1/scrape
Crawl endpoint: /v1/crawl
Search endpoint: /v1/search
Extract endpoint: /v1/extract

3. Make your first request

The scrape endpoint is the fastest way to test the service. It returns clean HTML, markdown, or JSON depending on what you need.

Scrape a single page: Send a POST with the URL and desired output format
Crawl a site: Provide a seed URL and depth to follow links automatically
Search the web: Query keywords and get structured results
Extract data: Define a JSON schema and let Vantyx Crawl fill it in from any page

4. Choose your output format

Vantyx Crawl supports multiple response formats so you can pick what fits your pipeline best.

Markdown: Clean, readable text with headings and lists — great for LLM prompts
HTML: Raw markup when you need the original structure intact
JSON: Structured data ready to consume programmatically
Extracted schema: Custom fields defined by your own JSON schema for consistent extraction across pages

5. Handle pagination and rate limits

When crawling large sites, Vantyx Crawl returns paginated results. Use the cursor parameter to fetch subsequent pages without losing progress.

Crawl depth: Control how many link levels deep you want to follow
Max pages: Set an upper bound on total pages returned per crawl job
Rate limits: Check your dashboard for current tier limits and upgrade if needed

6. Integrate into your workflow

Vantyx Crawl works well with popular tools and frameworks.

LangChain: Use the scrape endpoint as a document loader for RAG pipelines
Pandas AI: Feed extracted tables directly into data analysis workflows
Custom scripts: Any HTTP client can call the API — Python, Node.js, curl, or Postman all work out of the box

7. Monitor and optimize

Your dashboard shows real-time usage stats, error rates, and response times. Use these metrics to:

Tune crawl depth: Shallow crawls are faster; deep crawls capture more pages but cost more credits
Optimize extraction schemas: Fewer fields mean lower latency and cheaper requests
Schedule jobs: Run large crawls during off-peak hours to avoid rate limit pressure

Common use cases

Vantyx Crawl powers a wide range of applications.

Market research: Scrape competitor pricing, product pages, and reviews at scale
Lead generation: Extract contact information from directories and business listings
Content aggregation: Build news aggregators or blog monitors with automated crawling
AI training data: Collect clean, structured datasets for fine-tuning models
Price monitoring: Track e-commerce prices across multiple retailers in real time

Troubleshooting tips

If you run into issues during setup, check these common causes.

401 errors: Verify your API key is correct and hasn't expired. Regenerate it from the dashboard if needed
Empty responses: Some pages block automated requests. Try adjusting headers or using the extract endpoint instead of raw scrape
Crawl timeouts: Large sites may need more time. Increase your timeout value or reduce crawl depth
Schma validation errors: Ensure your JSON schema uses valid field types and required flags match the data you expect

Next steps

Now that you've set up Vantyx Crawl, explore the full API reference for detailed parameter descriptions. You can also visit Vantyx Crawl to learn about pricing tiers and advanced features like scheduled crawls and team collaboration.

FAQ

How do I get my Vantyx Crawl API key?

Sign up at Vantyx Crawl and navigate to the dashboard. Your API key is displayed in the settings panel where you can also regenerate it if needed.

What output formats does Vantyx Crawl support?

Vantyx Crawl supports markdown, HTML, JSON, and custom extracted schemas. Choose the format that best fits your downstream processing pipeline.

How do I crawl a website with multiple pages?

Use the /v1/crawl endpoint with a seed URL, set your desired depth level, and specify max pages to control how many links are followed. Results come back paginated using cursor-based pagination.

Can I use Vantyx Crawl with LangChain?

Yes. The scrape endpoint works as a document loader in LangChain pipelines, making it easy to feed scraped content into RAG applications or other AI workflows.