CaptureKit

Extract Content as Markdown

Convert any webpage to clean Markdown text using the CaptureKit content API — strip ads and boilerplate to get the pure article or documentation content.

Overview

The CaptureKit content endpoint fetches a webpage, strips navigation, ads, and layout chrome, then returns the main body as clean Markdown. Use it for building RAG datasets, indexing documentation, or archiving articles.

Prerequisites

  • A CaptureKit API key — get one at app.capturekit.dev
  • Install dependencies for your language:
pip install requests

Steps

Fetch the page content

Call GET /v1/content with the url parameter.

import requests

API_KEY  = "YOUR_API_KEY"
response = requests.get(
    "https://api.capturekit.dev/v1/content",
    headers={"x-api-key": API_KEY},
    params={"url": "https://stripe.com/docs/payments"},
)
data = response.json()
print(data)

Access the Markdown content

The response includes markdown, title, description, author, published_at, and word_count.

print(f"Title      : {data.get('title')}")
print(f"Author     : {data.get('author', 'N/A')}")
print(f"Word count : {data.get('word_count')} words")
print()
print("--- Markdown preview ---")
markdown = data.get("markdown", "")
print(markdown[:1000])

Crawl and archive a list of documentation pages

Fetch and save multiple pages as Markdown files for offline search or RAG ingestion.

import os, time, requests

API_KEY = "YOUR_API_KEY"
os.makedirs("docs_archive", exist_ok=True)

PAGES = [
    "https://stripe.com/docs/payments",
    "https://stripe.com/docs/billing",
    "https://stripe.com/docs/connect",
]

for page_url in PAGES:
    r    = requests.get("https://api.capturekit.dev/v1/content",
        headers={"x-api-key": API_KEY}, params={"url": page_url})
    data = r.json()
    slug = page_url.rstrip("/").split("/")[-1]
    path = f"docs_archive/{slug}.md"

    with open(path, "w") as f:
        f.write(f"# {data.get('title', slug)}\n\n")
        f.write(data.get("markdown", ""))

    print(f"Saved: {path} ({data.get('word_count', 0)} words)")
    time.sleep(1)

print("Done!")

The extracted Markdown is ideal as context chunks for a RAG (Retrieval-Augmented Generation) pipeline. Chunk by headings and embed with your preferred vector store for semantic search over any website's documentation.

On this page