spider-client

Python SDK for Spider Cloud API

MIT 111 个版本
Spider <jeff@spider.cloud>
主页
Homepage
安装
pip install spider-client
poetry add spider-client
pipenv install spider-client
conda install spider-client
描述

Spider Cloud Python SDK

The Spider Cloud Python SDK offers a toolkit for straightforward website scraping, crawling at scale, and other utilities like extracting links and taking screenshots, enabling you to collect data formatted for compatibility with language models (LLMs). It features a user-friendly interface for seamless integration with the Spider Cloud API.

Installation

To install the Spider Cloud Python SDK, you can use pip:

pip install spider_client

Usage

  1. Get an API key from spider.cloud
  2. Set the API key as an environment variable named SPIDER_API_KEY or pass it as a parameter to the Spider class.

Here's an example of how to use the SDK:

from spider import Spider

# Initialize the Spider with your API key
app = Spider(api_key='your_api_key')

# Scrape a single URL
url = 'https://spider.cloud'
scraped_data = app.scrape_url(url)

# Crawl a website
crawler_params = {
    'limit': 1,
    'proxy_enabled': True,
    'metadata': False,
    'request': 'http'
}
crawl_result = app.crawl_url(url, params=crawler_params)

Scraping a URL

To scrape data from a single URL:

url = 'https://example.com'
scraped_data = app.scrape_url(url)

Crawling a Website

To automate crawling a website:

url = 'https://example.com'
crawl_params = {
    'limit': 200,
    'request': 'smart_mode'
}
crawl_result = app.crawl_url(url, params=crawl_params)

Crawl Streaming

Stream crawl the website in chunks to scale.

    def handle_json(json_obj: dict) -> None:
        assert json_obj["url"] is not None

    url = 'https://example.com'
    crawl_params = {
        'limit': 200,
    }
    response = app.crawl_url(
        url,
        params=params,
        stream=True,
        callback=handle_json,
    )

Perform a search for websites to crawl or gather search results:

query = 'a sports website'
crawl_params = {
    'request': 'smart_mode',
    'search_limit': 5,
    'limit': 5,
    'fetch_page_content': True
}
crawl_result = app.search(query, params=crawl_params)

Extract all links from a specified URL:

url = 'https://example.com'
links = app.links(url)

Transform

Transform HTML to markdown or text lightning fast:

data = [ { 'html': '<html><body><h1>Hello world</h1></body></html>' } ]
params = {
    'readability': False,
    'return_format': 'markdown',
}
result = app.transform(data, params=params)

Taking Screenshots of a URL(s)

Capture a screenshot of a given URL:

url = 'https://example.com'
screenshot = app.screenshot(url)

Checking Available Credits

You can check the remaining credits on your account:

credits = app.get_credits()

Unblocker

Access blocked or protected content with anti-bot bypass:

url = 'https://protected-site.com'
result = app.unblocker(url)

Unblocker with AI Extraction

Unblock and extract structured data using AI:

url = 'https://protected-site.com/products'
result = app.unblocker(url, params={
    'custom_prompt': 'Extract all product names and prices as JSON'
})
# Extracted data is available in result[0]['metadata']['extracted_data']

Unblocker with JSON Schema Extraction

Use JSON Schema for structured, validated extraction output:

url = 'https://protected-site.com/products'
result = app.unblocker(url, params={
    'extraction_schema': {
        'name': 'products',
        'description': 'Product listing extraction',
        'schema': '''{
            "type": "object",
            "properties": {
                "products": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "name": {"type": "string"},
                            "price": {"type": "number"}
                        },
                        "required": ["name", "price"]
                    }
                }
            }
        }''',
        'strict': True
    }
})
# Extracted data conforms to the schema in result[0]['metadata']['extracted_data']

AI Studio Methods

AI Studio methods require an active AI Studio subscription. See spider.cloud/ai/pricing for plans.

AI Crawl

AI-guided crawling using natural language prompts:

result = app.ai_crawl(
    url='https://example.com',
    prompt='Find all blog posts and extract titles and summaries'
)

AI Scrape

AI-guided scraping using natural language prompts:

result = app.ai_scrape(
    url='https://example.com/products',
    prompt='Extract all product names, prices, and descriptions'
)

AI-enhanced web search using natural language:

result = app.ai_search(prompt='Find the best Python web scraping libraries')

AI Browser

AI-guided browser automation:

result = app.ai_browser(
    url='https://example.com/login',
    prompt='Click the sign in button and fill the email field with test@example.com'
)

AI-guided link extraction and filtering:

result = app.ai_links(
    url='https://example.com',
    prompt='Find all links to product pages and documentation'
)

Streaming

If you need to stream the request use the third param:

url = 'https://example.com'

crawler_params = {
    'limit': 1,
    'proxy_enabled': True,
    'metadata': False,
    'request': 'http'
}

links = app.links(url, crawler_params, True)

Content-Type

The following Content-type headers are supported using the fourth param:

  1. application/json
  2. text/csv
  3. application/xml
  4. application/jsonl
url = 'https://example.com'

crawler_params = {
    'limit': 1,
    'proxy_enabled': True,
    'metadata': False,
    'request': 'http'
}

# stream json lines back to the client
links = app.crawl(url, crawler_params, True, "application/jsonl")

Error Handling

The SDK handles errors returned by the Spider Cloud API and raises appropriate exceptions. If an error occurs during a request, an exception will be raised with a descriptive error message.

Contributing

Contributions to the Spider Cloud Python SDK are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on the GitHub repository.

License

The Spider Cloud Python SDK is open-source and released under the MIT License.

分类
Development Status :: 5 - Production/Stable Intended Audience :: Developers Intended Audience :: Information Technology Operating System :: OS Independent Topic :: Internet Topic :: Internet :: WWW/HTTP Topic :: Internet :: WWW/HTTP :: Indexing/Search Topic :: Software Development :: Libraries :: Python Modules
版本列表
0.1.88 2026-03-20
0.1.87 2026-03-02
0.1.85 2026-01-21
0.1.84 2026-01-20
0.1.82 2025-11-22
0.1.81 2025-11-22
0.1.80 2025-11-06
0.1.79 2025-11-01
0.1.77 2025-08-29
0.1.75 2025-08-21
0.1.74 2025-08-20
0.1.73 2025-08-20
0.1.72 2025-08-14
0.1.71 2025-08-03
0.1.70 2025-08-03
0.1.69 2025-07-20
0.1.68 2025-07-20
0.1.67 2025-07-20
0.1.65 2025-07-17
0.1.63 2025-07-14
0.1.62 2025-07-14
0.1.61 2025-07-14
0.1.60 2025-07-14
0.1.59 2025-07-14
0.1.58 2025-07-14
0.1.57 2025-07-14
0.1.56 2025-07-12
0.1.55 2025-07-12
0.1.54 2025-07-12
0.1.53 2025-07-10
0.1.52 2025-07-10
0.1.38 2025-06-19
0.1.37 2025-06-08
0.1.36 2025-05-18
0.1.35 2025-05-18
0.1.34 2025-05-05
0.1.33 2025-05-05
0.1.32 2025-03-26
0.1.31 2025-03-25
0.1.30 2025-03-25
0.1.28 2025-03-12
0.1.27 2025-03-10
0.1.26 2025-01-21
0.1.25 2024-12-12
0.1.24 2024-12-05
0.1.23 2024-11-07
0.1.22 2024-11-04
0.0.72 2024-10-11
0.0.71 2024-10-09
0.0.70 2024-09-14
0.0.69 2024-08-27
0.0.68 2024-08-20
0.0.67 2024-08-09
0.0.66 2024-08-08
0.0.65 2024-08-03
0.0.64 2024-07-24
0.0.63 2024-07-24
0.0.62 2024-07-23
0.0.61 2024-07-23
0.0.60 2024-07-23
0.0.59 2024-07-21
0.0.58 2024-07-14
0.0.57 2024-07-14
0.0.56 2024-07-13
0.0.55 2024-07-13
0.0.54 2024-07-13
0.0.53 2024-07-09
0.0.52 2024-07-07
0.0.51 2024-07-06
0.0.50 2024-07-06
0.0.49 2024-07-06
0.0.48 2024-07-04
0.0.47 2024-07-04
0.0.46 2024-07-04
0.0.45 2024-07-04
0.0.44 2024-07-04
0.0.43 2024-07-02
0.0.42 2024-07-02
0.0.41 2024-07-02
0.0.40 2024-07-02
0.0.39 2024-07-01
0.0.38 2024-07-01
0.0.37 2024-07-01
0.0.36 2024-06-30
0.0.35 2024-06-30
0.0.34 2024-06-30
0.0.33 2024-06-30
0.0.32 2024-06-30
0.0.31 2024-06-30
0.0.30 2024-06-30
0.0.29 2024-06-30
0.0.28 2024-06-30
0.0.27 2024-06-18
0.0.26 2024-06-18
0.0.25 2024-06-15
0.0.24 2024-06-07
0.0.23 2024-06-07
0.0.22 2024-05-27
0.0.21 2024-05-13
0.0.20 2024-05-13
0.0.11 2024-04-29
0.0.10 2024-04-26
0.0.9 2024-04-25
0.0.8 2024-04-25
0.0.7 2024-04-25
0.0.6 2024-04-23
0.0.5 2024-04-23
0.0.4 2024-04-23
0.0.3 2024-04-22
0.0.2 2024-04-22
0.0.1 2024-04-22