Skip to content

WebCrawlerLimits

Source code in docs_src/capo_bedrock_agent/types/web_crawler_limits.py 
 6
 7
 8
 9
10
class WebCrawlerLimits(TypedDict, closed=True):
    rate_limit: NotRequired["int"]
    """<p>The max rate at which pages are crawled, up to 300 per minute per host.</p>"""
    max_pages: NotRequired["int"]
    """<p> The max number of web pages crawled from your source URLs, up to 25,000 pages. If the web pages exceed this limit, the data source sync will fail and no web pages will be ingested. </p>"""

max_pages instance-attribute 

max_pages: NotRequired[int]

The max number of web pages crawled from your source URLs, up to 25,000 pages. If the web pages exceed this limit, the data source sync will fail and no web pages will be ingested.

rate_limit instance-attribute 

rate_limit: NotRequired[int]

The max rate at which pages are crawled, up to 300 per minute per host.