Skip to content

WebCrawlerConfiguration

Source code in docs_src/capo_qconnect/types/web_crawler_configuration.py 
16
17
18
19
20
21
22
23
24
25
26
27
28
class WebCrawlerConfiguration(TypedDict, closed=True):
    url_configuration: "capo_qconnect.types.url_configuration.UrlConfiguration"
    """<p>The configuration of the URL/URLs for the web content that you want to crawl. You should be authorized to crawl the URLs.</p>"""
    crawler_limits: NotRequired[
        "capo_qconnect.types.web_crawler_limits.WebCrawlerLimits"
    ]
    """<p>The configuration of crawl limits for the web URLs.</p>"""
    inclusion_filters: NotRequired["capo_qconnect.types.url_filter_list.UrlFilterList"]
    """<p>A list of one or more inclusion regular expression patterns to include certain URLs. If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.</p>"""
    exclusion_filters: NotRequired["capo_qconnect.types.url_filter_list.UrlFilterList"]
    """<p>A list of one or more exclusion regular expression patterns to exclude certain URLs. If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.</p>"""
    scope: NotRequired["capo_qconnect.types.web_scope_type.WebScopeType"]
    """<p>The scope of what is crawled for your URLs. You can choose to crawl only web pages that belong to the same host or primary domain. For example, only web pages that contain the seed URL <code>https://docs.aws.amazon.com/bedrock/latest/userguide/</code> and no other domains. You can choose to include sub domains in addition to the host or primary domain. For example, web pages that contain <code>aws.amazon.com</code> can also include sub domain <code>docs.aws.amazon.com</code>.</p>"""

crawler_limits instance-attribute 

crawler_limits: NotRequired[WebCrawlerLimits]

The configuration of crawl limits for the web URLs.

exclusion_filters instance-attribute 

exclusion_filters: NotRequired[UrlFilterList]

A list of one or more exclusion regular expression patterns to exclude certain URLs. If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.

inclusion_filters instance-attribute 

inclusion_filters: NotRequired[UrlFilterList]

A list of one or more inclusion regular expression patterns to include certain URLs. If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.

scope instance-attribute 

scope: NotRequired[WebScopeType]

The scope of what is crawled for your URLs. You can choose to crawl only web pages that belong to the same host or primary domain. For example, only web pages that contain the seed URL https://docs.aws.amazon.com/bedrock/latest/userguide/ and no other domains. You can choose to include sub domains in addition to the host or primary domain. For example, web pages that contain aws.amazon.com can also include sub domain docs.aws.amazon.com.

url_configuration instance-attribute 

url_configuration: UrlConfiguration

The configuration of the URL/URLs for the web content that you want to crawl. You should be authorized to crawl the URLs.