Class SitemapLoader

Deprecated

  • Import from "@langchain/community/document_loaders/web/sitemap" instead. This entrypoint will be removed in 0.3.0.

Hierarchy (view full)

Implements

Constructors

constructor

Properties

allowUrlPatterns caller chunkSize selector? textDecoder? timeout webPath

Methods

load loadAndSplit parseSitemap scrape imports scrapeAll

Constructors

constructor

Properties

allowUrlPatterns

allowUrlPatterns: undefined | (string | RegExp)[]

caller

caller: AsyncCaller

chunkSize

chunkSize: number

The size to chunk the sitemap URLs into for scraping.

Default

{300}

Optionalselector

selector?: SelectorType

The selector to use to extract the text from the document. Defaults to "body".

OptionaltextDecoder

textDecoder?: TextDecoder

The text decoder to use to decode the response. Defaults to UTF-8.

timeout

timeout: number

The timeout in milliseconds for the fetch request. Defaults to 10s.

webPath

webPath: string

Methods

load

  • load(): Promise<Document<Record<string, any>>[]>

  • Extracts the text content from the loaded document using the selector and creates a Document instance with the extracted text and metadata. It returns an array of Document instances.

    Returns Promise<Document<Record<string, any>>[]>

    A Promise that resolves to an array of Document instances.

loadAndSplit

  • loadAndSplit(splitter?): Promise<Document<Record<string, any>>[]>

  • Parameters

    • Optionalsplitter: BaseDocumentTransformer<DocumentInterface<Record<string, any>>[], DocumentInterface<Record<string, any>>[]>

    Returns Promise<Document<Record<string, any>>[]>

    A Promise that resolves with an array of Document instances, each split according to the provided TextSplitter.

    Deprecated

    Use this.load() and splitter.splitDocuments() individually. Loads the documents and splits them using a specified text splitter.

parseSitemap

scrape

Staticimports

  • imports(): Promise<{
        load: ((content:
            | string
            | Buffer
            | AnyNode
            | AnyNode[], options?: null | CheerioOptions, isDocument?: boolean) => CheerioAPI);
    }>

  • A static method that dynamically imports the Cheerio library and returns the load function. If the import fails, it throws an error.

    Returns Promise<{
        load: ((content:
            | string
            | Buffer
            | AnyNode
            | AnyNode[], options?: null | CheerioOptions, isDocument?: boolean) => CheerioAPI);
    }>

    A Promise that resolves to an object containing the load function from the Cheerio library.

StaticscrapeAll

  • scrapeAll(urls, caller, timeout, textDecoder?, options?): Promise<CheerioAPI[]>

  • Fetches web documents from the given array of URLs and loads them using Cheerio. It returns an array of CheerioAPI instances.

    Parameters

    • urls: string[]

      An array of URLs to fetch and load.

    • caller: AsyncCaller
    • timeout: undefined | number
    • OptionaltextDecoder: TextDecoder
    • Optionaloptions: CheerioOptions

    Returns Promise<CheerioAPI[]>

    A Promise that resolves to an array of CheerioAPI instances.

Settings

Member Visibility

On This Page

Constructors
Properties
Methods