Class SitemapLoader

Interface representing the parameters for initializing a SitemapLoader. SitemapLoaderParams

Hierarchy (view full)

Implements

Constructors

constructor

Properties

allowUrlPatterns caller chunkSize headers? selector? textDecoder? timeout webPath

Methods

load loadAndSplit parseSitemap scrape imports scrapeAll

Constructors

constructor

Properties

allowUrlPatterns

allowUrlPatterns: undefined | (string | RegExp)[]

caller

caller: AsyncCaller

chunkSize

chunkSize: number

The size to chunk the sitemap URLs into for scraping.

Default

{300}

Optionalheaders

headers?: HeadersInit

The headers to use in the fetch request.

Optionalselector

selector?: SelectorType

The selector to use to extract the text from the document. Defaults to "body".

OptionaltextDecoder

textDecoder?: TextDecoder

The text decoder to use to decode the response. Defaults to UTF-8.

timeout

timeout: number

The timeout in milliseconds for the fetch request. Defaults to 10s.

webPath

webPath: string

Methods

load

  • load(): Promise<Document<Record<string, any>>[]>

  • Extracts the text content from the loaded document using the selector and creates a Document instance with the extracted text and metadata. It returns an array of Document instances.

    Returns Promise<Document<Record<string, any>>[]>

    A Promise that resolves to an array of Document instances.

loadAndSplit

  • loadAndSplit(splitter?): Promise<Document<Record<string, any>>[]>

  • Parameters

    • Optionalsplitter: BaseDocumentTransformer<DocumentInterface<Record<string, any>>[], DocumentInterface<Record<string, any>>[]>

    Returns Promise<Document<Record<string, any>>[]>

    A Promise that resolves with an array of Document instances, each split according to the provided TextSplitter.

    Deprecated

    Use this.load() and splitter.splitDocuments() individually. Loads the documents and splits them using a specified text splitter.

parseSitemap

scrape

Staticimports

  • imports(): Promise<{
        load: ((content:
            | string
            | Buffer
            | AnyNode
            | AnyNode[], options?: null | CheerioOptions, isDocument?: boolean) => CheerioAPI);
    }>

  • A static method that dynamically imports the Cheerio library and returns the load function. If the import fails, it throws an error.

    Returns Promise<{
        load: ((content:
            | string
            | Buffer
            | AnyNode
            | AnyNode[], options?: null | CheerioOptions, isDocument?: boolean) => CheerioAPI);
    }>

    A Promise that resolves to an object containing the load function from the Cheerio library.

StaticscrapeAll

  • scrapeAll(urls, caller, timeout, textDecoder?, options?): Promise<CheerioAPI[]>

  • Fetches web documents from the given array of URLs and loads them using Cheerio. It returns an array of CheerioAPI instances.

    Parameters

    • urls: string[]

      An array of URLs to fetch and load.

    • caller: AsyncCaller
    • timeout: undefined | number
    • OptionaltextDecoder: TextDecoder
    • Optionaloptions: CheerioOptions & {
          headers?: HeadersInit;
      }

    Returns Promise<CheerioAPI[]>

    A Promise that resolves to an array of CheerioAPI instances.

Settings

Member Visibility

On This Page

Constructors
Properties
Methods