• Splits a list of documents into sublists based on a maximum token limit.

    Parameters

    • docs: Document<Record<string, any>>[]

      The list of documents to be split.

    • lengthFunc: ((...args: any[]) => any)

      A function that calculates the number of tokens in a list of documents.

        • (...args): any
        • Parameters

          • Rest...args: any[]

          Returns any

    • tokenMax: number

      The maximum number of tokens allowed in a sublist.

    Returns Document[][]

    • A list of document sublists, each sublist contains documents whose total number of tokens does not exceed the tokenMax.
    • Throws an error if a single document has more tokens than the tokenMax.