Class EmbeddingsFilter

Class that represents a document compressor that uses embeddings to drop documents unrelated to the query.

Example

const embeddingsFilter = new EmbeddingsFilter({
  embeddings: new OpenAIEmbeddings(),
  similarityThreshold: 0.8,
  k: 5,
});
const retrievedDocs = await embeddingsFilter.filterDocuments(
  getDocuments(),
  "What did the speaker say about Justice Breyer in the 2022 State of the Union?",
);
console.log({ retrievedDocs });

Hierarchy (view full)

Constructors

constructor

Properties

embeddings k? similarityFn similarityThreshold?

Methods

compressDocuments isBaseDocumentCompressor

Constructors

constructor

Properties

embeddings

embeddings: EmbeddingsInterface

Embeddings to use for embedding document contents and queries.

Optionalk

k?: number = 20

The number of relevant documents to return. Can be explicitly set to undefined, in which case similarity_threshold` must be specified. Defaults to 20

similarityFn

similarityFn: ((X: number[][], Y: number[][]) => number[][]) = cosineSimilarity

Similarity function for comparing documents.

Type declaration

    • (X, Y): number[][]

    • This function calculates the row-wise cosine similarity between two matrices with the same number of columns.

      Parameters

      • X: number[][]

        The first matrix.

      • Y: number[][]

        The second matrix.

      Returns number[][]

      A matrix where each row represents the cosine similarity values between the corresponding rows of X and Y.

      Throws

      If the number of columns in X and Y are not the same.

OptionalsimilarityThreshold

similarityThreshold?: number

Threshold for determining when two documents are similar enough to be considered redundant. Must be specified if k is not set.

Methods

compressDocuments

  • compressDocuments(documents, query): Promise<DocumentInterface<Record<string, any>>[]>

  • Abstract method that must be implemented by any class that extends BaseDocumentCompressor. This method takes an array of Document objects and a query string as parameters and returns a Promise that resolves with an array of compressed Document objects.

    Parameters

    • documents: DocumentInterface<Record<string, any>>[]

      An array of Document objects to be compressed.

    • query: string

      A query string.

    Returns Promise<DocumentInterface<Record<string, any>>[]>

    A Promise that resolves with an array of compressed Document objects.

StaticisBaseDocumentCompressor

Settings

Member Visibility

On This Page

Constructors
Properties
Methods