Interface SageMakerEndpointInput

The SageMakerEndpointInput interface defines the input parameters for the SageMakerEndpoint class, which includes the endpoint name, client options for the SageMaker client, the content handler, and optional keyword arguments for the model and the endpoint.

interface SageMakerEndpointInput {
    cache?: boolean | BaseCache<Generation[]>;
    callbackManager?: CallbackManager;
    callbacks?: Callbacks;
    clientOptions: SageMakerRuntimeClientConfig;
    concurrency?: number;
    contentHandler: SageMakerLLMContentHandler;
    endpointKwargs?: Record<string, unknown>;
    endpointName: string;
    maxConcurrency?: number;
    maxRetries?: number;
    metadata?: Record<string, unknown>;
    modelKwargs?: Record<string, unknown>;
    onFailedAttempt?: FailedAttemptHandler;
    streaming?: boolean;
    tags?: string[];
    verbose?: boolean;
}

Hierarchy

  • BaseLLMParams
    • SageMakerEndpointInput

Properties

cache? callbackManager? callbacks? clientOptions concurrency? contentHandler endpointKwargs? endpointName maxConcurrency? maxRetries? metadata? modelKwargs? onFailedAttempt? streaming? tags? verbose?

Properties

Optionalcache

cache?: boolean | BaseCache<Generation[]>

OptionalcallbackManager

callbackManager?: CallbackManager

Deprecated

Use callbacks instead

Optionalcallbacks

callbacks?: Callbacks

clientOptions

clientOptions: SageMakerRuntimeClientConfig

Options passed to the SageMaker client.

Optionalconcurrency

concurrency?: number

Deprecated

Use maxConcurrency instead

contentHandler

contentHandler: SageMakerLLMContentHandler

The content handler class that provides an input and output transform functions to handle formats between LLM and the endpoint.

OptionalendpointKwargs

endpointKwargs?: Record<string, unknown>

Optional attributes passed to the InvokeEndpointCommand

endpointName

endpointName: string

The name of the endpoint from the deployed SageMaker model. Must be unique within an AWS Region.

OptionalmaxConcurrency

maxConcurrency?: number

The maximum number of concurrent calls that can be made. Defaults to Infinity, which means no limit.

OptionalmaxRetries

maxRetries?: number

The maximum number of retries that can be made for a single call, with an exponential backoff between each attempt. Defaults to 6.

Optionalmetadata

metadata?: Record<string, unknown>

OptionalmodelKwargs

modelKwargs?: Record<string, unknown>

Key word arguments to pass to the model.

OptionalonFailedAttempt

onFailedAttempt?: FailedAttemptHandler

Custom handler to handle failed attempts. Takes the originally thrown error object as input, and should itself throw an error if the input error is not retryable.

Optionalstreaming

streaming?: boolean

Optionaltags

tags?: string[]

Optionalverbose

verbose?: boolean

Settings

Member Visibility

On This Page

Properties