Module org.nasdanika.ai

Package org.nasdanika.ai

Class TextFloatVectorChunkingEmbeddingModel<T>

java.lang.Object

org.nasdanika.ai.TextFloatVectorChunkingEmbeddingModel<T>

Type Parameters:

T - a container of tokens, e.g. int[] or char[] or List<Integer>

All Implemented Interfaces:

Coordinates, EmbeddingGenerator<String,List<List<Float>>>, EmbeddingModel<String,List<List<Float>>>, FloatVectorEmbeddingGenerator<String>, Model, TextEmbeddingGenerator<List<List<Float>>>, TextFloatVectorEmbeddingModel, VectorEmbeddingGenerator<String,Float>, Composable<EmbeddingGenerator<String,List<List<Float>>>>

Direct Known Subclasses:

TextFloatVectorCharChunkingEmbeddings, TextFloatVectorEncodingChunkingEmbeddingModel

public abstract class TextFloatVectorChunkingEmbeddingModel<T>
extends Object
implements TextFloatVectorEmbeddingModel

Nested Class Summary

Nested classes/interfaces inherited from interface org.nasdanika.ai.EmbeddingGenerator

EmbeddingGenerator.Requirement

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	TextFloatVectorChunkingEmbeddingModel(TextFloatVectorEmbeddingModel target, int chunkSize, int overlap)

Method Summary

Modifier and Type	Method	Description
List<String>	chunk(String input)
protected abstract String	decode(T tokens)	Decodes a string from an array of tokens
protected abstract T	encode(String input)	Encodes a string into tokens
List<List<Float>>	generate(String input)
reactor.core.publisher.Mono<List<List<Float>>>	generateAsync(String input)
int	getDimensions()
int	getMaxInputTokens()
String	getName()	Model name
String	getProvider()	Provider name - OpenAI, Ollama, ...
String	getVersion()
boolean	isTooLong(String input)
protected abstract int	size(T tokens)
protected abstract T	slice(T tokens, int offset, int length)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nasdanika.ai.EmbeddingGenerator

adapt, compose, compose, composeAsync, generate, generateAsync, then

Constructor Details

TextFloatVectorChunkingEmbeddingModel

protected TextFloatVectorChunkingEmbeddingModel(TextFloatVectorEmbeddingModel target,
 int chunkSize,
 int overlap)

Parameters:

target -

chunkSize - Chunk size, if non-positive, then target max input tokens is used as chunk size

overlap -

Method Details

getProvider

public String getProvider()

Description copied from interface: Coordinates

Provider name - OpenAI, Ollama, ...

Specified by:

getProvider in interface Coordinates

Returns:

getName

public String getName()

Description copied from interface: Coordinates

Model name

Specified by:

getName in interface Coordinates

Returns:

getVersion

public String getVersion()

Specified by:

getVersion in interface Coordinates

getMaxInputTokens

public int getMaxInputTokens()

Specified by:

getMaxInputTokens in interface Model

isTooLong

public boolean isTooLong(String input)

Specified by:

isTooLong in interface TextFloatVectorEmbeddingModel

Parameters:

input -

Returns:

true if the input is too long for a given model

getDimensions

public int getDimensions()

Specified by:

getDimensions in interface VectorEmbeddingGenerator<String,Float>

Returns:

number of dimensions, -1 if unknown

chunk

public List<String> chunk(String input)

generateAsync

public reactor.core.publisher.Mono<List<List<Float>>> generateAsync(String input)

Specified by:

generateAsync in interface EmbeddingGenerator<String,List<List<Float>>>

generate

public List<List<Float>> generate(String input)

Specified by:

generate in interface EmbeddingGenerator<String,List<List<Float>>>

encode

protected abstract T encode(String input)

Encodes a string into tokens

Parameters:

input -

Returns:

decode

protected abstract String decode(T tokens)

Decodes a string from an array of tokens

Parameters:

tokens -

Returns:

size

protected abstract int size(T tokens)

slice

protected abstract T slice(T tokens,
 int offset,
 int length)