Module org.nasdanika.ai

Package org.nasdanika.ai

Class ChunkingEmbeddings<T>

java.lang.Object

org.nasdanika.ai.ChunkingEmbeddings<T>

Type Parameters:

T - a container of tokens, e.g. int[] or char[] or List<Integer>

All Implemented Interfaces:

Coordinates, Embeddings, Model

Direct Known Subclasses:

CharChunkingEmbeddings, EncodingChunkingEmbeddings

public abstract class ChunkingEmbeddings<T>
extends Object
implements Embeddings

Nested Class Summary

Nested classes/interfaces inherited from interface org.nasdanika.ai.Embeddings

Embeddings.Requirement

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	ChunkingEmbeddings(Embeddings target, int chunkSize, int overlap)

Method Summary

Modifier and Type	Method	Description
List<String>	chunk(String input)
protected abstract String	decode(T tokens)	Decodes a string from an array of tokens
protected abstract T	encode(String input)	Encodes a string into tokens
List<List<Float>>	generate(String input)	Generates embeddings for a single string
reactor.core.publisher.Mono<List<List<Float>>>	generateAsync(String input)	Asynchronously generates embeddings for a single string
int	getDimensions()
int	getMaxInputTokens()
String	getName()	Model name
String	getProvider()	Provider name - OpenAI, Ollama, ...
String	getVersion()
boolean	isTooLong(String input)
protected abstract int	size(T tokens)
protected abstract T	slice(T tokens, int offset, int length)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nasdanika.ai.Embeddings

generate, generateAsync

Constructor Details

ChunkingEmbeddings

protected ChunkingEmbeddings(Embeddings target,
 int chunkSize,
 int overlap)

Parameters:

target -

chunkSize - Chunk size, if non-positive, then target max input tokens is used as chunk size

overlap -

Method Details

getProvider

public String getProvider()

Description copied from interface: Coordinates

Provider name - OpenAI, Ollama, ...

Specified by:

getProvider in interface Coordinates

Returns:

getName

public String getName()

Description copied from interface: Coordinates

Model name

Specified by:

getName in interface Coordinates

Returns:

getVersion

public String getVersion()

Specified by:

getVersion in interface Coordinates

getMaxInputTokens

public int getMaxInputTokens()

Specified by:

getMaxInputTokens in interface Model

isTooLong

public boolean isTooLong(String input)

Specified by:

isTooLong in interface Embeddings

Returns:

true if the input is too long for a given model

getDimensions

public int getDimensions()

Specified by:

getDimensions in interface Embeddings

Returns:

number of dimentions

chunk

public List<String> chunk(String input)

generateAsync

public reactor.core.publisher.Mono<List<List<Float>>> generateAsync(String input)

Description copied from interface: Embeddings

Asynchronously generates embeddings for a single string

Specified by:

generateAsync in interface Embeddings

Returns:

generate

public List<List<Float>> generate(String input)

Description copied from interface: Embeddings

Generates embeddings for a single string

Specified by:

generate in interface Embeddings

Returns:

encode

protected abstract T encode(String input)

Encodes a string into tokens

Parameters:

input -

Returns:

decode

protected abstract String decode(T tokens)

Decodes a string from an array of tokens

Parameters:

tokens -

Returns:

size

protected abstract int size(T tokens)

slice

protected abstract T slice(T tokens,
 int offset,
 int length)