Module org.nasdanika.ai

Package org.nasdanika.ai

Class EncodingChunkingEmbeddings

java.lang.Object

org.nasdanika.ai.ChunkingEmbeddings<com.knuddels.jtokkit.api.IntArrayList>

org.nasdanika.ai.EncodingChunkingEmbeddings

All Implemented Interfaces:

Coordinates, Embeddings, Model

public class EncodingChunkingEmbeddings
extends ChunkingEmbeddings<com.knuddels.jtokkit.api.IntArrayList>

Nested Class Summary

Nested classes/interfaces inherited from interface org.nasdanika.ai.Embeddings

Embeddings.Requirement

Constructor Summary

Constructors

Constructor	Description
EncodingChunkingEmbeddings(Embeddings target, int chunkSize, int overlap, com.knuddels.jtokkit.api.Encoding encoding)
EncodingChunkingEmbeddings(Embeddings target, int chunkSize, int overlap, com.knuddels.jtokkit.api.EncodingType encodingType)

Method Summary

Modifier and Type	Method	Description
protected String	decode(com.knuddels.jtokkit.api.IntArrayList tokens)	Decodes a string from an array of tokens
protected com.knuddels.jtokkit.api.IntArrayList	encode(String input)	Encodes a string into tokens
protected int	size(com.knuddels.jtokkit.api.IntArrayList tokens)
protected com.knuddels.jtokkit.api.IntArrayList	slice(com.knuddels.jtokkit.api.IntArrayList tokens, int offset, int length)

Methods inherited from class org.nasdanika.ai.ChunkingEmbeddings

chunk, generate, generateAsync, getDimensions, getMaxInputTokens, getName, getProvider, getVersion, isTooLong

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nasdanika.ai.Embeddings

generate, generateAsync

Constructor Details

EncodingChunkingEmbeddings

public EncodingChunkingEmbeddings(Embeddings target,
 int chunkSize,
 int overlap,
 com.knuddels.jtokkit.api.Encoding encoding)

EncodingChunkingEmbeddings

public EncodingChunkingEmbeddings(Embeddings target,
 int chunkSize,
 int overlap,
 com.knuddels.jtokkit.api.EncodingType encodingType)

Method Details

encode

protected com.knuddels.jtokkit.api.IntArrayList encode(String input)

Description copied from class: ChunkingEmbeddings

Encodes a string into tokens

Specified by:

encode in class ChunkingEmbeddings<com.knuddels.jtokkit.api.IntArrayList>

Returns:

decode

protected String decode(com.knuddels.jtokkit.api.IntArrayList tokens)

Description copied from class: ChunkingEmbeddings

Decodes a string from an array of tokens

Specified by:

decode in class ChunkingEmbeddings<com.knuddels.jtokkit.api.IntArrayList>

Returns:

size

protected int size(com.knuddels.jtokkit.api.IntArrayList tokens)

Specified by:

size in class ChunkingEmbeddings<com.knuddels.jtokkit.api.IntArrayList>

slice

protected com.knuddels.jtokkit.api.IntArrayList slice(com.knuddels.jtokkit.api.IntArrayList tokens,
 int offset,
 int length)

Specified by:

slice in class ChunkingEmbeddings<com.knuddels.jtokkit.api.IntArrayList>