Module org.nasdanika.ai

Package org.nasdanika.ai

Class CharChunkingEmbeddings

java.lang.Object

org.nasdanika.ai.ChunkingEmbeddings<char[]>

org.nasdanika.ai.CharChunkingEmbeddings

All Implemented Interfaces:

Coordinates, Embeddings, Model

public class CharChunkingEmbeddings
extends ChunkingEmbeddings<char[]>

A simple implementation which treats a character as a token. Can be used for testing and when an encoder is unknown or not available.

Nested Class Summary

Nested classes/interfaces inherited from interface org.nasdanika.ai.Embeddings

Embeddings.Requirement

Constructor Summary

Constructors

Constructor	Description
CharChunkingEmbeddings(Embeddings target, int chunkSize, int overlap)

Method Summary

Modifier and Type	Method	Description
protected String	decode(char[] tokens)	Decodes a string from an array of tokens
protected char[]	encode(String input)	Encodes a string into tokens
protected int	size(char[] tokens)
protected char[]	slice(char[] tokens, int offset, int length)

Methods inherited from class org.nasdanika.ai.ChunkingEmbeddings

chunk, generate, generateAsync, getDimensions, getMaxInputTokens, getName, getProvider, getVersion, isTooLong

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nasdanika.ai.Embeddings

generate, generateAsync

Constructor Details

CharChunkingEmbeddings

public CharChunkingEmbeddings(Embeddings target,
 int chunkSize,
 int overlap)

Method Details

encode

protected char[] encode(String input)

Description copied from class: ChunkingEmbeddings

Encodes a string into tokens

Specified by:

encode in class ChunkingEmbeddings<char[]>

Returns:

decode

protected String decode(char[] tokens)

Description copied from class: ChunkingEmbeddings

Decodes a string from an array of tokens

Specified by:

decode in class ChunkingEmbeddings<char[]>

Returns:

size

protected int size(char[] tokens)

Specified by:

size in class ChunkingEmbeddings<char[]>

slice

protected char[] slice(char[] tokens,
 int offset,
 int length)

Specified by:

slice in class ChunkingEmbeddings<char[]>