org.noear.solon.ai.rag.splitter

Class TokenSizeTextSplitter

java.lang.Object

org.noear.solon.ai.rag.splitter.TextSplitter

org.noear.solon.ai.rag.splitter.TokenSizeTextSplitter

All Implemented Interfaces:

DocumentSplitter

public class TokenSizeTextSplitter
extends TextSplitter

词元大小文本分割器

Since:

3.1

Author:

noear

Constructor Summary

Constructors

Constructor and Description
TokenSizeTextSplitter()
TokenSizeTextSplitter(int chunkSize)
TokenSizeTextSplitter(int chunkSize, int minChunkSizeChars)
TokenSizeTextSplitter(int chunkSize, int minChunkSizeChars, int minChunkLengthToEmbed, int maxChunkCount, boolean keepSeparator)

Method Summary

Modifier and Type	Method and Description
protected String	decodeTokens(com.knuddels.jtokkit.api.Encoding encoding, List<Integer> tokens) 解码符号
protected List<Integer>	encodeTokens(com.knuddels.jtokkit.api.Encoding encoding, String text) 编码符号
void	setEncodingRegistry(com.knuddels.jtokkit.api.EncodingRegistry encodingRegistry) 设置编码库
void	setEncodingType(com.knuddels.jtokkit.api.EncodingType encodingType) 设置编码类型
protected List<String>	splitText(String text)

Methods inherited from class org.noear.solon.ai.rag.splitter.TextSplitter

split, splitDocument

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.noear.solon.ai.rag.DocumentSplitter

split

Constructor Detail

TokenSizeTextSplitter

public TokenSizeTextSplitter()

TokenSizeTextSplitter

public TokenSizeTextSplitter(int chunkSize)

TokenSizeTextSplitter

public TokenSizeTextSplitter(int chunkSize,
                             int minChunkSizeChars)

TokenSizeTextSplitter

public TokenSizeTextSplitter(int chunkSize,
                             int minChunkSizeChars,
                             int minChunkLengthToEmbed,
                             int maxChunkCount,
                             boolean keepSeparator)

Method Detail

setEncodingRegistry

public void setEncodingRegistry(com.knuddels.jtokkit.api.EncodingRegistry encodingRegistry)

设置编码库

setEncodingType

public void setEncodingType(com.knuddels.jtokkit.api.EncodingType encodingType)

设置编码类型

splitText

protected List<String> splitText(String text)

Specified by:

splitText in class TextSplitter

encodeTokens

protected List<Integer> encodeTokens(com.knuddels.jtokkit.api.Encoding encoding,
                                     String text)

编码符号

decodeTokens

protected String decodeTokens(com.knuddels.jtokkit.api.Encoding encoding,
                              List<Integer> tokens)

解码符号