Module org.nasdanika.models.rag.core

Package org.nasdanika.rag.core

Class PdfTextSplitter

java.lang.Object

org.nasdanika.rag.core.PdfTextSplitter

public class PdfTextSplitter
extends Object

Extracts text from PDF and splits into chunks. This class tries to keep paragraphs together and split them into sentences if keeping together is not possible.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	PdfTextSplitter.Chunk

Constructor Summary

Constructors

Constructor	Description
PdfTextSplitter(int size, int overlap, int tolerance, Function<String,List<String>> tokenizer)

Method Summary

Modifier and Type	Method	Description
protected String	getLineSeparator()
protected String	getParagraphSeparator()
protected String	getWordSeparator()
List<PdfTextSplitter.Chunk>	split(org.nasdanika.models.pdf.Document document)	Splits docuent into chunks.
protected List<String>	splitIntoSentences(String text)
protected List<String>	splitIntoWords(String text)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

PdfTextSplitter

public PdfTextSplitter(int size,
 int overlap,
 int tolerance,
 Function<String,List<String>> tokenizer)

Parameters:

size - Chunk size in tokens

overlap - Chunk overlap in tokens.

tolerance - Size tolerance to allow keep paragraphs and sentences together if possible

tokenCounter -

Method Details

splitIntoSentences

protected List<String> splitIntoSentences(String text)

splitIntoWords

protected List<String> splitIntoWords(String text)

getWordSeparator

protected String getWordSeparator()

getLineSeparator

protected String getLineSeparator()

getParagraphSeparator

protected String getParagraphSeparator()

split

public List<PdfTextSplitter.Chunk> split(org.nasdanika.models.pdf.Document document)

Splits docuent into chunks.

Parameters:

document -

Returns: