Content Splitting
After content has been extracted, it needs to be split into smaller, manageable parts—called "chunks"—before being indexed into the vector store. Proper splitting improves both the retrieval accuracy and embedding performance of AI agents.
Flowable supports multiple splitting strategies, each suitable for different use cases.
Context-Aware Splitting
This method uses the Markdown structure of the content to split it along semantic sections (headings, subheadings, etc.). It preserves context by including all parent headings in each content chunk.
Given the following input:
# Heading 1
This is a general description
## Heading 2
### Heading 3.1
My content
### Heading 3.2
Another content
The output would consist of three chunks:
# Heading 1
This is a general description
# Heading 1
## Heading 2
### Heading 3.1
My content
# Heading 1
## Heading 2
### Heading 3.2
Another content
This approach ensures each chunk contains enough context for AI reasoning while minimizing unnecessary repetition.
When dealing with large text blocks or improperly configured headers (see Header Mapping for Microsoft Word), chunks may exceed the size limits supported by the vector store.
Token Text Splitting
This method splits content into fixed-size token-based chunks, without analyzing document structure or semantics.
It is best suited for plain text or unstructured content.
Default Configuration
- Chunk size: 800 tokens
- Minimum chunk size (characters): 350
- Minimum chunk length to embed: 5 tokens
- Maximum number of chunks: 10,000
Example:
A document with 1,600 tokens would be split into 2 chunks of 800 tokens each.
This method guarantees consistent chunk sizes and is useful for performance-tuned pipelines, though it may lose contextual boundaries between sections.
No Splitting (Disabled)
This option disables content splitting entirely. The original extracted text will be indexed as a single, unbroken chunk.
Recommended when:
- Documents are short and self-contained
- You need to preserve full context
- You rely on external chunking logic (e.g., via OpenAI’s native processing)
Be aware that very large content may exceed vector embedding limits or reduce performance in semantic search.
Custom
The custom splitting APIs are currently experimental and may change in future releases.
For advanced use cases, you can implement your own chunking logic by providing a custom Java class.
Implement the following interface: com.flowable.agent.engine.impl.knowledgebase.pipeline.steps.chunkconverter.KnowledgeBaseChunkConverter
This gives you full control over how content is divided—supporting domain-specific rules, smart heuristics, or any advanced behavior.