The output fields of a Chunking transformation include a
ChunkedText
field that stores the chunks and a
ProcessedText
field that stores the processed text. If the Chunking
transformations performs both chunking and text processing, the
ProcessedText
field stores the output after applying text processing
functions to the chunks.
As an active transformation, the Chunking transformation delivers an
output that's usually equal to or more than the
number of rows you input into the transformation.
Empty or null rows aren't factored into the
splitting methods and don't appear in the
ChunkedText
field.
You can pass the output fields to a Vector Embedding transformation to generate vector
embeddings. If the Chunking transformation performs both chunking and text processing, use
ProcessedText
to generate vector embeddings. However, you should also
pass
ChunkedText
to the downstream Vector Embedding and Target
transformations and include the chunks in the vector embedding metadata that's inserted into
the vector database.