AI Accelerator - Pipelines 4.1.0 release notes

The preparer pipeline for operations that transform the shape of their input data with an additional dimension now unnest their result collections. This allows the output of preparers to be consumed much more easily by other preparers or knowledge bases. Unnested results are returned with a new part_id column to track the new dimension. There is also a new unique_id column to uniquely identify the combination of the source key and part_id.

The external model providers embeddings, openai_embeddings, and nim_embeddings can now send a batch of inputs in a single request, rather than multiple concurrent requests. This can improve performance and hardware utilization. The feature is fully configurable and can also be disabled.

The enumeration column returned by the chunk_text() primitive function is now part_id instead of chunk_id to match the other Preparer primitives/operations.