Class ChunkTransformer<T>

Namespace: Wolfgang.Etl.Transformers

Assembly: Wolfgang.Etl.Transformers.dll

A transformer that batches the input sequence into fixed-size groups and yields each batch as a single IReadOnlyList<T> output item.

public sealed class ChunkTransformer<T> : ITransformAsync<T, IReadOnlyList<T>> where T : notnull

Type Parameters

T: The type of items flowing through the transformer. Must be non-null.

Inheritance: object

ChunkTransformer<T>

Implements: ITransformAsync<T, IReadOnlyList<T>>

Inherited Members: object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods: TransformerExtensions.Then<TSource, TIntermediate, TDestination>(ITransformAsync<TSource, TIntermediate>, ITransformAsync<TIntermediate, TDestination>)

Examples

// batch rows into groups of 1000 for bulk loading
var batches = new ChunkTransformer<Row>(size: 1000);

// batch and report progress as items are consumed
var progress = new Progress<int>(n => Console.WriteLine($"{n} rows chunked"));
var tracked = new ChunkTransformer<Row>(size: 1000, progress);

Remarks

ChunkTransformer<T> is the transformer equivalent of LINQ's Enumerable.Chunk (introduced in .NET 6): it groups consecutive input items into Size-element arrays. The last chunk may be smaller than Size if the input length is not an exact multiple of Size.

Each yielded chunk is backed by a freshly-allocated T[] (exposed as IReadOnlyList<T>) - chunks do not share backing storage with the transformer or with each other, so consumers can retain each chunk without affecting subsequent output.

Useful for batching writes to a downstream loader (e.g. SqlBulkCopy) without materializing the entire stream.

Implements only Wolfgang.Etl.Abstractions.ITransformAsync<TSource, TDestination> - no cancellation, no Skip/Max - to keep the hot loop minimal. An optional IProgress<T> sink may be supplied to receive the running count of source items consumed; when none is supplied the hot path allocates and reports nothing.

Constructors

ChunkTransformer(int)

Initializes a new instance with the given chunk size.

public ChunkTransformer(int size)

Parameters

size int: The maximum number of items in each yielded chunk. Must be at least 1.

Exceptions

ArgumentOutOfRangeException: size is less than 1.

ChunkTransformer(int, IProgress<int>?)

Initializes a new instance with the given chunk size and a progress sink that receives the running count of source items consumed.

public ChunkTransformer(int size, IProgress<int>? progress)

Parameters

size int: The maximum number of items in each yielded chunk. Must be at least 1.
progress IProgress<int>: A sink that receives the cumulative number of source items consumed so far, reported once per yielded chunk (including the partial final chunk). May be null to disable reporting.

Exceptions

ArgumentOutOfRangeException: size is less than 1.

Properties

Size

The maximum number of items in each yielded chunk.

public int Size { get; }

Property Value

int

Methods

TransformAsync(IAsyncEnumerable<T>)

Asynchronously yields successive chunks of Size consecutive items from items. The last chunk may contain fewer items if the source length is not a multiple of Size.

public IAsyncEnumerable<IReadOnlyList<T>> TransformAsync(IAsyncEnumerable<T> items)

Parameters

items IAsyncEnumerable<T>: The asynchronous source sequence.

Returns

IAsyncEnumerable<IReadOnlyList<T>>: An asynchronous sequence of read-only lists, each containing up to Size items.

Exceptions

ArgumentNullException: items is null.