Class DistinctByTransformer<TSource, TKey>

Namespace: Wolfgang.Etl.Transformers

Assembly: Wolfgang.Etl.Transformers.dll

A transformer that yields each input item with a unique key, where the key is produced by a caller-supplied selector. Subsequent items whose key has already been seen are dropped.

public sealed class DistinctByTransformer<TSource, TKey> : ITransformAsync<TSource, TSource> where TSource : notnull where TKey : notnull

Type Parameters

TSource: The type of items in the input sequence. Must be non-null.
TKey: The type of key used for de-duplication. Must be non-null.

Inheritance: object

DistinctByTransformer<TSource, TKey>

Implements: ITransformAsync<TSource, TSource>

Inherited Members: object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods: TransformerExtensions.Then<TSource, TIntermediate, TDestination>(ITransformAsync<TSource, TIntermediate>, ITransformAsync<TIntermediate, TDestination>)

Examples

// first occurrence of each customer id
var uniqueCustomers = new DistinctByTransformer<Order, int>(o => o.CustomerId);

// case-insensitive distinct by name
var uniqueNames = new DistinctByTransformer<Person, string>
(
    p => p.Name,
    StringComparer.OrdinalIgnoreCase
);

Remarks

DistinctByTransformer<TSource, TKey> is the transformer equivalent of LINQ's Enumerable.DistinctBy (introduced in .NET 6). Keys are compared using either the supplied IEqualityComparer<T> or Default if none is provided. Order is preserved: the item bearing the first occurrence of each key is yielded.

The key selector is synchronous - key extraction is normally a cheap pure function (a property access, an integer ID, etc.) and an async overload would only add per-item overhead. If the key requires I/O to compute, project the key in a preceding SelectTransformer stage and use plain DistinctTransformer<T> on the result.

Memory footprint: a fresh HashSet<T> of keys is allocated per call to TransformAsync(IAsyncEnumerable<TSource>) and grows in proportion to the number of unique keys in the stream.

Implements only Wolfgang.Etl.Abstractions.ITransformAsync<TSource, TDestination> - no progress, no cancellation, no Skip/Max - to keep the hot loop minimal.

Constructors

DistinctByTransformer(Func<TSource, TKey>)

Initializes a new instance with the given key selector, using Default for key comparison.

public DistinctByTransformer(Func<TSource, TKey> keySelector)

Parameters

keySelector Func<TSource, TKey>: A function that extracts the comparison key from an input item.

Exceptions

ArgumentNullException: keySelector is null.

DistinctByTransformer(Func<TSource, TKey>, IEqualityComparer<TKey>?)

Initializes a new instance with the given key selector and equality comparer.

public DistinctByTransformer(Func<TSource, TKey> keySelector, IEqualityComparer<TKey>? comparer)

Parameters

keySelector Func<TSource, TKey>: A function that extracts the comparison key from an input item.
comparer IEqualityComparer<TKey>: The comparer used to determine key equality. If null, Default is used.

Exceptions

ArgumentNullException: keySelector is null.

Methods

TransformAsync(IAsyncEnumerable<TSource>)

Asynchronously yields the first item in items for each unique key produced by the configured selector, dropping later items with already-seen keys.

public IAsyncEnumerable<TSource> TransformAsync(IAsyncEnumerable<TSource> items)

Parameters

items IAsyncEnumerable<TSource>: The asynchronous source sequence.

Returns

IAsyncEnumerable<TSource>: An asynchronous sequence containing the first item per distinct key.

Exceptions

ArgumentNullException: items is null.