Class DistinctTransformer<T>

Namespace

Wolfgang.Etl.Transformers

Assembly

Wolfgang.Etl.Transformers.dll

A transformer that yields each unique item from the input sequence, dropping subsequent duplicates.

public sealed class DistinctTransformer<T> : ITransformAsync<T, T> where T : notnull

Type Parameters

T

The type of items flowing through the transformer. Must be non-null.

Inheritance

object

DistinctTransformer<T>

Implements

ITransformAsync<T, T>

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods

TransformerExtensions.Then<TSource, TIntermediate, TDestination>(ITransformAsync<TSource, TIntermediate>, ITransformAsync<TIntermediate, TDestination>)

Examples

// default equality
var unique = new DistinctTransformer<string>();

// case-insensitive
var insensitive = new DistinctTransformer<string>(StringComparer.OrdinalIgnoreCase);

Remarks

DistinctTransformer<T> is the transformer equivalent of LINQ's Distinct<TSource>(IEnumerable<TSource>). Items are compared using either the supplied IEqualityComparer<T> or Default if none is provided. Order is preserved: the first occurrence of each item is yielded.

Memory footprint: a fresh HashSet<T> is allocated per call to TransformAsync(IAsyncEnumerable<T>) and grows in proportion to the number of unique items in the stream. For unbounded streams or streams with many unique items, prefer DistinctByTransformer<TSource, TKey> with a small key type, or apply Distinct only after a stage that bounds cardinality.

Implements only Wolfgang.Etl.Abstractions.ITransformAsync<TSource, TDestination> - no progress, no cancellation, no Skip/Max - to keep the hot loop minimal.

Constructors

DistinctTransformer()

Initializes a new instance using Default.

public DistinctTransformer()

DistinctTransformer(IEqualityComparer<T>?)

Initializes a new instance using the supplied equality comparer.

public DistinctTransformer(IEqualityComparer<T>? comparer)

Parameters

comparer IEqualityComparer<T>

The comparer used to determine equality. If null, Default is used.

Methods

TransformAsync(IAsyncEnumerable<T>)

Asynchronously yields the first occurrence of each unique item in items, dropping later duplicates.

public IAsyncEnumerable<T> TransformAsync(IAsyncEnumerable<T> items)

Parameters

items IAsyncEnumerable<T>

The asynchronous source sequence.

Returns

IAsyncEnumerable<T>

An asynchronous sequence containing each item from items at most once.

Exceptions

ArgumentNullException

items is null.