Class Pipeline

Namespace: Wolfgang.Etl.Abstractions

Assembly: Wolfgang.Etl.Abstractions.dll

Entry point for building a fluent ETL pipeline. Start with one of the Extract overloads, chain zero or more Transform calls, terminate with one of the Load overloads, then invoke RunAsync().

public static class Pipeline

Inheritance: object

Pipeline

Inherited Members: object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Examples

await Pipeline
    .Extract(csvExtractor).WithProgress(extractProgress)
    .Transform(enrich)
    .Load(sqlLoader).WithProgress(loadProgress)
    .WithName("nightly-import")
    .RunAsync(token);

Remarks

Overload resolution and capability interfaces. The Extract, Transform, and Load methods are overloaded by interface capability — no-progress/no-cancellation, with cancellation, with progress, or both. C# overload resolution uses the static type at the call site. A class that implements IExtractWithProgressAndCancellationAsync<TSource, TProgress> but is passed via a variable declared as IExtractAsync<TSource> will silently bind to the bare overload, and the pipeline will neither forward a cancellation token nor accept a progress sink. Pass stages using their most-derived interface (or concrete class) to get the intended behavior.

Methods

Extract<TSource>(IExtractAsync<TSource>)

Begins a pipeline from an extractor that supports neither progress nor cancellation. The pipeline's CancellationToken will not be forwarded into this stage.

public static IExtractStage<TSource> Extract<TSource>(IExtractAsync<TSource> extractor) where TSource : notnull

Parameters

extractor IExtractAsync<TSource>: The extractor that seeds the pipeline.

Returns

IExtractStage<TSource>: An IExtractStage<TSource> for chaining.

Type Parameters

TSource: The type of item produced by the extractor.

Exceptions

ArgumentNullException: extractor is null.

Extract<TSource>(IExtractWithCancellationAsync<TSource>)

Begins a pipeline from an extractor that supports cancellation but does not report progress.

public static IExtractStage<TSource> Extract<TSource>(IExtractWithCancellationAsync<TSource> extractor) where TSource : notnull

Parameters

extractor IExtractWithCancellationAsync<TSource>: The extractor that seeds the pipeline.

Returns

IExtractStage<TSource>: An IExtractStage<TSource> for chaining.

Type Parameters

TSource: The type of item produced by the extractor.

Exceptions

ArgumentNullException: extractor is null.

Extract<TSource, TProgress>(IExtractWithProgressAndCancellationAsync<TSource, TProgress>)

Begins a pipeline from a progress-reporting extractor that also supports cancellation. The returned stage exposes WithProgress(IProgress<TProgress>).

public static IExtractStageWithProgress<TSource, TProgress> Extract<TSource, TProgress>(IExtractWithProgressAndCancellationAsync<TSource, TProgress> extractor) where TSource : notnull where TProgress : notnull

Parameters

extractor IExtractWithProgressAndCancellationAsync<TSource, TProgress>: The extractor that seeds the pipeline.

Returns

IExtractStageWithProgress<TSource, TProgress>: An IExtractStageWithProgress<TSource, TProgress> for chaining.

Type Parameters

TSource: The type of item produced by the extractor.
TProgress: The type of progress report emitted by the extractor.

Exceptions

ArgumentNullException: extractor is null.

Extract<TSource, TProgress>(IExtractWithProgressAsync<TSource, TProgress>)

Begins a pipeline from a progress-reporting extractor that does not support cancellation. The pipeline's CancellationToken will not be forwarded into this stage. The returned stage exposes WithProgress(IProgress<TProgress>).

public static IExtractStageWithProgress<TSource, TProgress> Extract<TSource, TProgress>(IExtractWithProgressAsync<TSource, TProgress> extractor) where TSource : notnull where TProgress : notnull

Parameters

extractor IExtractWithProgressAsync<TSource, TProgress>: The extractor that seeds the pipeline.

Returns

IExtractStageWithProgress<TSource, TProgress>: An IExtractStageWithProgress<TSource, TProgress> for chaining.

Type Parameters

TSource: The type of item produced by the extractor.
TProgress: The type of progress report emitted by the extractor.

Exceptions

ArgumentNullException: extractor is null.