Class LoaderBase<TDestination, TProgress>

Namespace: Wolfgang.Etl.Abstractions

Assembly: Wolfgang.Etl.Abstractions.dll

Provides a basic implementation for data loaders that write data of type TDestination to a target destination. Library authors can use this base class to create custom loaders by inheriting from it and implementing LoadWorkerAsync and CreateProgressReport methods.

public abstract class LoaderBase<TDestination, TProgress> : ILoadWithProgressAndCancellationAsync<TDestination, TProgress>, ILoadWithProgressAsync<TDestination, TProgress>, ILoadWithCancellationAsync<TDestination>, ILoadAsync<TDestination> where TDestination : notnull where TProgress : notnull

Type Parameters

TDestination: The type of the destination object being written
TProgress: The type of the progress object

Inheritance: object

LoaderBase<TDestination, TProgress>

Implements: ILoadWithProgressAndCancellationAsync<TDestination, TProgress>

ILoadWithProgressAsync<TDestination, TProgress>

ILoadWithCancellationAsync<TDestination>

ILoadAsync<TDestination>

Inherited Members: object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Properties

CurrentItemCount

The current number of items loaded so far.

public int CurrentItemCount { get; }

Property Value

int

Remarks

It is the responsibility of the derived class to call IncrementCurrentItemCount() as each item is loaded. The base class has no way of knowing when an item has been processed.

CurrentSkippedItemCount

The current number of items skipped so far during loading.

public int CurrentSkippedItemCount { get; }

Property Value

int

MaximumItemCount

The maximum number of items to load. Once the loader has reached this limit, it should stop loading items as if it had reached the end of the sequence

public int MaximumItemCount { get; set; }

Property Value

int

Examples

foreach (var item in items.Skip(SkipItemCount).Take(MaxItemCount))
{
    // Process the item
}

Remarks

This is useful for partially loading data from a source, especially when the source is large or infinite or during development.

Exceptions

ArgumentOutOfRangeException: The specified value is less than 0.

ReportingInterval

The number of milliseconds between progress updates.

public int ReportingInterval { get; set; }

Property Value

int

Exceptions

ArgumentOutOfRangeException: Value cannot be less than 1.

SkipItemCount

The number of items skipped before loading. The loader should skip the specified number of items before starting to process the remaining items.

public int SkipItemCount { get; set; }

Property Value

int

Examples

foreach (var item in items.Skip(SkipItemCount).Take(MaxItemCount))
{
    // Process the item
}

Remarks

This is useful for skipping the beginning of the list during testing or because it may already be loaded

Exceptions

ArgumentOutOfRangeException: The specified value is less than 0.

Methods

CreateProgressReport()

Creates a progress report of type TProgress. This gives the derived class the opportunity to implement a custom progress report that is specific to the loading process.

protected abstract TProgress CreateProgressReport()

Returns

TProgress: Progress of type TProgress

CreateProgressTimer(IProgress<TProgress>)

Creates the IProgressTimer used to drive progress callbacks. Override this method in a derived class to inject a custom timer (for example, a custom implementation that allows manual control in unit tests).

protected virtual IProgressTimer CreateProgressTimer(IProgress<TProgress> progress)

Parameters

progress IProgress<TProgress>: The progress sink that will receive callbacks.

Returns

IProgressTimer: A started IProgressTimer instance.

IncrementCurrentItemCount()

Increments the CurrentItemCount in a thread safe manner.

[SuppressMessage("IDE0058", "IDE0058:Expression value is never used", Justification = "Interlocked.Increment return value intentionally discarded; only the side-effect matters.")]
protected void IncrementCurrentItemCount()

Remarks

Simply calling CurrentItemCount++ or CurrentItemCount += 1 is not thread safe. This method ensures that CurrentItemCount is incremented safely.

IncrementCurrentSkippedItemCount()

Increments the CurrentSkippedItemCount in a thread safe manner.

[SuppressMessage("IDE0058", "IDE0058:Expression value is never used", Justification = "Interlocked.Increment return value intentionally discarded; only the side-effect matters.")]
protected void IncrementCurrentSkippedItemCount()

Remarks

Simply calling CurrentSkippedItemCount++ or CurrentSkippedItemCount += 1 is not thread safe. This method ensures that CurrentSkippedItemCount is incremented safely.

LoadAsync(IAsyncEnumerable<TDestination>)

Asynchronously loads data of type TDestination into the target destination.

public virtual Task LoadAsync(IAsyncEnumerable<TDestination> items)

Parameters

items IAsyncEnumerable<TDestination>: The items to be loaded to the destination.

Returns

Task: A task representing the asynchronous operation.

Remarks

Items may be an empty sequence if no data is available or if the loading fails.

Exceptions

ArgumentNullException: Argument items is null

LoadAsync(IAsyncEnumerable<TDestination>, IProgress<TProgress>)

Asynchronously loads data of type TDestination into the target destination.

public virtual Task LoadAsync(IAsyncEnumerable<TDestination> items, IProgress<TProgress> progress)

Parameters

items IAsyncEnumerable<TDestination>: The items to be loaded to the destination.
progress IProgress<TProgress>: A provider for progress updates.

Returns

Task: A task representing the asynchronous operation.

Remarks

Items may be an empty sequence if no data is available or if the loading fails.

Exceptions

ArgumentNullException: Argument items is null
ArgumentNullException: Argument progress is null

LoadAsync(IAsyncEnumerable<TDestination>, IProgress<TProgress>, CancellationToken)

Asynchronously loads data of type TDestination into the target destination.

public virtual Task LoadAsync(IAsyncEnumerable<TDestination> items, IProgress<TProgress> progress, CancellationToken token)

Parameters

items IAsyncEnumerable<TDestination>: The items to be loaded to the destination.
progress IProgress<TProgress>: A provider for progress updates.
token CancellationToken: A CancellationToken to observe while waiting for the task to complete.

Returns

Task: A task representing the asynchronous operation.

Remarks

Items may be an empty sequence if no data is available or if the loading fails.

Exceptions

ArgumentNullException: Argument items is null
ArgumentNullException: Argument progress is null

LoadAsync(IAsyncEnumerable<TDestination>, CancellationToken)

Asynchronously loads data of type TDestination into the target destination.

public virtual Task LoadAsync(IAsyncEnumerable<TDestination> items, CancellationToken token)

Parameters

items IAsyncEnumerable<TDestination>: The items to be loaded to the destination.
token CancellationToken: A CancellationToken to observe while waiting for the task to complete.

Returns

Task: A task representing the asynchronous operation.

Remarks

Items may be an empty sequence if no data is available or if the loading fails.

Exceptions

ArgumentNullException: Argument items is null

LoadWorkerAsync(IAsyncEnumerable<TDestination>, CancellationToken)

This method is the core implementation of the loading logic and should be overridden by derived classes.

protected abstract Task LoadWorkerAsync(IAsyncEnumerable<TDestination> items, CancellationToken token)

Parameters

items IAsyncEnumerable<TDestination>: The items to be loaded to the destination.
token CancellationToken: A CancellationToken to observe while waiting for the task to complete.

Returns

Task: A task representing the asynchronous operation.

Remarks

Items may be an empty sequence if no data is available or if the loading fails.

Exceptions

ArgumentNullException: Argument items is null