Class TestLoader<T>

Namespace: Wolfgang.Etl.TestKit

Assembly: Wolfgang.Etl.TestKit.dll

A simple in-memory loader for use in tests, examples, and benchmarks. Iterates all items from the source stream and, when constructed with collectItems: true, makes them available via GetCollectedItems() after the load completes.

public class TestLoader<T> : LoaderBase<T, Report>, ILoadWithProgressAndCancellationAsync<T, Report>, ILoadWithProgressAsync<T, Report>, ILoadWithCancellationAsync<T>, ILoadAsync<T>

Type Parameters

T: The type of item to load.

Inheritance: object

LoaderBase<T, Report>

TestLoader<T>

Implements: ILoadWithProgressAndCancellationAsync<T, Report>

ILoadWithProgressAsync<T, Report>

ILoadWithCancellationAsync<T>

ILoadAsync<T>

Inherited Members: LoaderBase<T, Report>.LoadAsync(IAsyncEnumerable<T>)

LoaderBase<T, Report>.LoadAsync(IAsyncEnumerable<T>, CancellationToken)

LoaderBase<T, Report>.LoadAsync(IAsyncEnumerable<T>, IProgress<Report>)

LoaderBase<T, Report>.LoadAsync(IAsyncEnumerable<T>, IProgress<Report>, CancellationToken)

LoaderBase<T, Report>.LoadWorkerAsync(IAsyncEnumerable<T>, CancellationToken)

LoaderBase<T, Report>.CreateProgressReport()

LoaderBase<T, Report>.IncrementCurrentItemCount()

LoaderBase<T, Report>.IncrementCurrentSkippedItemCount()

LoaderBase<T, Report>.ReportingInterval

LoaderBase<T, Report>.CurrentItemCount

LoaderBase<T, Report>.CurrentSkippedItemCount

LoaderBase<T, Report>.MaximumItemCount

LoaderBase<T, Report>.SkipItemCount

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Examples

// Test scenario — collect and assert after load:
var loader = new TestLoader<MyRecord>(collectItems: true);
await loader.LoadAsync(extractor.ExtractAsync());
var results = loader.GetCollectedItems();
Assert.Equal(expected, results);

// Second run — buffer is cleared at the start of each LoadAsync:
await loader.LoadAsync(extractor.ExtractAsync());
var results2 = loader.GetCollectedItems(); // contains only the second run's items

// Benchmark scenario — measure throughput without storing items:
var loader = new TestLoader<MyRecord>(collectItems: false);
await loader.LoadAsync(extractor.ExtractAsync());
// loader.GetCollectedItems() returns null in this mode

Remarks

When collectItems is false, the loader still enumerates every item so that benchmarks measure realistic throughput across the full pipeline, but nothing is stored. GetCollectedItems() returns null in this mode.

When collectItems is true, items are accumulated in an internal buffer throughout the load. GetCollectedItems() returns a snapshot of that buffer at the moment it is called — callers may call it mid-load to inspect items received so far, or post-load for the full result. Each new LoadAsync call clears the buffer before it begins.

Constructors

TestLoader(bool)

Initializes a new TestLoader<T>.

public TestLoader(bool collectItems)

Parameters

collectItems bool: When true, items are accumulated in an internal buffer during each load operation and made available via GetCollectedItems(). When false, items are enumerated but not stored — GetCollectedItems() returns null.

Methods

CreateProgressReport()

Creates a progress report of type TProgress. This gives the derived class the opportunity to implement a custom progress report that is specific to the extraction process.

protected override Report CreateProgressReport()

Returns

Report: Progress of type TProgress

GetCollectedItems()

Returns a snapshot of the items collected so far, or null if the loader was constructed with collectItems: false.

public IReadOnlyList<T>? GetCollectedItems()

Returns

IReadOnlyList<T>: A IReadOnlyList<T> containing a point-in-time copy of the collected items, or null when collection is disabled.

Examples

await loader.LoadAsync(extractor.ExtractAsync());
var items = loader.GetCollectedItems();
Assert.NotNull(items);
Assert.Equal(3, items.Count);

Remarks

The snapshot is taken at the moment this method is called. It may be called mid-load to inspect items received so far, or post-load for the complete result. Each new LoadAsync call clears the internal buffer before enumeration begins, so a post-load call always reflects the most recent run only.

LoadWorkerAsync(IAsyncEnumerable<T>, CancellationToken)

This method is the core implementation of the loading logic and should be overridden by derived classes.

protected override Task LoadWorkerAsync(IAsyncEnumerable<T> items, CancellationToken token)

Parameters

items IAsyncEnumerable<T>: The items to be loaded to the destination.
token CancellationToken: A CancellationToken to observe while waiting for the task to complete.

Returns

Task: Task

Remarks

Items may be an empty sequence if no data is available or if the extraction fails.

Exceptions

ArgumentNullException: Argument items is null