Class FixedWidthMultiRecordExtractor

Namespace: Wolfgang.Etl.FixedWidth

Assembly: Wolfgang.Etl.FixedWidth.dll

Reads a fixed-width file that interleaves multiple record types — for example a mainframe batch file with a header, detail, and trailer layout on different lines — and yields each line as the object it maps to (#19).

public sealed class FixedWidthMultiRecordExtractor : ExtractorBase<object, FixedWidthReport>, IExtractWithProgressAndCancellationAsync<object, FixedWidthReport>, IExtractWithCancellationAsync<object>, IExtractWithProgressAsync<object, FixedWidthReport>, IExtractAsync<object>, IReportsItemErrors, IAsyncDisposable, IDisposable

Inheritance: object

ExtractorBase<object, FixedWidthReport>

FixedWidthMultiRecordExtractor

Implements: IExtractWithProgressAndCancellationAsync<object, FixedWidthReport>

IExtractWithCancellationAsync<object>

IExtractWithProgressAsync<object, FixedWidthReport>

IExtractAsync<object>

IReportsItemErrors

IAsyncDisposable

IDisposable

Inherited Members: ExtractorBase<object, FixedWidthReport>.ExtractAsync()

ExtractorBase<object, FixedWidthReport>.ExtractAsync(CancellationToken)

ExtractorBase<object, FixedWidthReport>.ExtractAsync(IProgress<FixedWidthReport>)

ExtractorBase<object, FixedWidthReport>.ExtractAsync(IProgress<FixedWidthReport>, CancellationToken)

ExtractorBase<object, FixedWidthReport>.DisposeAsync()

ExtractorBase<object, FixedWidthReport>.Dispose()

ExtractorBase<object, FixedWidthReport>.ReportingInterval

ExtractorBase<object, FixedWidthReport>.CurrentItemCount

ExtractorBase<object, FixedWidthReport>.CurrentSkippedItemCount

ExtractorBase<object, FixedWidthReport>.CurrentErrorItemCount

ExtractorBase<object, FixedWidthReport>.MaximumItemCount

ExtractorBase<object, FixedWidthReport>.SkipItemCount

ExtractorBase<object, FixedWidthReport>.WorkerResilience

ExtractorBase<object, FixedWidthReport>.ErrorPolicy

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Examples

using var extractor = new FixedWidthMultiRecordExtractor(reader)
    .When(line => line[0] == 'H', typeof(HeaderRecord))
    .When(line => line[0] == 'D', typeof(DetailRecord))
    .When(line => line[0] == 'T', typeof(TrailerRecord));

await foreach (var record in extractor.ExtractAsync(token))
{
    switch (record)
    {
        case HeaderRecord h: /* ... */ break;
        case DetailRecord d: /* ... */ break;
        case TrailerRecord t: /* ... */ break;
    }
}

Remarks

Register one rule per record type with When(Func<string, bool>, Type): a predicate over the raw line (typically a discriminator character such as line[0] == 'D') and the POCO type to materialize when it matches. Rules are evaluated in registration order — the first match wins. A line that matches no rule is handled per UnmatchedLineHandling, unless a fallback type was registered with Otherwise(Type).

Each record type keeps its own independent [FixedWidthField] layout. The yielded records are the concrete types you registered — pattern-match on them at the call site.

Ownership semantics match FixedWidthExtractor<TRecord>: a caller-supplied TextReader is not disposed; a Stream is wrapped in an internal 64 KB StreamReader that Dispose() releases while the stream itself stays open.

Constructors

FixedWidthMultiRecordExtractor(Stream, ILogger<FixedWidthMultiRecordExtractor>)

Initializes a new FixedWidthMultiRecordExtractor from a Stream using the default options, with diagnostic logging.

[Obsolete("Use the constructor that takes FixedWidthMultiRecordExtractorOptions. This overload will be removed in a future release.")]
public FixedWidthMultiRecordExtractor(Stream stream, ILogger<FixedWidthMultiRecordExtractor> logger)

Parameters

stream Stream: The stream to use.
logger ILogger<FixedWidthMultiRecordExtractor>: The logger to use for diagnostic output.

Exceptions

ArgumentNullException: stream is null.

FixedWidthMultiRecordExtractor(Stream, FixedWidthMultiRecordExtractorOptions?, ILogger<FixedWidthMultiRecordExtractor>?)

Initializes a new FixedWidthMultiRecordExtractor that reads from the specified Stream using an internal StreamReader with a 64 KB buffer. The caller retains ownership of the stream. Set Encoding to decode with a specific encoding (defaults to UTF8).

public FixedWidthMultiRecordExtractor(Stream stream, FixedWidthMultiRecordExtractorOptions? options = null, ILogger<FixedWidthMultiRecordExtractor>? logger = null)

Parameters

stream Stream: The readable source stream.
options FixedWidthMultiRecordExtractorOptions: Options that control behaviour, including the encoding. When null — or omitted — the documented defaults apply.
logger ILogger<FixedWidthMultiRecordExtractor>: An optional ILogger<TCategoryName> for diagnostic output. Pass null (the default) to disable logging.

Exceptions

ArgumentNullException: stream is null.

FixedWidthMultiRecordExtractor(TextReader, ILogger<FixedWidthMultiRecordExtractor>?)

Initializes a new FixedWidthMultiRecordExtractor that reads from the specified TextReader. The caller owns the reader's lifetime.

public FixedWidthMultiRecordExtractor(TextReader reader, ILogger<FixedWidthMultiRecordExtractor>? logger = null)

Parameters

reader TextReader: The reader to pull fixed-width lines from.
logger ILogger<FixedWidthMultiRecordExtractor>: An optional ILogger<TCategoryName> for diagnostic output. Pass null (the default) to disable logging.

Exceptions

ArgumentNullException: reader is null.

FixedWidthMultiRecordExtractor(TextReader, FixedWidthMultiRecordExtractorOptions?, ILogger<FixedWidthMultiRecordExtractor>?)

Initializes a new FixedWidthMultiRecordExtractor that reads lines from reader, with the base-stage configuration taken from options (ADR-0009). Encoding on the record is not used for a TextReader source.

[SuppressMessage("ApiDesign", "RS0026:Do not add multiple public overloads with optional parameters", Justification = "Shipped shape; unlike the single-record stages there is no hidden (TextReader) overload to fall back on, so dropping either default is a source break. The constructor set is settled in the 2026-12-15 removal wave (#373 / #343).")]
public FixedWidthMultiRecordExtractor(TextReader reader, FixedWidthMultiRecordExtractorOptions? options, ILogger<FixedWidthMultiRecordExtractor>? logger = null)

Parameters

reader TextReader: The text source to read lines from.
options FixedWidthMultiRecordExtractorOptions: The stage configuration; null keeps the defaults.
logger ILogger<FixedWidthMultiRecordExtractor>: The logger; null logs nothing.

Exceptions

ArgumentNullException: reader is null.

Properties

CurrentFilteredLineCount

The number of physical lines read that produced no record and were not counted as skipped or rejected: header lines, blank lines dropped by SkipBlankLines, and unmatched lines dropped by Skip.

public int CurrentFilteredLineCount { get; }

Property Value

int

CurrentLineNumber

The 1-based physical line number of the line most recently read. Thread-safe so it may be sampled from a progress timer thread.

public long CurrentLineNumber { get; }

Property Value

long

CurrentRejectedItemCount

The number of matched lines dropped by Skip. Distinct from the SkipItemCount pagination budget.

public int CurrentRejectedItemCount { get; }

Property Value

int

Encoding

The Encoding used by this instance. Superseded by Encoding.

[Obsolete("Set Encoding on FixedWidthMultiRecordExtractorOptions and pass it to the constructor instead. This property will be removed in a future release.")]
public Encoding Encoding { get; init; }

Property Value

Encoding

Remarks

Retained for source compatibility with 0.10.x. It is honoured only when the constructor was given no options object; a caller who passes options is using the supported route and that value wins. The two cannot conflict in existing code, because the options constructor did not exist before 0.11.0.

FieldDelimiter

An optional delimiter present between columns in the source file, or null (the default) for pure fixed-width input. Applies to every registered record type.

public string? FieldDelimiter { get; init; }

Property Value

string

HasHeader

Convenience wrapper over HeaderLineCount — true maps to 1, false to 0.

public bool HasHeader { get; init; }

Property Value

bool

HeaderLineCount

The number of header lines to skip at the start of the file before routing begins. Defaults to 0. Use this only for banner lines that precede the record body; a leading H record that you want to capture should be registered with When(Func<string, bool>, Type) instead.

public int HeaderLineCount { get; init; }

Property Value

int

MalformedLineHandling

What to do when a matched line cannot be parsed into its record type — too short, or a field value that will not convert. Defaults to ThrowException; Skip drops the line and continues. ReturnDefault is not supported here (the substitute type would be ambiguous) and throws InvalidOperationException if set.

public MalformedLineHandling MalformedLineHandling { get; init; }

Property Value

MalformedLineHandling

OnError

An optional dead-letter sink invoked once for each line that fails to parse (#29). With Skip the line is reported and dropped; with the default ThrowException it is reported before the exception is re-thrown.

public Action<FixedWidthError>? OnError { get; init; }

Property Value

Action<FixedWidthError>

SkipBlankLines

When true (the default), zero-length lines are skipped before any predicate runs, so discriminators may index the line without guarding against empty input. When false, a blank line is treated as an unmatched line.

public bool SkipBlankLines { get; init; }

Property Value

bool

UnmatchedLineHandling

What to do with a data line that matches no When(Func<string, bool>, Type) rule and for which no Otherwise(Type) fallback was registered. Defaults to ThrowException.

public UnmatchedLineHandling UnmatchedLineHandling { get; init; }

Property Value

UnmatchedLineHandling

ValueParser

The value parser applied to every field of every record type. Defaults to DefaultParser.

public FixedWidthValueParser ValueParser { get; init; }

Property Value

FixedWidthValueParser

Methods

CreateProgressReport()

Creates a progress report of type TProgress. This gives the derived class the opportunity to implement a custom progress report that is specific to the extraction process.

protected override FixedWidthReport CreateProgressReport()

Returns

FixedWidthReport: Progress of type TProgress

CreateProgressTimer(IProgress<FixedWidthReport>)

Creates the Wolfgang.Etl.Abstractions.IProgressTimer used to drive progress callbacks. Override this method in a derived class to inject a custom timer (for example, a custom implementation that allows manual control in unit tests).

protected override IProgressTimer CreateProgressTimer(IProgress<FixedWidthReport> progress)

Parameters

progress IProgress<FixedWidthReport>: The progress sink that will receive callbacks.

Returns

IProgressTimer: A started Wolfgang.Etl.Abstractions.IProgressTimer instance.

Dispose(bool)

Releases resources held by this extractor. Override in a derived class to dispose resources it owns (streams, connections, etc.), then call base.Dispose(disposing). The base implementation only marks the instance disposed and is idempotent.

protected override void Dispose(bool disposing)

Parameters

disposing bool: true when called from Wolfgang.Etl.Abstractions.ExtractorBase<TSource, TProgress>.Dispose() or Wolfgang.Etl.Abstractions.ExtractorBase<TSource, TProgress>.DisposeAsync() (dispose managed resources); false when called from a finalizer.

ExtractWorkerAsync(CancellationToken)

This method is the core implementation of the extraction logic and should be overridden by derived classes.

protected override IAsyncEnumerable<object> ExtractWorkerAsync(CancellationToken token)

Parameters

token CancellationToken: A CancellationToken to observe while waiting for the task to complete.

Returns

IAsyncEnumerable<object>: IAsyncEnumerable<TSource> The result may be an empty sequence if no data is available or if the extraction fails.

OnItemError(ItemErrorContext)

Translates MalformedLineHandling into the base per-item error policy and reports the failure to the OnError dead-letter sink. Skip maps to Wolfgang.Etl.Abstractions.ItemErrorAction.Skip; ThrowException (the default) maps to Wolfgang.Etl.Abstractions.ItemErrorAction.Abort. ReturnDefault is rejected up front, so it never reaches this hook.

protected override ItemErrorAction OnItemError(ItemErrorContext context)

Parameters

context ItemErrorContext

Returns

ItemErrorAction

Otherwise(Type)

Registers a fallback record type for lines that match no When(Func<string, bool>, Type) rule. When a fallback is set it takes precedence over UnmatchedLineHandling — the otherwise-unmatched line is parsed as recordType instead of being thrown or skipped. Returns this extractor so calls can be chained.

public FixedWidthMultiRecordExtractor Otherwise(Type recordType)

Parameters

recordType Type: The catch-all POCO type for unmatched lines.

Returns

FixedWidthMultiRecordExtractor

Exceptions

ArgumentNullException: recordType is null.
InvalidOperationException: recordType has an invalid layout.

When(Func<string, bool>, Type)

Registers a rule: when predicate returns true for a line, that line is parsed as recordType. Rules are evaluated in the order they are registered and the first match wins. Returns this extractor so calls can be chained.

public FixedWidthMultiRecordExtractor When(Func<string, bool> predicate, Type recordType)

Parameters

predicate Func<string, bool>: A discriminator over the raw line — for example line => line[0] == 'D'. Blank lines are not passed to the predicate when SkipBlankLines is true (the default), so a discriminator may index the line safely.
recordType Type: The POCO type to materialize, decorated with [FixedWidthField] attributes and having a public parameterless constructor.

Returns

FixedWidthMultiRecordExtractor

Exceptions

ArgumentNullException: predicate or recordType is null.
InvalidOperationException: recordType has an invalid layout (for example duplicate column indexes or a mapped property with no public setter).