# PODNAME: Iterator::Flex::Manual::Authoring
# ABSTRACT: How to write an iterator

__END__

=pod

=for :stopwords Diab Jerius Smithsonian Astrophysical Observatory Raku

=head1 NAME

Iterator::Flex::Manual::Authoring - How to write an iterator

=head1 VERSION

version 0.34

=head1 DESCRIPTION

=head2 Iterator Phases

Iterators must manage the four different phases that the iterator might be in:

=over

=item * initialization

=item * iteration

=item * exhaustion

=item * error

=back

For more details, see L<Iterator::Flex::Manual::Overview/Iterator life-cycle>.

=head3 Initialization

When an iterator is constructed it is typically passed some state
information; it may be an array or a hash, a database handle, or a
file pointer.

The constructor must save the relevant pieces of information
(typically through closed-over variables; see below) and initialize
variables which keep track of where the iterator is in the data stream.

For example, if the iterator operates on an array, it will need to keep
track of the index of the element it must return next.

Some iterators don't need or have access to that information.  If an
iterator operates on a file handle, returning the next line in the file,
the file handle keeps track of the next line in the file, so the
iterator doesn't need to.  Similarly, if an iterator is retrieving
data from a database via a cursor, the database will keep track of
where it is in the data stream.

=head3 Iteration

In the iteration phase, the iterator identifies the data to return,
updates its internal state (if necessary) so that it will return the
correct data on the next iteration, and returns the data.

If the data stream has been exhausted, then the iterator must indicate
this by calling the C<signal_exhaustion> method. This method
implements the exhaustion policy requested by the user who set up the
iterator (either returning a sentinel value or throwing an exception.

After this the iterator enters the L</Exhaustion> phase.

If there is an error (e.g. if a database connection is dropped), the
iterator must signal this by calling the C<signal_error> method.  Not
all iterators have an error phase.

=head3 Exhaustion

Unlike other iteration implementations, it is legal to call an
iterator's C<next> method after the iterator is exhausted.
In the exhaustion phase, the iterator simply invokes
the C<signal_exhaustion> method

=head3 Error

Not all iterators have an error phase, but if they are in one, they
simply call C<signal_error>.

=head2 Capabilities

An iterator must do at least one thing: return the next datum from the
data stream.  This is the I<next> capability. L<Iterator::Flex>
iterators can support a number of other capabilities; see in
L<Iterator::Flex::Manual::Overview>,

=head2 Building an Iterator

Iterators are constructed by passing an attribute hash, C<%AttrHash>
to the C<Iterator::Flex> factory, which uses it to construct an
appropriate iterator class, instantiate it, and return it to the user.

The attribute hash (whose contents are documented much greater detail
in L<Iterator::Flex::Manual::Overview/Iterator Parameters>) describes
the iterator's capabilities and provides implementations.

The main iterator routine (C<next>) must be a closure, with state
contained in closed over variables. Every time a new iterator is
constructed, a new closure is generated.

Writing an iterator generally involves writing a subroutine which
returns the C<%AttrHash> containing the closures.  As an example,
we will construct an iterator which operates on arrays, providing
a number of capabilities.

For simplicity, we'll write a C<construct> subroutine which is passed
a reference to the array to iterator over, and returns the C<%AttrHash>.
Later we'll see how to create a one-off iterator and a standalone iterator
class using the concepts we've explored.

Our C<construct> subroutine will be called as

 $ref_AttrHash = construct( \@array );

=head3 Creating the C<next> capability

First, let's concentrate on the heart of the iterator, the C<next> capability,
which I<must> be implemented as a closure. 

C<next> has three responsibilities:

=over

=item *

return the next data element

=item *

signal exhaustion

=item *

(optionally) signal an error.

=back

It usually also ensures that the C<current> and C<previous>
capabilities return the proper values.  Because it is called most
often, it should be as efficient as possible.

C<next> cannot keep state internally. Our C<construct> subroutine will
store the state in lexical variables which only our instance of
C<next> will have access to.

To illustrate, here's an implementation of C<next> for iteration
over an array:

 my $next = sub {
     if ( $next == $len ) {
         # if first time through, set current
         $prev = $current
           if ! $self->is_exhausted;
         return $current = $self->signal_exhaustion;
     }
     $prev    = $current;
     $current = $next++;

     return $arr->[$current];
 };

Notice that the subroutine doesn't take I<any> parameters. Also notice
that it uses a number of variables that are I<not defined> in the
subroutine, e.g. C<$arr>, C<$next>, etc. These are lexical variables
in C<configure> and are initialized I<outside> of the C<$next>
closure.

C<$arr> is the array we're operating on, C<$len> is its length (so we
don't have to look it up every time). Because it's cheap to retain the
state of an array (it's just an index), we can easily keep track of
what is needed to implement the
L<prev|Iterator::Flex::Manual::Overview/prev> and
L<current|Iterator::Flex::Manual::Overview/current> capabilities;
those are stored in C<$prev> and C<$current>.

Finally, there's C<$self>, which is a handle for our iterator.  It's
not used for any performance critical work.

These must all be properly initialized by C<construct> before C<$next>
is created; we'll go over that later. Let's first look at the code for
the C<$next> closure.

The code is divided into two sections; the first deals with data exhaustion>:

     if ( $next == $len ) {
         # if first time through, set prev
         $prev = $current
           if ! $self->is_exhausted;
         return $current = $self->signal_exhaustion;
     }

Every time the iterator is invoked, it checks if it has run out of
data.  If it is has (e.g. C<$next == $len>) then the iterator
sets up the exhaustion phase.  The C<is_exhausted> predicate will be
true if the iterator is I<already> in the exhaustion phase.  If it is,
it doesn't need to perform work required to handle other capabilities.
In our case, the first time the iterator is in the exhausted state it
must set C<$prev> so that it correctly returns the last element in the
array (which will be $current from the last successful iteration).

Then, it signals exhaustion by returning its C<signal_exhaustion>
method (and setting C<$current> to that value, so the C<current>
capability will return the correct value).

Recall that it is the I<client> code that determines how the iterator
will signal exhaustion (i.e, via a sentinel value or an exception). The
iterator itself doesn't care; it simply returns the result of the
C<signal_exhaustion> method, which will set the C<is_exhausted> object
predicate and then either return a sentinel value or throw an
exception.

In other iterator implementations (e.g. C++, Raku), calling C<next>
(or other methods) on an exhausted iterator is undefined behavior.
This is I<not> true for C<Iterator::Flex> iterators.  An exhausted
iterator must always respond, identically, to a call to C<next>, so
must always return the result of the C<signal_exhaustion> method.

The second part of the code takes care of returning the correct data
and setting the iterator up for the succeeding call to C<next>. It
also ensures that the I<< L<current|Iterator::Flex::Manual::Overview/current> >>
and I<< L<prev|Iterator::Flex::Manual::Overview/prev> >> capabilities will
return the proper values:

     $prev    = $current;
     $current = $next++;

     return $arr->[$current];

=head3 Other capabilities

For completeness, here's the implementation of the rest of the iterator's capabilities:

 my $reset   = sub { $prev = $current = undef;  $next = 0; };
 my $rewind  = sub { $next = 0; };
 my $prev    = sub { return defined $prev ? $arr->[$prev] : undef; };
 my $current = sub { return defined $current ? $arr->[$current] : undef; };

They have been written as closures accessing the lexical variables,
but they could also have been written as methods if the iterator chose
to store its state in some other fashion.  Only C<next> must be a closure.

=head4 Initialization Phase

Finally, we'll get to the iterator initialization phase, which may make
more sense now that we've gone through the other phases.  Recall that
we are using closed over variables to keep track of state.

Our code should look something like this:

  sub construct ( $array ) {

    # initialize lexical variables here
    my $next = ...;
    my $prev = ...;
    my $current = ...;
    my $arr = ...;
    my $len = ...;

    my $self = ...;

    # create our closures
    my $next = sub { ... };
    my $prev = sub { ... };
    ...

    # return our %AttrHash:
    return {
             _self => \$self,
              next => $next,
              prev => $prev,
           current => $current,
             reset => $reset,
            rewind => $rewind,
    };
  }

The first five lexical variables are easy:

  my $next = 0;
  my $prev = undef;
  my $current = undef;
  my $arr = $array ;
  my $len = $array->@*;

Now, what about C<$self>? It is a reference to our iterator object,
but the object hasn't be created yet; that's done when C<%AttrHash> is
passed to C<Iterator::Flex::Factory>. So where does C<$self> get
initialized?  The answer lies in the C<_self> entry in C<%AttrHash>,
which holds a reference to C<$self>.  When L<Iterator::Flex::Factory>
creates the iterator object it uses the C<_self> entry to initialize
C<$self>.  (Note that C<$self> is B<not> a reference to a hash.  You
cannot store data in it.)

=head3 Wrapping up

At this point C<construct> is functionally complete; given an array it'll return
a hash that can be fed to the iterator factory.

=head2 Passing the C<%AttrHash> to the factory

Iterators may be constructed on-the-fly, or may be formalized as classes.

=head3 A one-off iterator

This approach uses L<Iterator::Flex::Factory/construct_from_attrs> to create an
iterator object from our C<%AttrHash>:

  my @array = ( 1..100 );
  my $AttrHash = construct( \@array );
  $iter = Iterator::Flex::Factorye::construct_from_attrs( $AttrHash, \%opts );

In addition to C<%AttrHash>, C<construct_from_attrs> takes another options hash, which
is where the exhaustion policy is set.

In this case, we can choose one of the following entries

=over

=item *

exhaustion => 'throw';

On exhaustion, throw an exception object of class C<Iterator::Flex::Failure::Exhausted>.

=item *

exhaustion => [ return => $sentinel ];

On exhaustion, return the specified sentinel value.

=back

The default is

  exhaustion => [ return => undef ];

At this point C<$iter> is initialized and ready for use.

=head3 An iterator class

Creating a class requires a few steps more, and gives the following benefits:

=over

=item *

A much cleaner interface, e.g.

  $iter = Iterator::Flex::Array->new( \@array );

vs. the multi-liner above.

=item *

The ability to freeze and thaw the iterator

=item *

some of the construction costs can be moved from run time to compile  time.

=back

An iterator class must

=over

=item *

subclass L<Iterator::Flex::Base>;

=item *

provide two class methods, C<new> and C<construct>; and

=item *

register its capabilities.

=back

=head4 new

The C<new> method converts from the API most comfortable to your usage
to the internal API used by L<Iterator::Flex::Base>.  By convention,
the last argument should be reserved for a hashref containing general
iterator arguments (such as the C<exhaustion> key).  This hashref is
documented in L<Iterator::Flex::Base/new_from_attrs>.

The super class' constructor takes two arguments: a variable containing
iterator specific data (state), and the above-mentioned general
argument hash.  The state variable can take any form, it is not
interpreted by the C<Iterator::Flex> framework.

Here's the code for L<Iterator::Flex::Array/new>:

  sub new ( $class, $array, $pars={} ) {
      throw_failure( parameter => "argument must be an ARRAY reference" )
        unless Ref::Util::is_arrayref( $array );
      $class->SUPER::new( { array => $array }, $pars );
  }

It's pretty simple. It saves the general options hash if present,
stores the passed array (the state) in a hash, and passes both of
them to the super class' constructor.  (A hash is used here because
L<Iterator::Flex::Array> can be serialized, and extra state is
required to do so).

=head4 construct

The C<construct> class method's duty is to return a C<%AttrHash>.
It's called as

  $AttrHash = $class->construct( $state );

where C<$state> is the state variable passed to
L<Iterator::Flex::Base/new>.  Unsurprisingly, it is remarkably similar
to the C<construct> subroutine developed earlier.

There are a few differences:

=over

=item *

The signature changes, as this is a class method, rather than a subroutine.

=item *

There are additional C<%AttrHash> entries available: C<_roles>, which
supports run-time enabling of capabilities and C<freeze>, which
supports serialization.

=item *

Capabilities other than C<next> can be implemented as actual class
methods, rather than closures.  This decreases the cost of creating
iterators (because they only need to be compiled once, rather than for
every instance of the iterator) but increases run time costs, as they
cannot use closed over variables to access state information.

=back

=head4 Registering Capabilities

Unlike when using L<Iterator::Flex::Factory/construct_from_attr>,
which helpfully looks at C<%AttrHash> to determine which capabilities
are provided (albeit at run time), classes are encouraged to register
their capabilities at compile time via the C<_add_roles> method.  For
the example iterator class, this would be done via

  __PACKAGE__->_add_roles( qw[
        State::Registry
        Next::ClosedSelf
        Rewind::Closure
        Reset::Closure
        Prev::Closure
        Current::Closure
  ] );

(These are all accepted shorthand for roles in the
L<Iterator::Flex::Role> namespace.)

If capabilities must be added at run time, use the C<_roles> entry in C<%AttrHash>.

The specific roles used here are:

=over

=item Next::ClosedSelf

This indicates that the C<next> capability uses a closed over C<$self> variable,
and that C<Iterator::Flex> should use the C<_self> hash entry to initialize it.

=item State::Registry

This indicates that the exhaustion state should be stored in the central iterator Registry.
Another implementation uses a closed over variable (and the role C<State::Closure>).
See L<Iterator::Flex::Manual::Internals/Exhaustion>.

=item Reset::Closure

=item Prev::Closure

=item Current::Closure

=item Rewind::Closure

These indicate that the named capability is present and implemented as a closure.

=back

=head4 All together

  package My::Array;

  use strict;
  use warnings;

  use parent 'Iterator::Flex::Base';

  sub new {
      my $class = shift;
      my $gpar = Ref::Util::is_hashref( $_[-1] ) ? pop : {};

      throw_failure( parameter => "argument must be an ARRAY reference" )
        unless Ref::Util::is_arrayref( $_[0] );

      $class->SUPER::new( { array => $_[0] }, $gpar );
  }

  sub construct {
     my ( $class, $state ) = @_;

     # initialize lexical variables here
     ...
     my $arr = $state->{array};

     my %AttrHash = ( ... );
     return \%AttrHash;
 }

  __PACKAGE__->_add_roles( qw[
        State::Registry
        Next::ClosedSelf
        Rewind::Closure
        Reset::Closure
        Prev::Closure
        Current::Closure
  ] );

  1;

=head1 SUPPORT

=head2 Bugs

Please report any bugs or feature requests to bug-iterator-flex@rt.cpan.org  or through the web interface at: L<https://rt.cpan.org/Public/Dist/Display.html?Name=Iterator-Flex>

=head2 Source

Source is available at

  https://gitlab.com/djerius/iterator-flex

and may be cloned from

  https://gitlab.com/djerius/iterator-flex.git

=head1 SEE ALSO

Please see those modules/websites for more information related to this module.

=over 4

=item *

L<Iterator::Flex|Iterator::Flex>

=item *

L<Iterator::Flex::Manual|Iterator::Flex::Manual>

=back

=head1 AUTHOR

Diab Jerius <djerius@cpan.org>

=head1 COPYRIGHT AND LICENSE

This software is Copyright (c) 2018 by Smithsonian Astrophysical Observatory.

This is free software, licensed under:

  The GNU General Public License, Version 3, June 2007

=cut
