package Data::HyperLogLog::Shared;
use strict;
use warnings;
our $VERSION = '0.04';
require XSLoader;
XSLoader::load('Data::HyperLogLog::Shared', $VERSION);

sub CLONE_SKIP { 1 }  # blessed C-pointer handle: never clone into ithreads (double-free)
1;
__END__

=encoding utf-8

=head1 NAME

Data::HyperLogLog::Shared - shared-memory HyperLogLog cardinality estimator for Linux

=head1 SYNOPSIS

    use Data::HyperLogLog::Shared;

    # default precision 14 (16384 registers, ~0.8% std error), anonymous mapping
    my $hll = Data::HyperLogLog::Shared->new;

    $hll->add("alice");
    $hll->add("bob");
    $hll->add("alice");                 # already seen: no register increases

    my $n = $hll->count;                # ~2 (estimated distinct items)

    # bulk add in a single lock acquisition
    my $new = $hll->add_many([ map { "user-$_" } 1 .. 1000 ]);

    # merge another estimator of equal precision (register-wise max)
    my $other = Data::HyperLogLog::Shared->new;
    $other->add_many([ map { "user-$_" } 500 .. 1500 ]);
    $hll->merge($other);
    my $union = $hll->count;            # ~1500 distinct across both

    # share across processes via a backing file
    my $shared = Data::HyperLogLog::Shared->new("/tmp/visitors.hll", 14);

    # freeze and ship: query it read-only (lock-free) on other machines
    $shared->freeze;
    my $ro = Data::HyperLogLog::Shared->new_readonly("/tmp/visitors.hll");
    $ro->count;

=head1 DESCRIPTION

A HyperLogLog estimator in shared memory: it counts the number of B<distinct>
items it has seen (a probabilistic distinct-count) in a fixed, tiny amount of
memory, regardless of how many items pass through it. It never stores the items;
it keeps only an array of C<m = 2**precision> single-byte registers (16 KB at the
default precision 14) plus a fixed ~16 KB region for the cross-process reader
table, so the whole mapping is about 32 KB at precision 14 (see C<mmap_size> in
L</STATS>). It yields a relative standard error of roughly B<0.8%>.

Each item is hashed with XXH3; the top C<precision> bits of the hash select a
register, and the position of the first set bit in the remaining bits is folded
into that register as a running maximum. The harmonic mean of the registers,
with a small-range linear-counting correction, gives the estimate.

Because the register array lives in a shared mapping, B<several processes share
one estimator>: any process that opens the same backing file, inherits the
anonymous mapping across C<fork>, or reopens a passed memfd, sees the others'
additions and contributes its own. A write-preferring futex rwlock with
dead-process recovery guards mutation, so many processes may C<add> and C<count>
concurrently. Two estimators of equal precision can be combined with C<merge>
(register-wise max), which gives the cardinality of the B<union> of their item
sets without double-counting overlaps.

Items are added by their B<byte> content (encode wide/utf8 strings first).
B<Linux-only>. Requires 64-bit Perl.

=head1 METHODS

=head2 Constructors

    my $hll = Data::HyperLogLog::Shared->new($path, $precision);
    my $hll = Data::HyperLogLog::Shared->new;                       # anonymous, precision 14
    my $hll = Data::HyperLogLog::Shared->new(undef, 16);            # anonymous, precision 16
    my $hll = Data::HyperLogLog::Shared->new_memfd($name, $precision);
    my $hll = Data::HyperLogLog::Shared->new_from_fd($fd);
    my $ro  = Data::HyperLogLog::Shared->new_readonly($path);       # frozen file, read-only

C<$path> is the backing file (C<undef> or omitted for an anonymous mapping).
C<$precision> is optional and defaults to B<14>; it must be between B<4> and
B<18> (C<new> croaks otherwise). The register count is C<m = 2**precision>
(precision 4 -> 16 registers, 18 -> 262144), and the relative standard error
is about C<1.04 / sqrt(m)> (precision 14 -> ~0.81%, 16 -> ~0.41%, 12 ->
~1.63%). When reopening an existing file or memfd, the stored precision wins
and the caller's argument does not change it -- but it is still range-checked,
so an out-of-range value croaks. C<new_memfd> creates a Linux memfd
(transferable via its C<memfd> descriptor); C<new_from_fd> reopens one in
another process. The descriptor you pass is duplicated (C<F_DUPFD_CLOEXEC>),
so it stays yours to close and closing it does not disturb the handle.
C<new_readonly> opens a B<frozen> file read-only for lock-free querying (see
L</"FROZEN (READ-ONLY) MODE">).

=head2 Adding and counting

    my $bumped = $hll->add($item);          # 1 if a register increased, else 0
    my $added  = $hll->add_many(\@items);   # count of items that increased a register
    my $n      = $hll->count;               # estimated distinct items (rounded)
    $hll->clear;                            # reset to empty (count -> 0)

C<add> hashes C<$item> (taken by its bytes; wide characters croak, encode first)
and updates the relevant register, returning 1 if that register increased and 0
otherwise. A return of 0 does B<not> mean the item was seen before -- it means
its hash did not beat the current register value; treat the return only as "did
this change the sketch". C<add_many> takes an array reference and does the whole
batch under a single write lock, returning how many of its elements increased a
register. C<count> returns the current estimate rounded to the nearest integer.

=head2 Merging

    $hll->merge($other);

Folds C<$other>'s registers into C<$hll> by register-wise maximum, so C<$hll>
then estimates the cardinality of the B<union> of the two item sets. Both
estimators must have the same precision (C<merge> croaks on a mismatch).
C<$other> is read under its own lock into a private snapshot first, so merging
is deadlock-free even if two processes merge each other concurrently; C<$other>
is not modified.

=head2 Introspection and lifecycle

    $hll->precision; $hll->registers; $hll->count; $hll->stats;
    $hll->path; $hll->memfd; $hll->sync; $hll->unlink;   # or Class->unlink($path)

C<precision> is the configured precision; C<registers> is the register count
C<m> (C<2**precision>). C<sync> flushes the mapping to its backing store (a
no-op for anonymous and memfd estimators, which have none); C<unlink> removes
the backing file (also callable as C<< Class->unlink($path) >>); C<path> returns
the backing path (C<undef> for anonymous, memfd, or fd-reopened estimators) and
C<memfd> the backing descriptor -- the memfd of a C<new_memfd> estimator or the
dup'd fd of a C<new_from_fd> estimator, and -1 for file-backed or anonymous
estimators.

=head1 STATS

C<stats()> returns a hashref: C<precision>, C<registers> (the register count
C<m>), C<count> (the current rounded estimate), C<ops> (running count of
mutating operations -- C<add>, C<add_many>, C<merge>, C<clear>), C<mmap_size>
(bytes of the shared mapping), C<frozen> (1 if the estimator has been sealed by
C<freeze>, else 0), and C<readonly> (1 if this handle is a read-only view, from
C<new_readonly> or the handle that called C<freeze>, else 0).

=head1 ACCURACY

The relative standard error is approximately C<1.04 / sqrt(m)> where
C<m = 2**precision>. At the default precision 14 that is about 0.8%; roughly two
thirds of estimates fall within one standard error of the truth and almost all
within three. For very small cardinalities (when the estimate is at or below
C<2.5 * m> and some registers are still zero) the estimator switches to linear
counting, which is accurate in that range. There is no large-range (2^32)
correction: with 64-bit hashes the hash space is effectively collision-free for
any cardinality this is used for.

=head1 SHARING ACROSS PROCESSES

The estimator lives in a shared mapping, shared the same three ways as the rest
of the family: a B<backing file> (every process calls C<< new($path, ...) >> on
the same path), an B<anonymous mapping inherited across C<fork>>, or a B<memfd>
whose descriptor is passed to an unrelated process (over a UNIX socket via
C<SCM_RIGHTS>, or via C</proc/$pid/fd/$n>) and reopened with
C<< new_from_fd($fd) >>. Because the mapping is shared, B<every process adds into
and reads from the same set of registers>, so the cardinality reflects the union
of what all of them have seen.

    # producer and consumer share one estimator with no coordination
    my $hll = Data::HyperLogLog::Shared->new;            # before fork
    unless (fork) { $hll->add_many([ map { "ev-$_" } 1 .. 1000 ]); exit }
    wait;
    print $hll->count, "\n";   # ~1000, counting the child's additions

=head1 FROZEN (READ-ONLY) MODE

A file-backed estimator can be B<frozen> and then shipped to other machines,
where consumers open it B<read-only> and query it with B<no locking at all>.

    # producer: build, freeze, ship the file
    my $hll = Data::HyperLogLog::Shared->new("/tmp/visitors.hll", 14);
    $hll->add_many(\@known);
    $hll->freeze;                 # seal: now immutable, and $hll itself is read-only
    # ... copy /tmp/visitors.hll to another host ...

    # consumer (any process, same architecture): read-only, lock-free
    my $ro = Data::HyperLogLog::Shared->new_readonly("/tmp/visitors.hll");
    $ro->count;

C<freeze> takes the write lock, marks the estimator B<permanently immutable>
(there is no unfreeze -- rebuild the file to change it), and flushes the seal
to disk. A frozen estimator rejects every mutator (C<add>, C<add_many>,
C<merge>, C<clear>) with a croak, and a read-write reopen
(C<< new($path, ...) >>) of a sealed file is B<refused> -- so a shipped
artifact can never be silently mutated out from under its readers.

C<new_readonly($path)> maps the file C<O_RDONLY> / C<PROT_READ> and B<requires
it to be frozen> (it croaks on a file that was never C<freeze>d). Because a
sealed estimator's registers and geometry are immutable, C<count> and C<stats>
read them B<directly, taking no reader lock> -- the mapping is never written,
so a read-only view works from a read-only file descriptor or a read-only
filesystem, and any number of processes can share one C<PROT_READ> mapping.
C<frozen> and C<readonly> report the two states.

B<Portability.> The on-disk format is native binary (native-endian 64-bit
words), so a frozen file may be copied only between machines of the B<same
architecture>; a wrong-endian file is rejected at open by the magic check.
B<Copy the file to each consumer> -- do not share one file over a network
filesystem: the lock is a Linux futex (process-local to one kernel), and the
"no live writer" contract assumes a static copy. Linux-only; 64-bit Perl.

=head1 SECURITY

Backing files are created with mode C<0600> (owner-only) by default, so only
the creating user can open and attach them. To share a backing file across
users, pass an explicit octal file mode such as C<0660> as the last argument
to C<new>; the mode is applied when the file is created, and when a file left
behind by an interrupted create is re-initialized (see L</CRASH SAFETY>); a
file already in use keeps its own permissions. The file is opened with
C<O_NOFOLLOW>, so a symlink planted at the path is refused, and created with
C<O_EXCL>; the on-disk header is validated when the file is attached. Any
process you grant write access to a shared mapping is trusted not to corrupt
its contents while other processes are using it.

=head1 CRASH SAFETY

Mutation is guarded by a futex-based write-preferring rwlock with PID-encoded
ownership; if a holder dies, the next contender detects the dead owner and
recovers. Each register update is a single byte store, so a crash leaves the
estimator consistent up to the last completed C<add>.
B<Limitation>: PID reuse is not detected (very unlikely in practice).

Reader-slot exhaustion (slotless readers): dead-process recovery attributes a
crashed lock holder's contribution through its reader-slot. The slot table holds
1024 entries (one per concurrent reader process). If more than that many reader
processes share one mapping at once, a reader that cannot claim a slot proceeds
"slotless" -- it still takes the read lock but leaves no per-process record. If
such a slotless reader is then killed while holding the read lock, its share of
the lock cannot be attributed to a dead process, so writer recovery cannot
reclaim it and writers may block until the mapping is recreated. Reaching this
needs more than 1024 concurrent reader processes on one mapping plus a crash in
the brief read-lock window; the dead-process slot reclaim keeps the table from
filling with stale entries, so in practice it is very unlikely.

An interrupted create is recovered too. A creator killed after the backing
file is sized but before its header is committed leaves a full-size, all-zero
file. C<new> re-initializes such a file automatically, but only when it is
exactly the size the requested geometry needs, is owned by your effective uid,
and is still entirely zero -- a file holding data is never re-initialized. If
the creator got as far as writing part of the header, the file cannot be told
apart from a corrupt one and C<new> croaks with C<incomplete HyperLogLog file
left by an interrupted create; remove it and retry>. A file left behind by an
interrupted create never held data, so removing it is safe -- but a file whose
header was corrupted after the fact reaches the same croak, so confirm it is
an abandoned create before deleting anything you care about.

=head1 SEE ALSO

L<Data::Intern::Shared>, L<Data::SortedSet::Shared>,
L<Data::SpatialHash::Shared>, and the rest of the C<Data::*::Shared> family.

=head1 AUTHOR

vividsnow

=head1 LICENSE

This is free software; you can redistribute it and/or modify it under the same
terms as Perl itself.

=cut
