=head1 NAME

Net::Clacks::UpgradeGuide - Notes on upgrading L<Net::Clacks>

=head1 SYNOPSIS

This document lists important information when upgrading from previous versions of L<Net::Clacks>.

Please be sure to read all relevant sections of this short guide. While i myself don't like breaking/incompatible
changes, bugfixes and major enhancements sometimes make them the most painless way of handling things.

On the bright side, after you've gone through this document and implemented required changes (if any), you get
to enjoy a newer, shinier version of your favourite real time interprocess messaging system. While i strive to
remove bugs in each release, sometimes new ones creep in. Just keep bugging me about them until i fix them as well.

=head1 SECURITY AND BACKWARD COMPATIBILITY

WARNING: The author of this module only tests against stable Perl versions that are still considered "in maintenance".
Since <Net::Clacks> contains network code, the author of this module strongly recommends to always keep your Perl installation
as well as Net::Clacks up-to-date for security reasons.

Net::Clacks may work with older versions of Perl that are out of maintenance, but these may have unfixed security flaws.

=head1 IMPORTANT UPGRADE NOTES

=head2 VERSION 6

Version 6 (and higher) of L<Net::Clacks::Server> imlements a smarter interclacks sync. Make
sure to upgrade all nodes on your local clacks network at the same time! While the protocol itself
is mostly backward compatible, interclacks sync will fail otherwise.

Version 6 also includes a smarter shutdown sequence for L<Net::Clacks::Client> and the ability to 
persistantly store the clackscache data in a file in L<Net::Clacks::Server>. The file format has also
changed somewhat from previous beta versions of this feature due to the implementation of smarter 
sync. If you have used persistance before, you might (or might not) have to reset/remove the persistance
file.

This change to "smarter" syncing has increased stability of some of my systems using "Net::Clacks", but
has been in use for only a limited time. You should test with your own software before upgrading.

=head2 VERSION 7

Version 7 (and higher) of L<Net::Clacks::Server> and L<Net::Clacks::Client> implement a lot of
bugfixes and improvements. This includes authentication timeouts, somewhat smarter automatic reconnects
and stuff like that.

While the protocol in theory is backwards compatible to version 6, it is strongly recommended that you
upgrade ALL nodes (clients and servers) in your network at the same time. I have done only limited testing
with backward compatibility and i would not recommending a mix&match approach on critical systems.

=head2 VERSION 8

On Systems that support Unix domain sockets and have IO::Socket::UNIX installed,
L<Net::Clacks::Server> and L<Net::Clacks::Client> can now use Unix domain sockets
for local communication. See the examples for. This might or might not drastically
lower CPU usage, depending on your hardware, software, weather on moon phase.

Version 8 also includes a number of bugfixes and improvements. This includes a better
way of detecting closed/broken connection which should prevent servers and clients
from holding on to closed connection until it times out. This should also lower CPU
usage under certain circumstances. This is far from perfect, though and may lead to
some false positives (e.g. accidental closure of perfectly fine connections) thanks
to the combination of SSL and non-blocking sockets.

Version 8 is fully backwards compatible with Version 7 on the network layer (no protocol change),
but as always it is recommended to update all servers and clients at the same time if possible.

One important client API change (sort of) is the generation of messages with type "reconnected"
after a connection has been re-established. After receiving this message, a client application
must resend any LISTEN calls it wants to make. While in previous versions, this was accomplished
by checking for type "disconnect" messages, this was unreliable at best. The "reconnected" message
is generated internally B<after> a new connection has been established, except on the first ever
connection. Technically, at that point in time L<Net::Clacks::Client> has spooled the Auth request
to the server, but may not have recieved the answer yet, but i'll assume here that you have configured
your client correctly.

=head2 VERSION 9

Version 9 added the socketchmod option to chmod() the socket file so other users can connect to the socket, too.

=head2 VERSION 10

WARNING: BREAKING CHANGES! It is required to update all servers and clients at the same time.

Version 10 disables SSL on Unix Domain Sockets for better local performance. This version also adds the
ability to run interclacks through Unix Domain Sockets, in case you want to run a master/slave setup locally. This
should not really affect security, though. If an attacker has enough permissions to spy on a Unix Domain Socket,
they will most likely also have the ability to gain access to the configuration files and SSL keys of the Clacks server
running on the same host..

This version also has a slightly improved timeout handling when a slave reconnects to the master at the cost
of a bit more traffic during the initial setup phase.

In the protocol, error messages have been implemented via the "E" flag in the "OVERHEAD" command. L<Net::Clacks::Client>
forwards this to the client software via a clacks message of type "error_message".

In theory, the only *breaking* incompatible change is contained in handling Unix Domain Sockets. You could (try to) upgrade
to Version 10 client-by-client if you don't use Unix Domain Sockets, but this is neither tested nor recommended.

=head2 VERSION 11

Under certain circumstances L<Net::Clacks::Client> would hang or otherwise error out during DESTROY(). This is due to
the fact that the underlying socket might already have been DESTROYed by perl. I could reliably produce hangs in 
certain applications when calling exit(0) in the main loop.

In Version 11, the DESTROY method now uses eval() to try and catch any errors. It is recommended that you use the new
disconnect() function in L<Net::Clacks::Client> BEFORE exiting the program or deleting the last reference to the instance.
This will then try to send out any remaining commands in the output buffer. Simply exiting the program will still work,
but there is a good chance that anything still remaining in the output buffer after your last call to doNetwork() will
NOT be sent.

Version 11 also implements a Unicode/UTF8 bugfix. This should have no visible side effects in pretty much all use-cases of
L<Net::Clacks>

Upgrading to version 11 on a client-by-client basis should work fine since there have been no incompatible changes,
but this has only seen limited testing. As always, it is recommended to upgrade everything in one go.

=head2 VERSION 12

This is mostly an internal bugfix release. It now again allows in install Net::Clacks directly, instead of requiring
the user to name the full sub-module name ("Net::Clacks::Client").

In this release, a new disconnect() method was added to L<Net::Clacks::ClacksCache>. This will allow to disconnect
cleanly from the server.

=head2 VERSION 13

Before this version, L<Net::Clacks::Server> will always cache deleted keys for one week. Caching is needed for clean
reconnection of interclacks links.

But the long caching time can lead to all sorts of problems, especially with things like cached web sessions or temporary
cookies, that sort of thing. Cache time of deleted keys is really a trade off between the ability of interclacks links to
sync cleanly after long interruptions of the connection and the memory and bandwidth required to do so.

Version 13 now defaults to 1 hour (3600 seconds) of caching deleted keys. This can be changed in the
"deletedcachetime" config setting.

Syncing now also better handles timeout situations during long sync by avoiding them. Well, i hope it does. Fingers
crossed, really...

This release is backwards compatible with Version 12. But it is highly recommended to upgrade at least all server nodes
to this version as soon as possible to avoid all sorts of problems.

=head2 VERSION 14

Version 14 has only internal bugfixes and no externally incompatible changes.

The main "feature" of this release is more internal error checking. This is to detect broken connections sooner and a bit
more reliable.

=head2 VERSION 15

Version 15 adds the configuration variable "persistanceinterval" (defaults to 10 seconds for backward compatibility). This defined the
interval in which a persistance file is written. You could lower it for more frequent checkpointing, but more importantly you could
increase to longer time spans to reduce disk activity.

=head2 VERSION 16

Version 16 Server now croaks with a descriptive message if it encounters an invalid persistance file.

This version does not introduce any incompatible changes.

=head2 VERSION 17

Version 17 Server has better checks for Net::Clacks::Client call arguments and croaks or carps where appropriate.

This version does not introduce any incompatible changes but it will produce callback traces when called with missing arguments. Calling new() now
croaks/dies when arguments are missing (as it should have from the beginning). 

=head2 VERSION 18

WARNING: BREAKING CHANGES regarding persistance files.

Version 18 provides better reliability for the Clacks Server when using a persistance file, including startup problems and prevention of data loss.

Here is the problem in older versions: If the server process is killed while writing the persistance file, the file on disk may be invalid (partial data) or completely blank.
Blank files at least prevented the server from starting up, but otherwise incomplete files could lead to data corruption in cached data and/or all other sorts of problems. This
relates to the fact that the server only does very minimal checks on its internal cache during runtime for performance reasons. The original persistance file was just a quick
hack to dump/restore the cache between runs and did not account for system crashes or the process being killed while writing to disk.

This new version of L<Net::Clacks::Server> tackles this in in a multi-stage approach:

1.) The persistance file now includes the ENDBYTE string as a third line. This is used to check if the file is complete.

2.) Before overwriting the current persistance file, it is COPIED to a file with '_bck' added to the configured name.

3.) The file is written under a new name (with '_' added at the end) and then MOVED over the correct file name. File move (rename) within the same directory should be
an atomic operation of the operating system, so it either works or doesn't (no stopping half-way inbetween).

4.) An invalid persistance file doesn't prevent the clacks server startup. If it detects an invalid file, it first tries to load the previous version ('_bck'), then
as a last desperate measure to prevent data loss, it tries to load the temporary file ('_'). If that also fails, it starts 'blankety-blank'.

Clacks caching was always designed as a CACHE, not as the final storage for important data. This upgrade to Version 18 changes the file format of the persistance file slighty. The
older format is now detected as invalid, resulting in a "blankety-blank" startup. If you really want or need (?!?!?!) to preserve the cached data for some reason, there are
two ways to accomplish this:

1.) Stop the clacks server and upgrade the Net::Clacks package. Edit the persistance file and add a third line with the word "ENDBYTES" without quotes as its only content.

or

2.) If you are running a master/slave setup with interclacks, you can stop ONLY the server that has the persistance file. Then upgrade Net::Clacks and start the server. It should
start "blankety-blank" and automatically resyncronize cached data from the interclacks network. It works the same if all your clacks servers use persistance files for some weird reason,
just restart them one-by-one, waiting inbetween for the interclacks sync to finish. 
You can use the example rawclient.pl. Connect to the server you keep running and start the MONITOR command. While interclacks sync is in progress, you will see KEYSYNC commands
flying between the servers. If you stop seeing them for a few seconds, KEYSYNC is finished.

=head2 VERSION 19

Version 19 allows for more flexible config file handling. You can define the environment variable PC_CONFIG_PATHS to define the search paths for the config file and the use a the config file name without path. This is
very handy for deploying the same start scripts and config files to multiple systems with different environments and installation paths.

for example:

  export PC_CONFIG_PATHS=/home/username/src/chatserver/configs:/usr/share/chatserver/configs
  perl chatserver.pl minecraftchat.xml

This would search the config file minecraftchat.xml in /home/username/src/chatserver/configs first, and fall back to /usr/share/chatserver/configs.

Version 19 also allows you to define other environment variables starting with PC_.* and those will replace the corresponding text strings in the config file (only in memory, not on disk).

Say you want to change the TCP port where the server is listening when you are debugging. If you define the port in the config like this:

  <port>PC_CLACKSPORT</port>

then you can say
  export PC_CLACKSPORT=18889
  perl chatserver.pl minecraftchat.xml

or even

  PC_CLACKSPORT=18889 perl chatserver.pl minecraftchat.xml

This change is largely backwards compatible. If you don't change anything on your system, the only change in behaviour will be a few additional messages on STDOUT during startup.


=head2 VERSION 20

Bugfix release: Fix package name in Net/Clacks.pm

Net::Clacks is dummy class for documentation purposes, so there should be no functional changes (except to fix "cpan install Net::Clacks" and some issues with MetaCPAN)

=head2 VERSION 21

WARNING: BREAKING CHANGES! It is required to update all servers and clients at the same time.

This version implements a network-optimized change on how setAndStore works by introducing a decicated SETANDSTORE command. This halfs the network load for all
setAndStore operations, because the command and payload gets only send once instead of twice. This is especially useful on slow links with larger payloads.

This version is dedicated to Sven Guckes, who passed away on 20th February 2022. Accordingly, his name will now occasionally passed
in the overhead of the Clacks protocol.

"Do you not know that man is not dead while his name is still spoken?"
(Terry Pratchett, Going Postal)

=head2 VERSION 22

WARNING: BREAKING CHANGES! It is REQUIRED to update all servers and clients at the same time. 

Version 22 fixes some warnings and wrong print statements regarding persistance files.

This version also makes sure that the persistance file is saved on exit, to make sure we have the latest data on file, instead
of something potentially outdated.

Another important change is the way interclacks sync works. Before sending the keys to the interclacks partner, every interclacks node sends an "OVERHEAD T" timestamp. This allows the
other side to calculate (roughly) the time offset between the nodes and adjust the recieved timestamps accordingly. If the nodes are both synced to the same system time, this is pretty
much of no importance. But if one of the nodes has a large offset (think "Raspberry Pi when it can't reach a time server during boot"), syncing via time stamps can get messed up. Version 22
of Net::Clacks tries its best to rectify that. Of course, this can't replace proper time sync of the interclacks nodes, but it's better than nothing, i guess.

This version also implements "forget stale entries". That means, it removes cached entries which have not been accessed (STORE, SETANDSTORE, INCREMENT, DECREMENT, RETRIEVE) in a long time. The
default timeframe if 24 hours, but this can be configured with the "stalecachetime" parameter (in seconds). This means that the persistancefile format changes, but this will be upgraded transparently
by the software during startup. This changes the interclacks KEYSYNC command to include the last access time.

=head2 VERSION 23

This purely internal change does not break backward compatibility.

Version 23 switches Net::Clacks to use function signatures (features 'signatures'). This makes the code a bit clearer to read, properly enforces the correct number of function arguments and will
allow slimming down the argument checking in future releases.

WARNING: While this change should work with Perl 5.20 and up, the author of this module only tests against stable Perl versions that are still considered "in maintenance". As of this module release,
these are 5.32.x and 5.34.x. Since <Net::Clacks> contains network code, the author of this module recommends to always keep your Perl installation as well as Net::Clacks up-to-date for security reasons.

=head2 VERSION 24

WARNING: BREAKING CHANGES! It is REQUIRED to update all servers and clients at the same time. 

This is a bugfix and stability enhancement release.

Version 24 fixes interclacks KEYSYNC problems. Some keys failed to sync after the clacks servers were running for some time (affects versions 22 and 23, possibly earlier versions as well).
This was caused by a badly designed cache cleaning algorithm that wrongly removed the "cache time" while keeping the actual data, which in turn lead to sending out invalid
KEYSYNC commands to the interclacks partner.

I rewrote the way clacks cache entries are stored internally. This also changes the persistance file format, which will be upgraded in-place automatically (in nearly all cases).

The caching algorithm should now be much more stable and easier to maintain (fingers crossed).

The automatic cache cleaning interval can now be set by the "cachecleaninterval" config, which defaults to 60 seconds. "deletedcachetime" sets how long deleted cache entries are kept for
interclacks sync (default 1 hour), and "stalecachetime" sets how long to keep cache entries that have not been accessed (default 1 day).

This release also has a lot of code refactoring and other internal changes.

This release is dedicated to the dog Sheila, faithful companion of my friend @NightStorm_KPC. Rest in peace, Sheila.

=head2 VERSION 25

This version is (in theory) fully backward compatible. The only change is the addition of the L<Net::Clacks::PostgreSQL2Clacks> module.

This Version is dedicated to Tobias, my coworker whos software always wants to know (in real time!) if critical data has changed in the database. WITHOUT polling the database constantly.
Displaying up-to-date data to the user? What a strange and novel concept.

=head2 VERSION 26

This fixes some documentation issues.

=head2 VERSION 27

This fixes some small design issues:

* Unix domain sockets don't use SSL. If only UDS are used in a Server, there is no requirement to specify server certificates.

* runOnce() in L<Net::Clacks::PostgreSQL2Clacks> now returns the lines it would have formerly printed to STDERR. The non-returning run() still prints to STDERR.

=head2 VERSION 28

This version is (in theory) fully backward compatible (i hope).

Version 28 of Net::Clacks::Server now supports multiple users with different permissions. The primary user (same as in previous versions of Net::Clacks) is the only one
that can run Interclacks connections.

The additional users can have their permissions limited to read/write/manage (or a combination thereof). See "clacks_master.xml" in examples for details.

* read permission allows to read cached variables and listen to events.

* write permission allows to add/alter/delete cached variables and transmit events

* manage permission allows to manage the server (MONITOR, disconnect other clients, etc)

It's important to note that a write permission does NOT imply a read permission. Basically, you can have, for example, sensors that only transmit their state. They don't need the
ability to read data or listen to other events.

From Version 28 onwards, Makefile.PL and the documentation also clearly state the minimum Perl version required, which currently is v5.36. As a general rule, i only support the two
most recent stable versions of Perl, similar to the developers of Perl itself. If no new Perl feature is required, Net::Clacks will be able to run on older Perl versions as well,
but i will only test on supported versions.

L<Net::Clacks::Server> also has an additional way to call the event loop, runOnce(). This allows you to run your own event loop and just call runOnce() every few milliseconds or so.

Special thanks goes to kcott from PerlMonks for the MIN_PERL_VERSION stuff in Makefile.PL

=head2 VERSION 29

Increase the Listen queue (=how many new connections can be handled per cycle) from 1 to 20. This should increase reliability of initial startup (and reconnects after a network problem) of clients more reliable.

This version is fully backwards compatible to Version 28

=head2 VERSION 30

Support "socketcommands" in XML config. This can be used for example for changing the usergroup of a Unix Domain Socket (so multiple linux users in the same group can access the clacks UDS socket).

    <socketcommands>
        <item>chmod g+rwx /home/PC_LINUXUSER/temp/PC_PROJECTNAME_LC_clacks.sock</item>
        <item>chgrp clacksusers /home/PC_LINUXUSER/temp/PC_PROJECTNAME_LC_clacks.sock</item>
    </socketcommands>

This version is fully backwards compatible to Version 29

=head2 VERSION 31

Fix missing requirements for the "DBI" module in Makefile.PL

This version is fully backwards compatible to Version 30

=head2 VERSION 32

Fix various performance problems and reliability issues under high load.

This version is fully backwards compatible to Version 31

=head2 VERSION 33

Fix problems with MANIFEST

This version is fully backwards compatible to Version 32

=head2 VERSION 34

Net::Clacks::Client now supports "fastdisconnect" that closes a connection immediately without sending QUIT first. (used in forkin clients)

ClacksCache now only connects to server on first use, instead on instantiation.

This version is fully backwards compatible to Version 33

=head2 VERSION 35

This version retunes some timeouts that may or may not have caused problems in the past.

It now also provides more automated tests and a documentation of the protocol.

This version is fully backwards compatible to Version 34, minus the bugs i hopefully managed to squash.

=head2 VERSION 36

This version squashes some long standing bugs regarding connection management. Very busy servers should now run more stable.

This version is fully backwards compatible to Version 34, minus the bugs i hopefully managed to squash.

=head1 AUTHOR

Rene Schickbauer, E<lt>cavac@cpan.orgE<gt>

=head1 Source code repository

The official source code repository is located at:
L<https://cavac.at/public/mercurial/Net-Clacks/>

=head1 COPYRIGHT AND LICENSE

Copyright (C) 2008-2024 Rene Schickbauer

This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself, either Perl version 5.10.0 or,
at your option, any later version of Perl 5 you may have available.

=cut

