Revision history for App-Test-Generator

0.36	Sat May  9 18:54:04 EDT 2026

	[Bug fixes]
	- Fix uninitialized $version warning in CPAN Testers "no failures"
	  message when the CPAN API returns a 404 (no release data available).
	- Fix "Can't open lib/App/Test/Generator.pm" error when test-generator-index
	  is run from a repository other than ATG itself; ATG version is now
	  read from $INC{'App/Test/Generator.pm'} with fallback to the
	  configured module_file path.

0.35	Sat May  9 09:40:08 EDT 2026

	[Enhancements]
	- Dashboard footer now shows the App::Test::Generator version used
	  to generate it, linking to the MetaCPAN distribution page.
	- Executive Summary wording now varies by mutation score: positive
	  framing for high scores, an actionable hint for low scores, and a
	  clear message when no mutation data is available yet.  Thresholds
	  use med_threshold and low_threshold from %config rather than
	  magic numbers.
	- Coverage dashboard per-file HTML links now work correctly; Devel::Cover
	  instrumentation restored to use cover -test for accurate per-file
	  HTML generation.
	- Dashboard file links now correctly target blib-lib-* filenames as
	  generated by Devel::Cover, fixing persistent 404 errors on all
	  per-file coverage pages.
	- generate-test-dashboard now derives the Devel::Cover -select pattern
	  dynamically from GITHUB_REPOSITORY, making the script portable across
	  CPAN distributions without hardcoded module paths.
	- Fuzz schema generation now correctly looks in t/conf rather than
	  xt/conf for existing schemas to augment.
	- Redundant exclusion of mutant_*.t from prove invocation removed;
	  mutant stubs are in xt/ and were never matched by the t/ find anyway.

	[Bug fixes]
	- Fix https://www.cpantesters.org/cpan/report/04c7279a-476f-11f1-bf55-cb595875c975
	  Make t/type_params.t an extended test

0.34	Sun May  3 10:30:24 EDT 2026

    [Bug fixes]
    - Emitter::Perl: _emit_void_test() generated ok(!defined $result || 1, ...)
      which is a tautology and always passes regardless of return value.
      Fixed to ok(!defined $result, '$method returns nothing (void)').
    - Generator: schema files named after Perl builtins (e.g. abs.yml) caused
      generated tests to emit "use abs" which fails to compile.  Added
      _is_perl_builtin() guard to prevent builtin names being used as module
      names in use_ok() calls.
    - bin/test-generator-index: mutant stub generator incorrectly called
      new_ok() for class methods such as App::Test::Generator::generate().
      Added _is_class_method() helper to detect $class vs $self and emit
      the correct calling convention in generated stubs.
    - bin/test-generator-index: CPAN Testers failure table showed all
      expected NA rows for Perl < 5.036 as noise.  Rows below the detected
      Perl version cliff are now omitted with a count notice, leaving only
      unexpected failures visible.
    - CoverageGuidedFuzzer: _load_json_module() used require $mod where $mod
      contained '::' which Perl does not convert to path separators for
      variable require.  Fixed with s{::}{/}g conversion before require.

    [Enhancements]
    - Comprehensive test suite added across all modules:
      - Function tests (white-box) for all private helpers
      - Black-box unit tests for every public method against POD API spec
      - Integration tests covering 10 end-to-end pipelines
      - Edge case / destructive / boundary-condition tests
      - Extended tests targeting surviving mutants and coverage gaps
    - mutate.yml and dashboard.yml are now portable to any CPAN module.
      Copying both files to another module's .github/workflows/ directory
      will produce a working mutation test run and coverage dashboard
      without modification.  ATG-specific behaviour is gated on
      github.repository = nigelhorne/App-Test-Generator.
    - dashboard.yml: added missing local/bin PATH setup so installed
      ATG scripts are found correctly on non-ATG modules.
    - mutate.yml: Sample/Module.pm excluded from mutation scoring to
      remove 30 artificial survivors from the overall score.
    - Emitter::Perl mutation score improved from 37.5% to 100% through
      targeted direct assertions on all _emit_*() helper return values.
    - t/test-generator-index.t: 33 new tests for detect_perl_version_cliff(),
      parse_version(), extract_perl_versions(), make_key(),
      confidence_score(), and perldelta_url().

0.33	Tue Apr 21 16:09:24 EDT 2026

    - Fixed NumericBoundary mutator incorrectly identifying the '<' in
      open() mode strings and readline operators (<$fh>) as numeric
      comparison operators, producing invalid mutants that caused
      compilation errors in the mutated code. Two guards added to
      mutate(): skip any operator whose next sibling is a PPI symbol
      token (catching <$fh>), and skip operators not parented by a
      PPI::Statement, PPI::Structure::Condition, or
      PPI::Structure::Block (catching quoted mode strings in open()
      argument lists). The same readline guard is also applied inside
      the transform closure to prevent mutation at apply time if PPI
      tokenisation differs between the analysis and mutation documents.
    - Rationalised program names so that they can be used in other modules
      without the need to have copies
    - Added detect_prereq_version_cliffs() to cross-reference declared
      prerequisites from the MetaCPAN API against installed module
      versions reported by CPAN Testers. Flags cases where failing
      reports have an installed version below the declared minimum,
      surfacing dependency version mismatches (e.g. Test::Mockingbird
      0.02 installed where >= 0.03 is required) that the existing
      detect_version_cliffs() misses because it only examines prereq
      metadata within the reports themselves. Falls back to scanning
      raw CPAN Testers report output for version complaint patterns if
      the MetaCPAN API is unavailable. Results are merged into the
      existing Dependency Version Cliffs section alongside findings
      from detect_version_cliffs(). Added report HTML caching in
      fetch_report_html() to avoid fetching the same report twice when
      both aggregate_dependency_stats() and detect_prereq_version_cliffs()
      process the same GUIDs.
    - bin/fuzz-harness-generator: fixed shell injection risk in final
      system() call, replaced string eval with block eval for JSON
      module loading, added explicit use File::Spec, normalised if()
      spacing throughout, added structured comment blocks for all four
      helper functions, added YAML input validation before calling
      generate()
    - bin/generate-test-dashboard: removed duplicate cover -report json
      step that was running twice per dashboard build
    - bin/test-generator-index: added structured comment blocks for six
      private analysis functions; fixed detect_universal_failure() to
      partition genuine FAILs from NAs before counting, returning undef
      when all reports are NA so detect_perl_version_cliff() can
      diagnose correctly
    - SchemaExtractor.pm: added structured comment blocks for all private
      methods (Pass 3)
    - App::Test::Generator: added relationships_code emission so schemas
      containing relationship metadata produced by SchemaExtractor are
      serialised into the generated test file
    - Template test.tt: added semantic type test cases for filepath
      (including Windows paths, reserved device names, UNC paths),
      email, date_string, and iso8601_string; added relationship test
      loop covering all six relationship types (mutually_exclusive,
      required_group, conditional_requirement, dependency,
      value_constraint, value_conditional); added float edge cases for
      Inf, -Inf, and NaN with correct status handling based on whether
      max is defined

0.32	Sun Apr 12 08:16:47 EDT 2026

    - Added detect_scattered_failures() root cause detector. Surfaces a
      weak-confidence advisory when failures and passes coexist across
      2 or more common Perl series with no detectable version cliff or
      OS pattern, suggesting flaky tests, optional dependency
      differences, or CGI environment assumptions rather than a
      compatibility issue. Complements detect_universal_failure() which
      handles the opposite case of near-total failure. Confidence is
      intentionally set to 0.40 (Weak) since this is a catch-all
      signal rather than a precise diagnosis, ensuring it appears below
      stronger signals in the root causes table when multiple detectors
      fire simultaneously.
    - Added detect_universal_failure() root cause detector. Surfaces a
      high-confidence warning when failures occur across 3 or more
      distinct Perl versions and 2 or more OS types with fewer than
      10% passing reports, indicating a likely broken release rather
      than a version- or platform-specific compatibility issue. Likely
      causes listed in evidence: missing file in tarball, broken
      Makefile.PL, or undeclared dependency. Integrated into
      detect_root_causes() where it is evaluated first and sorted by
      confidence alongside the existing OS, locale, and Perl version
      cliff detectors.
    - Fixed blib/ paths appearing in coverage table instead of lib/
      paths. Devel::Cover instruments blib/ during testing; paths are
      now normalised to lib/ for display, with deduplication against
      any native lib/ entry.
    - Fixed structural coverage percentages in Executive Summary and
      Structural Coverage sections showing ~24% instead of ~93%.
      _coverage_totals now aggregates from individual own-project files
      rather than Devel::Cover's pre-aggregated Total key which
      includes all instrumented CPAN dependencies.
    - Fixed cyclomatic complexity badge colour and tooltip inverted.
      High complexity now correctly shows red/Needs improvement;
      low complexity shows green/Good. Second condition also fixed
      to use $complexity rather than $score.

0.31	Fri Apr 10 08:07:40 EDT 2026

    - Added TER3 (LCSAJ path coverage) column to mutation files table in
      the test dashboard index, showing percentage and raw fraction
      (e.g. "71.8% (352/491)")
    - Added TER1, TER2 and TER3 metrics to per-file mutation report pages,
      replacing the plain "Statement" and "Branch" labels with their
      formal Test Effectiveness Ratio names
    - Renamed the LCSAJ column header in the mutation files table to TER3
    - With min set to zero on an integer, sometimes negative or floats could be created - added abs() and int() calls as needed
    - Added --generate_mutant_tests=DIR option to generate_index.pl to
      produce a timestamped test stub file (t/mutant_YYYYMMDD_HHMMSS.t)
      for surviving mutants. High/Medium difficulty survivors get TODO
      test stubs; Low difficulty survivors get comment-only hints.
      Multiple mutations on the same source line are deduplicated into
      one stub listing all variants — one good test kills them all.
      Boundary value suggestions are generated for numeric mutations,
      clamped and deduplicated for non-negative contexts such as
      scalar() and length(). Environment variable hints are added where
      the source line references $ENV{...}. The enclosing subroutine
      name is shown in each stub for navigation context. File is skipped
      if there are no survivors or low-difficulty hints to report.
    - LCSAJ path dots on per-file mutation pages are now coloured blue
      (covered) or red (not covered), based on whether any line in the
      path range was executed during testing. Uncovered paths show
      [NOT COVERED] in the hover tooltip. The LCSAJ legend is updated
      to explain both colours.
    - Replaced TER3-only column in mutation files table with a
      TER1 / TER2 / TER3 triple, each component shown as a
      colour-coded badge (green/yellow/red). TER1=Statement,
      TER2=Branch, TER3=LCSAJ path coverage. Any component
      without data shows a grey n/a badge. Column header carries
      a tooltip defining all three metrics.
    - Added --generate_test=mutant option to generate_index.pl (used
      alongside --generate_mutant_tests=DIR). For NUM_BOUNDARY survivors,
      attempts to produce a runnable YAML schema file in t/conf/ using
      App::Test::Generator::SchemaExtractor rather than a TODO stub.
      The schema is augmented with boundary values from the surviving
      mutant (the exact boundary value plus one either side) and picked
      up automatically by t/fuzz.t on the next test run. Falls back to
      a TODO stub if SchemaExtractor fails, the enclosing sub cannot be
      found, or confidence in the extracted schema is too low. The
      --generate_test option is designed to accept future classes beyond
      'mutant'. Updated generate_test_dashboard Step 7 to pass
      --generate_test=mutant by default.
    - Made output_dir optional in App::Test::Generator::SchemaExtractor
      new() -- it is now only required if schema files will be written.
      extract_all() gains a no_write option to suppress file output and
      return schemas only, for use by callers that want to inspect or
      augment schemas before deciding where to write them.
    - Added --generate_fuzz flag to generate_index.pl. Scans t/conf/
      for existing YAML schema files and writes timestamped augmented
      copies (mutant_fuzz_YYYYMMDD_HHMMSS_FUNCTION.yml) with boundary
      values from surviving NUM_BOUNDARY mutants merged in. The original
      schema is never modified. Augmented schemas are picked up
      automatically by t/fuzz.t. Schemas with no matching survivors are
      skipped (with a verbose note). Boundary values are merged into
      whichever edge key already exists in the schema (edge_case_array
      or edge_cases), with deduplication. Schemas already prefixed with
      mutant_fuzz_ are skipped to prevent cascading augmentation.
      Updated generate_test_dashboard Step 7 to pass --generate_fuzz.

0.30	Thu Apr  2 07:17:16 EDT 2026
	Added mutation levels.
		Setting to fast will reduce the number of mutants by deduping and removing unnecessary mutants.
		See App::Test::Generator::Mutator::_is_redundant_level for a list of those optimised out
	Added basic LSCAJ data to the test dashboard.
	Added simple security string testing.
	Don't push too hard on builtins as they don't have good parameter validation.
	Ensure adding only the byte order marker honours $min
	More use of _DESCRIPTION

0.29	Thu Feb 26 12:57:59 EST 2026
	Added mutation testing
	Added guided testing to extract-schemas
	Some routines were incorrectly labelled as getter routines
	Added more edge cases
	Added fallback to extract parameters from classic Perl body styles
	Added Type::Param support (https://github.com/nigelhorne/App-Test-Generator/issues/4)
	Getter routines take no arguments
	Fixed string testing when both min and max are given
	Don't give $class or $self as parameters
	No input/output no longer croaks, because there are now a few tests that can be run
	Add a basic hashref to mandatory args
	Use UUID::Tiny and Readonly::Values::Boolean

0.28	Mon Feb  9 19:54:59 EST 2026
	Latest test dashboard
	Some getter/setter routines were missed
	Test getset routines
	Added 'isa' test.  Tests code dies when passed the wrong type of object

0.27	Thu Jan 29 07:54:01 EST 2026
	Fewer false positives for "optional"
	Sometimes "new:" was missing
	If a return value is 'length($foo)' it's an integer with a minimum value of 0
	Some random strings were mistakenly sent as tests that were not in the memberof set

0.26	Mon Jan 12 07:57:54 EST 2026
	Added strict_pod mode to SchemaExtractor, which croaks if it finds discrepancies between the POD and code
	Improved detection of optional/required arguments
	Improved detection of integer output types
	Improved POD parser of parameters
	Added --dry-run
	Improved config validation
	Better call of object methods in transforms
	Test the sanity of Unicode outputs
	Idempotent tests
	Prevent silent duplicate method overwrites
	Implement confidence_threshold
	Refactor _detect_accessor_methods
	Only generate one of call_code and position_code
	Check for global side effects
	Sanity test on DIES
	Added timeout tests to ensure the test doesn't hang

0.25	Thu Jan  1 15:41:06 EST 2026
	Ternary ? 1 : 0 return is taken to be an indicator of boolean returns
	Now parses this pod to know that it's supposed to return a string:
		Returns the sanitized HTML string
	Use :: to call methods in non-oo modules, rather than ->
	Flag when type is set to object by can is not set

0.24	Sun Dec 28 15:10:09 EST 2025
	Return chances of false positive file path semantics
	If the type of an variable can't be determined, guess at string, but lower the confidence level
	Error Message Extraction:
		Capture error messages from die/croak/confess
		Use them to understand what makes input invalid die "Age must be positive" if $age <= 0;
		Store that a negative/zero age is invalid
	Use test of scalar(@_) to check for number of optional/mandatory args
	Throw a float at a routine that only takes integers - it should error
	Added example extraction
		Parse POD for example calls and extract valid parameter combinations =head2 SYNOPSIS my $obj = Module->new(foo => 'bar', count => 5);
		Store these as known-good test values
	Corpus bool testing now uses ok rather than is
	Ensure new doesn't refer to other packages
	Set the isa property in the new function

0.23	Wed Dec 24 10:48:03 EST 2025
	Fix https://www.cpantesters.org/cpan/report/079dba46-d92f-11f0-9642-dde56d8775ea
	Don't give false negative that memberof isn't known
	SchemaExtractor output files now end in .yml not .yaml
	Ensure rand_str isn't called when memberof is set
	When test_empty is set, do the right thing for memberof

0.22	Sun Dec 21 20:59:58 EST 2025
	Added better API to generate
	Added parameter relationship detection (mutually exclusive, required groups, conditional requirements, dependencies, value constraints)
	Improved Object Detection
		Detect factory methods that return instances
		Recognize singleton patterns
		Identify when constructor needs specific parameters
		Handle inheritance (when parent class new() is needed)
	Better Default Value Extraction
		$param = $param || 'default_value'
		$param //= 'default_value'
		$param = defined $param ? $param : 'default'
	Expanded _detect_list_context to
		Better distinguish scalar vs list returns
		Detect void context methods (those that modify state)
		Recognize chaining patterns more reliably Identify error return conventions (undef on failure, etc.)
	Add confidence scoring transparency, explain WHY confidence is what it is:
		_analysis: confidence_factors:
			- "POD documentation present (+30)"
			- "Type validation in code (+20)"
			- "No constraint information (-10)"
	Add support for:
		- Signatures (sub foo($x, $y = 5) { })
		- Postfix dereferencing ($array->@*)
		- Subroutine attributes (sub foo :Returns(Int) { })
		- Field declarations (Perl 5.38+)
	Added a new section to generated YAML:
		yamltest_hints: boundary_values: [0, 1, 100, 255]
		detected from code invalid_inputs: ['', undef, -1]
		from validation checks equivalence_classes: []

0.21	Sun Dec 14 08:07:09 EST 2025
	Schemaextractor: don't put the package name as the argument
	Validate config settings better
	Fix max string testing with non-ASCII characters
	Changed rand_str to be a unified generator that randomly produces codepoint strings,
		grapheme clusters, ZWJ emoji sequences, or aggressive Unicode fuzz strings
	Schemaextractor: Added advanced type detection for DateTime objects, file handles, coderefs, and enum validation patterns
	Added enum as a synonym of memberof
	Added tests for unix_timestamp semantic type

0.20	Fri Dec  5 07:53:43 EST 2025
	Added the --version flag to fuzz-harness-generator
	Ensure the max value of string is honoured better
	Fix array context detection to only match return statements
	Improve chances of detecting a boolean output
	Make the list context detection more specific

0.19	Wed Nov 26 07:50:22 EST 2025
	Fixed the loop iterating over mandatory args
	Ensure mandatory_args honours matches

0.18	Tue Nov 25 11:58:12 EST 2025
	Removed one place sending '' if test_empty was disabled
	Don't send long strings if matches is set
	Fixed handling of position code with more than argument
	Renamed the sample module

0.17	Tue Nov 25 10:24:36 EST 2025
	Improved TEST_VERBOSE output for Corpus tests
	Make the strings more random
	Added test_non_ascii (default 1) to the configuration matrix
	Removed all legacy conf code
	Re-ordered the POD
	Don't give 42 if max < 42
	Added App::Test::Generator::SchemaExtractor

0.16	Thu Nov 20 08:38:17 EST 2025
	Use Data::Random::String to generate the string
	Use Data::Random::Structure to generate references to hashes and arrays
	Remove legacy support for Perl config files
	Handle the case when all inputs should cause a die(),
		allows App::Test::Generator to sanity test itself
	If something is expected to die, don't look at its return code
	More boolean edge cases
	Don't pass _NAME to validate_strict
	When a routine dies, it shouldn't return anything
	Begin to use semantic types
	Use Test::LectroTest to create more tests
	Added custom properties
	Separated the Template into its own class to ease maintenance
	Use reusable functions to generate cases
	Rand_set returns and array, so look at the first element

0.15	Tue Nov 11 19:30:18 EST 2025
	Documented how to Fuzz test a CPAN module
	populate_positions - don't attempt to deref a scalar
	carp when undef is a corpus test case, but test_undefs is not set
	Improved handling of cases array for WARNS and DIES
	Improved handling of validating position settings in config files
	Croak if the schema_file can't be opened
	Close stdin and use /dev/null
	Fixed logic in testing short strings when min >= 2

0.14	Fri Nov  7 20:26:32 EST 2025
	Fixed logic when neither memberof nor notmemberof were defined,
		which could cause an empty notmemberof array to be created
	Transform: better number of items for arrayrefs
	At least one of module and function must be defined
	Don't send wrong data types in transform testing built ins
	Document the edge_cases_array and ensure all tests in that array are run
	Better handing of "input: undef" in the config file
	Fix handling of matches in output schemas

0.13	Tue Nov  4 11:52:49 EST 2025
	Added better documentation for transforms
	transform: filled in more types for the foundation set of data
	Added "value" to the transform keywords
	Refactored and enabled the code to validate that $module exists and is installed
	Added deprecation notice about loading configs from perl code
	Allow yes/no as booleans in schema config settings
	Fix sprintf issue when the text contains a % sign

0.12	Sun Nov  2 19:38:10 EST 2025
	undef can now be used to indicate no input or output
	Sending with no args if now configured using "test_undef" rather than "test_empty"
	Use Data::Random::String::Matches to create random string that matches a regex
	Flag keywords that are yet to be handled
	$module can now be set to "builtin"
	Initial basic support for positional arguments and transforms

0.11	Sun Oct 26 14:39:19 EDT 2025
	Only add to candidate_bad those test cases which are requested
	Better matches and nomatch tests
	Only load JSON module when dedup is enabled
	Added the -o and -r options to bin/fuzz-harness-generator

0.10	Wed Oct 22 09:23:38 EDT 2025
	Send wrong data types - should die, e.g. string to an integer field
	Fix syntax error when Math::Simple is installed
	Added test_empty config (default is on)

0.09	Sun Oct 19 21:14:29 EDT 2025
	Better handling of false/true in config files
	Better handling of regex into the input structure

0.08	Thu Oct 16 20:44:22 EDT 2025
	Allow $new to be set but not defined, this will call new() with no arguments
	Started to add support for "nomatch"
	Allow specs that specify no input if they have output
	If a routine takes no input, it won't die if it has no input

0.07	Tue Oct 14 16:36:33 EDT 2025
	Bump minimum versions
	Added float that was missing some places
	Use Data::Random
	Fix GitHub#3
	Show how to use this module to automatically schedule random tests
	Test for code doing if($string) rather than if(defined($string)) which is confused if $string is '0'
	Allow the special word "undef" in the YAML specification for output

0.06	Fri Oct 10 08:55:36 EDT 2025
	Allow real config files to be read
	Validate the configuration file

0.05	Thu Oct  9 18:23:03 EDT 2025
	Removed duplicate regex candidate table
	Validate that the corpus inputs are arrayrefs
	Only load Class::Simple when needed
	Use rel2abs
	Load the configuration file in a (slightly) safer way
	The test code is now in a template toolkit
	Improved random test generator, more knowledgable about min/max, memberof and matches

0.04	Wed Oct  8 08:39:49 EDT 2025
	Add fuzzy regex generator
	Do basic hard-coded tests where possible, to get it started
	Allow pathnames in the module name
	Added qwrap - GitHub#1 - thanks to neo1ite
	Don't try to fuzz input if no %input is given
	Added %config - GitHub#2
	Generate tests for routines that take one unnamed parameter - GitHub#2
	Added fallback for perl_quote for hashes and objects

0.03	Mon Sep 29 18:18:36 EDT 2025
	If minimum is not set, verify 0 or empty fields are allowable
	Added the testing dashboard
	Fixed handling of memberof in input/output array creation
	If TEST_VERBOSE is set, print the generated dataset when running it
	Always ensure mandatory strings are passed when testing other arguments
	rand_int and rand_numb now also sometimes return very large and very small numbers
	Put utf-8 and NUL bytes into strings

0.02	Sun Sep 28 09:03:49 EDT 2025
	Use gtar on OS/X to generate the distro
	Added edge case test generator for booleans and memberof

0.01	Sun Sep 28 08:43:35 EDT 2025
	First draft
