[ablaom]

This PR is a major breaking update implementing the proposal at #64.

It additionally closes: #47, #59, #62 and replaces #66, #68, #70, #71.

Detailed document strings have been added to all public methods and New Documenter.jl documentation replaces the README.md docs. The demos in /examples are now integrated into the new documentation.

Draft release notes below tag breaking changes. The main change is to the output of train! which will be a vector of new ControlTrace objects, which, by default, record a complete history of the internal state of each control with intuitive API. Here's a demonstration:

using IterationControl
import IterationControl.SquareRooter  # demonstration "model" type
using DataFrames

model = SquareRooter(9)

julia> traces = IterationControl.train!(
    model,
    Step(),
    Threshold(0.01),
    WithLossDo(x->x<0.01),
)
[ Info: final loss: 0.0005493331704400362
[ Info: final training loss: 0.02343785763332562
[ Info: Stop triggered by Threshold(0.01) stopping criterion.
3-element Vector{Any}:
 ControlTrace{<:Step}, ending (new_iterations = 4, new_steps = 4)
 ControlTrace{<:Threshold}, ending (loss = 0.0005493331704400362, excess = -0.009450666829559964)
 ControlTrace{<:WithLossDo}, ending (loss = 0.0005493331704400362, called_back = true)

julia> trace = traces[end]
ControlTrace
  control: WithLossDo{var"#2#3"}(var"#2#3"(), false, nothing)
  loss: [ …, 2.5599999999999987, 0.1417301038062284, 0.0005493331704400362]
  called_back: Bool[ …, 0, 0, 1]
  table: (loss = …, called_back = …)
  triggered_stop: false

julia> trace.loss
4-element Vector{Float64}:
 16.0
  2.5599999999999987
  0.1417301038062284
  0.0005493331704400362

julia> DataFrame(trace)
4×2 DataFrame
 Row │ loss          called_back
     │ Float64       Bool
─────┼───────────────────────────
   1 │ 16.0                false
   2 │  2.56               false
   3 │  0.14173            false
   4 │  0.000549333         true

Requirements

• Merge of Add user interface point for accessing the internal state of a stopping criterion EarlyStopping.jl#37 and new EarlyStopping.jl 1.0 release

Draft release notes

• (enhancement - breaking) The form of the return value of
  IterationControl.train!(model, ...) has changed. Now a vector of ControlTrace
  objects is returned. By default, each trace records a complete history of (an public
  form) of the control's internal state. For example, the trace of any callback control
  records a history of the values returned by the calling function.

• (new default - breaking) Step() is now the same as Step(1)

• (new default - breaking) Callback() is now the same as Callback(identity)

• (new default - breaking) WithNumberDo() is now the same as WithNumberDo(identity)

• (new default - breaking) WithLossDo() is now the same as WithLossDo(identity)

• (new default - breaking) WithTraininLossesDo() is now the same as WithTrainingLossesDo(identity)

• (API change - breaking for custom controls) (for new implementations)
  takedown(control, verbosity, state) should now only return a short (possible empty)
  message string, which it should log to :info if it's non-empty and verbosity > 0. Ideally the message should fit on one line. (Detailed information can be accessed
  with EarlyStopping.public_state(control, state).) A new control trait
  needs_ingest(model) should be implemented for custom controls that require
  IterationControl.ingest!(model, datum) to be implemented to support model.

• (logging changes) What train! logs to Info (apart from control-specific logging)
  has changed; logging is less verbose by default.

• (mildly breaking) Controls that previously had a callback function as a field no longer have the type of the callback as a type parameter (Remove unnecessary restriction that a callback function <:Function. That is, allow any callable.  #59)

• Support for julia < 1.10 is dropped.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[codecov]

Codecov Report

❌ Patch coverage is 81.77340% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.94%. Comparing base (2aa4dd9) to head (098069e).
⚠️ Report is 3 commits behind head on dev.

Files with missing lines	Patch %	Lines
src/traces.jl	56.75%	32 Missing ⚠️
src/controls.jl	96.92%	2 Missing ⚠️
src/api.jl	75.00%	1 Missing ⚠️
src/train.jl	94.73%	1 Missing ⚠️
src/wrapped_controls.jl	92.85%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (2aa4dd9) and HEAD (098069e). Click for more details.

HEAD has 2 uploads less than BASE

Flag	BASE (2aa4dd9)	HEAD (098069e)
	4	2

Additional details and impacted files

@@            Coverage Diff             @@
##              dev      #72      +/-   ##
==========================================
- Coverage   96.19%   88.94%   -7.25%     
==========================================
  Files           9       12       +3     
  Lines         315      425     +110     
==========================================
+ Hits          303      378      +75     
- Misses         12       47      +35     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ablaom]

The codecov report seems out of date. The tests for display of ControlTrace are definitely there, but not reported as covered.