Repository navigation
Conversation
|
|
||
| class ThresholdFilterCodec(Codec): | ||
| """ | ||
| Filter codec that replaces all values whose magnitude is below a |
There was a problem hiding this comment.
Could we support replacing values below and values above the threshold? Some encodings of missing data use huge values, and so replacing only large magnitudes would also be very useful
| _threshold: int | float | ||
| _replacement: int | float | Replacement | ||
|
|
||
| codec_id: str = "replace.threshold" # type: ignore |
There was a problem hiding this comment.
I'm not a huge fan but don't have a better idea :)
| "nan_min", | ||
| "nan_mean", | ||
| "nan_max", | ||
| ] = 0, |
There was a problem hiding this comment.
For a replace |x| < t, a default of zero is ok, but if we generalise I'd prefer to not offer a default at all and just always be explicit
| "nan_max", | ||
| ] = 0, | ||
| ) -> None: | ||
| if not (threshold >= 0): |
There was a problem hiding this comment.
We should also check that the threshold is finite (I guess an infinite threshold would also be fun, but it should definitely not be NaN)
| "nan_max", | ||
| ] = 0, | ||
| ) -> None: | ||
| if not (threshold >= 0): |
There was a problem hiding this comment.
If we use |x| < t, then a threshold of zero is useless since it doesn't apply. I think it would be better to add an option for either being inclusive (<=) or exclusive (<)
| else self._replacement | ||
| ) | ||
|
|
||
| with np.errstate(invalid="ignore"): |
There was a problem hiding this comment.
We don't ignore this in the other codec, why here?
| assert codec.get_config() == dict( | ||
| id="replace.threshold", threshold=1e-3, replacement=0.0 | ||
| ) | ||
| with pytest.raises(ValueError): |
There was a problem hiding this comment.
This could also check the message against a regex
| # numcodecs-replace | ||
|
|
||
| Value replacement codecs for the [`numcodecs`] buffer compression API. | ||
| Value replacement codecs for the [`numcodecs`] buffer compression API: `ReplaceFilterCodec` replaces configured values, `ThresholdFilterCodec` replaces all values whose magnitude is below a threshold (e.g. to drop negligible values before lossy compression). |
There was a problem hiding this comment.
The docs/index.md should also be changed, and there you can doc-reference the actual classes so that they become clickable links, e.g.
[`ReplaceFilterCodec`][numcodecs_replace.ReplaceFilterCodec]
Summary
Adds
ThresholdFilterCodec(idreplace.threshold), a filter codec in the style ofReplaceFilterCodec: during encoding all values with|x| < thresholdare replaced byreplacement(zero by default; the specialReplacementvalues such asfinite_maxare supported and computed before the replacement), NaN values are never replaced; decoding passes the data through.ReplaceFilterCodecis unchangedreplace.threshold, tests (configuration/repr, float/int/inf/NaN cases,finite_maxreplacement, integer data), README lineMotivation
Under a mean error bound, spending a part of the budget on setting the smallest values of a heavy-tailed field to zero (which can then be masked by
MaskMetaCodecand skipped by the subsequent codecs) improves the compression ratio considerably, e.g. surface runoff in the compression lab challenges 209 -> 232. This codec expresses that step as a reusable, stackable filter.