MATH: rounding functions: cover full range of operands.
I see some weakness in the rounding functions not covering the full range of
'meaningless but possible' arguments, and tried to compose
'FRR' - Full Range Rounding alternatives. They include to use the - as far as I
tested - scaling re-scaling scheme with on average best results.
See attached sheet t0643_trunc_pub_04.gnumeric
listing some fails, and frr_trunc.patch
as a possible solution.
Testing level: no problems with the gnumeric test suite, IMHO well elaborated,
but would like third party retest. If liked by the dev committee I can
provide similar for roundup and round.
TL;DR:
accounting that users will not always define meaningful calculations
we do have some weak points in rounding besides bin-FP imprecision and
intended ( but questionable ) 'faking', resulting from:
- not accounting too small values for 'digits', e.g. 'trunc( 1E+100, -324'
can't be calculated in binary64, but decided, a clear '0', - not rounding denormals,
- some calculations are questionable, 'trunc( 1.7976931348623157E+308, -295 )'
-> 1.7976931348623E+308 is meaningful, 'trunc( 1.7976931348623157E+308, -293 )'
-> #ERROR! is not, - not accounting too big values for digits, e.g.
'trunc( -1.7976931348623157E308, 308 ) is difficult to calculate, but clear that
return unchanged is mathematically better than #ERROR!, ( results from faking? ), - 'trunc( 0, 309 )' would be better as '0' than #ERROR!
- similar 'trunc( 0.109374999992, 1024 )' would be correct as 0.109374999992,
- 'trunc( 1E-279, 279 )' -> 9.999999999999999E-280 is incorrect, as well
- 'trunc( 1E-278, 278 )' -> 1.0000000000000001E-278
and so on, see attached sheet which here produces ~315 mathematical fails with
gnumeric standard bin64.
I'd play around and think the following changes would improve:
- make MIN_ and MAX_10_EXP available,
- invent and implement an MIN_DEN_10_EXP, the minimum number X for which a
datatype can represent 1EX, - implement rounding functions accounting these borders in triaging,
- use the scaling - re-scaling scheme with fewest deviations,
I'd try to put it in numbers.h and plugins/fn-math/functions.c, see attached
patch, and compare results with above example, fails reduced to a third.