Skip to content

GitLab

  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • G goffice
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 39
    • Issues 39
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GNOME
  • goffice
  • Issues
  • #57

Closed
Open
Created Mar 29, 2022 by b. s.@newbie-02

go_dtoa imprecise for last digit of 'long' values?

( disclaimer as a 'newbie': it's at least as likely that I have built in a bug somewhere, as that gnumeric or go_dtoa have one, but I have to ask to move things forward. )

in a bigger project I - mean to - have spotted a little inaccuracy of go_dtoa for long doubles when using 'shortest'.
( "!" GNM_FORMAT_g, IMHO that's what is used for cell edit display and write to files with 'conventions '-1''? )

can boil it down to a simple reproducer:

16470255126.0330839185  and 
16470255126.0330839194  are distinct FP-values with 80-bit IEEE 754 'long doubles', they have a difference of 
                    ~9.313225746154785156E-10 gnumeric can calculate that but assigns the string 
16470255126.033083919   to both for 'cell edit', file save with option 'shortest', or when I implement a 
function using go_dtoa with option 'shortest'. 

Ulf Adam's 'ryu' stays distinctive and produces

1.64702551260330839185E10  and 
1.6470255126033083919E10   I think MSVC and gcc have adopted ryu for 'to_chars' ( or similar ) shortly, 
but evtl. only for C++ usage?. 

( it is not! a matter if I really need that accuracy, it is! a question of whether or not the building blocks of a system meet the specifications they state. the more we let go with errors the better we fulfill the IT mantra 'floating point math is inaccurate', but that's not what gnumeric set out for? For the UI it's a question to 'get what you see', editing two cells to the same string and then having a difference between them is 'irritating'. )

Edited May 15, 2022 by b. s.
Assignee
Assign to
Time tracking