boxes: Fix calculation of rounded rectangles

Since 68fba458 the function is used for more calculations, exposing
a bug when used with fractional scaling.

GNOME/mutter#609
4 jobs for !599 with rounded_rectangles in 8 minutes and 33 seconds (queued for 3 seconds)
latest detached
Status Job ID Name Coverage
  Review
passed #311274
check-commit-log

00:00:33

 
  Build
passed #311275
build-mutter

00:05:15

 
  Test
passed #311277
can-build-gnome-shell

00:01:18

passed #311276
test-mutter

00:02:45