The close dialog is added as a child to MetaWindowActor, and since commit [1] MetaWindowActor applies a transformation matrix which scales all it's children using the geometry scale factor. Now because the dialog actor is not a window (i.e. a MetaSurfaceActor), but a subclass of StWidget, the scale factor is also applied to the properties of the dialog by StThemeNode, so we end up applying the geometry scale twice to the close dialog.
Fix this by applying the inverted scale to the dialog, which leaves the
scaling only to MetaWindowActor. This means we also can't apply a pivot
point other than 0 to the dialog actor, so apply the 0.5-pivot point to
the _dialog child of the Dialog class (the actual visible dialog box)
and also perform scaling animations on this child.
[1] mutter@fb9e8768