When we clean up the uniform allocations after a frame,
it can happen that our space requirements actually increase,
due to padding that depends on the order of allocations.

Instead of asserting that it doesn't happen, just make
it work by growing our allocation.

Fixes: #3853