It’s an issue. I’ve groussed about it to Affinity, no traction although one “moderator” said it had been noted. I’ve also posted to Carbide, no response for those who code although someone “in the know” suggested this was a Carbide issue versus Affinity. Why don’t you also post this as an issue directly to the makers.
Here’s what I do; although I’m on a Mac, this may not work for you. I do all the design in Affinity Designer. I then cut-and-past into Graphic. This is an inexpensive Mac app that I also rely on for drawings with dimensioning, not yet a feature of Designer. Then I create the SVG’s with Graphic, the scale is as it should be.
As others mentioned, I think you can do this with Inkscape too. However, and I forget the specific reasons, I found this, after testing, inferior to Graphic for this purpose.