The Nomad 3 has the same bed layout as the Nomad 883 Pro as far as I’m aware, so here’s the layout: https://community.carbide3d.com/uploads/default/original/1X/15c685238373a9dd7e54a4ebf1d937fe3399567d.PDF
I usually used the 6mm locating holes for alignment as that’s what Carbide 3D used for their vise and flip jig.
And regarding the use with a mod vise, you might want to consider using a threaded table and putting partially threaded pins like these in it instead. That’s what Saunders Machine Works does with the metric bed + mod vise (the imperial one has fancier pins). This gives you many more places on the table to mount the vise, so the vise doesn’t have to travel so far, so it can be more compact.
Nitpick: M5 is a designation for threads, there’s no such thing as an “M5” locating hole or an “M5” dowel pin.
As for the diameter, you need to be careful about the tolerance of the dowel pins you buy. There are dowel pins meant for slip fits and locating (usually tolerance h6) and there are pins meant for transition or interference fits (e.g. tolerance m6).
You probably bought the latter category of pin, which could have a tolerance like +3/+11um.
The main thing to look for is the letter. The “h” set of tolerances never exceeds the specified diameter. It’s always -something/+0.
Regarding all the fits and tolerances stuff, I can’t recommend enough buying a reference book about it. In the US the standard is Machinery’s Handbook and for the rest of the world there are other books. Personally, I use this one.
I’d also recommend a textbook. I went for this one.