Metals are generally safe (leaving out Plutonium, Beryllium and such). Watch out for Cu (copper) alloys though - stay away from anything with Beryllium in it.
Plastics are generally safe - provided they aren’t machined until they burn. Burning release toxics, carcinogens, and teratogens. Just don’t do this.
Traditional “saw dust” is actually dangerous but some orders of magnitude less than CNC particles. Hardwood “saw dust” can be particularly nasty as some hardwoods contain noxious stuff. It’s best to treat “saw dust” with as much caution as CNC particles and waste.
Friable (“easily crumbled”) materials (e.g. wood, FR4, Garolite, carbon composite, MDF) are extremely dangerous to one health and lungs when CNCed. Medium Density Fiberboard (MDF) is one of the worst as it combines wood and potentially toxic glue. MDF is a common spoiler board material so care is necessary even when machining metals or plastics if it is used.
In high precision machining, the spoiler board is machined flat by the machine - so it is square and flat to the machine - and this generates dust at a incredible rate. There are special tools to make this process go fast - spoiler board tools - and this increases the rate even more.
As I’ve commented previously, a very safe solution can be engineered on any reasonable budget. Combined with an enclosure one can be very safe from CNC particles (“dust”) and from mechanical and auditory issues.
Another reason to deal well with CNC particles is… preventing fires and explosions. Believe it or not particles suspended in air can be quite testy - a “dust fire” or “dust explosion” can occur under the right conditions. This can be totally avoided with a little safe air handling.
Emptying waste barrels and dust collector filters can be surprisingly easy for exposure. It’s a good idea to empty outside and upwind, the use liner bags so rapid closure is possible and to consider a mask. Vacuuming yourself before entering a residence after an empty is not a bad idea.
Here are previous postings that provide some insight into enclosures, air, and hearing safety: