Hold down the shift key when clicking on objects to add or remove them from the current selection.
See: