Button set defective or bug?

I think the problem is that you're using Groups as interactive buttons. While it's possible to select a Group and put it into a button set, the built-in functionality often doesn't work for every item when the Group is clicked. 

Unfortunately, many of the available icons in the Content Library are actually groups of freeform objects, rather than single objects. (That's the case for the hands and the keyboard in your file.)

Here's one way to make a multi-object icon into one item on the timeline: insert a rectangle that's the size needed to hold the icon. Then, go into edit-states mode, and insert the icon directly into Normal state (like you inserted the checkmark into the Visited states). Format the objects in the icon as needed. Duplicate that state to create the other states you need, and reformat the icon in each as needed. 

Do that so each button is just 1 item on the Timeline (and not a Group). Also re-do the button set. Then the built-in functionality should work automatically so on only 1 button is in the Selected state at a time, and any button that was previously selected will become Visited. 

By the way, I suggest you remove the Visited state from the background image. Instead, make that a custom state (e.g., call it Faded). Why do that? An object with a Visited state will automatically change to Visited when clicked. That means the user will see the cursor change to indicate the background is clickable, and the background will change if the user clicks anywhere on the slide. But if you use a custom state, the background won't be a clickable object. Yet you can still change it to the custom state with the triggers attached to the buttons.