How do button states stack?

Storyline 360: Definition of Built-In States

• Normal: This is the neutral state for an object. By default, it's also the initial state for an object (i.e., how it looks when it first appears in the published output), but you can change the initial state if needed.
• Hover: This is how an object appears when learners move their mouse over it. If this state exists for an object, it'll automatically display when learners hover over it. You don't need to create a trigger to invoke the hover state, but if you want to use this state on a trigger, note that trigger conditions won’t restrict its built-in function.
• Down: This is how an object appears while learners are clicking it. If this state exists for an object, it'll automatically display when learners click it.
• Selected: This is how an object appears when it's been selected. It's generally used to indicate that a learner has chosen the object. For example, a check box uses the selected state to provide a visual cue (check mark), indicating it has been clicked.

Selected states are often used in conjunction with button sets, so only one object gets selected at a time. If the selected state exists for an object, it'll automatically display when learners click it. You don't need to create a trigger to invoke it, but if you want to use this state in a trigger, note that trigger conditions won’t restrict its built-in function.

• Visited: This is how an object appears after learners click it. It's useful when you want to provide learners with a visual indicator of the objects they've already clicked (for example, a series of buttons).

Storyline 360 remembers this state when learners revisit the same slide later unless you've configured the slide properties or layer properties to "reset to initial state."

If the visited state exists for an object, it'll automatically display after learners click it. You don't need to create a trigger to invoke it, but if you want to use this state in a trigger, note that trigger conditions won’t restrict its built-in function.

• Disabled: Use this state when you want to disable an object. A disabled object is visible to learners, but it won't respond when hovered over, clicked, or dragged. Unless it's the initial state of an object, you'll need to use a trigger to invoke it.
• Hidden: This state makes an object invisible. Unless it's the initial state of an object, you'll need to use a trigger to hide an object.

This is a long overdue issue with states in buttons in Storyline.  A lot discussions about this to be found in the community. One of the weirdest things is, that when you make custom buttons yourself. So an image with states...they behave differently then out-of-the box buttons ( with states ) you edit.

Some of the results you describe are mutually exclusive. For example, you describe wanting the Selected state’s outline to override the Visited state’s, but the Visited state’s text to overwrite the Selected’s text. 

Still, Math is right. Some consistency, leading to predictability in button states would be helpful.

I've done a LOT of work with button states and I've found that when something doesn't display correctly, there's usually a good reason. If I ran into a button display issue, and the resolution wasn't obvious, what I would end up doing is deleting the button altogether, and then recreating it from scratch. Simple enough. I'd be happy to look at this scene, and hunt down the issue alternatively. Could be there is left-over content that's not visible for some reason until the button is clicked. I've never head of states "overlapping," but there's likely unresolved formatting in the button state.