There is an old saying in the theater: "Anyone who puts kids or animals on the stage deserves what happens to them." That means that no matter how well-behaved, or well-trained you think they are, at some time they are going to revert to their true nature, and you can only hope it doesn't happen during a performance. The SL correlation is: "Anyone who uses groups deserves what happens to them." That means that no matter how well-behaved or how well-trained you hope they are, groups don't play nicely with anything, and especially not states, clicking on, and triggers. Sooner or later, you are likely to have problems with them.
As you discovered, you can't make them work.
The solution is to delete the check mark. Edit the rectangle it sits on and paste it onto the normal state. Since the Selected state is already created, you may also have to paste it on the selected state. Now you have one object, and can click it and change its states without fear of problems.
Another problem is this trigger:
No doubt you created this to try to get the state change to work with the group. But this function is built into any object that has a selected state. If you create a trigger that duplicates that built-in function, you will likely have problems with it. Delete all of those.
Once you have the states and triggers cleaned up, Select the upper two rectangles, right click, and choose Button set. Making those two objects a button set will allow the user to choose only one of them at a time, and to change their mind. When you make the other two a button set, be sure to choose a New Set.
You will need the Go button. The If...Then function just isn't sophisticated enough to handle showing one layer if both sets have been selected, and both are yes, or show another one if one of them is not selected without showing one after only the first has been selected and the second not yet. (Sound confusing? I intended it to be, because that's what happens to the system. It's a learning development environment, not a programming language.) <Rant over>
You need the Go button, but see the second option for another way to handle it.
For what it's worth, I find your color scheme very counter-intuitive. I am accustomed to having selected objects more visible and obvious, not less. If it were mine, I would reverse the Normal and selected colors.