How can I prevent the learner from advancing after failing the course level assessment?

Unfortunately in Storyline it's an all or nothing proposition (unless something has changed in v2 that I'm unaware of). That's why I use the little popups that let the user know what's going on. The only other way around it, and one I've used extensively, is to create a custom navigation. Then you can do all kinds of cartwheels with it. But, it expands your development timeline and project scope.