The aspect ratio is irrelevant. You need to consider your story size in relationship to your recording window.
For example: if my story size is 960x540 (16:9) and my recording window is 1920x1080 (16:9). That's 518,000 pixels in the story but over 2 million pixels in the recording. That's a lot of missing pixels and you'll get degraded quality.
Ideally, you either make your story size the same as your recording size or you record a smaller window size that matches your story size.
Here is an example I did quickly where I recorded a video at 1920x1080
As you can see, the recording that's closest to the story size looks better. Also, as you scale the browser window you'll see that the quality changes a bit.
You can insert a Peek recording, but you'll run into the same issue if the recording window is larger than the course size.
Hope that helps.