I created this what seems like years ago. :) Here is what I recall doing:
I have a system with two monitors.
Monitor A: Opened Storyline and set the Story Size to 1024x640 for this demo. (Storyline > Design tab > Story Size)
Monitor B: Set display resolution to 1280x800 (via the operating system's Display settings) and opened a second instance of SL on that monitor.
I started the recording from within Storyline on Monitor A (Slides tab > Record screen). Moved the area to be recorded onto Monitor B desktop and expanded it to full screen.
The recording was then placed into Storyline running on Monitor A.
A little crazy but I was just playing around on this one.
Both 1024x640 and 1280x800 are the same aspect ratio (16:10). Videos can become blurry when you zoom in or scale them from what they were originally recorded at. In this situation, I recorded the video at a display resolution that was slightly larger than the final Story size. End result? Looks sharper.
How this helps.
Richard