Is there any way to set the corner radius of individual corners in a rounded rectangle?

Storyline doesn't let you do that. However, here's an option to consider: 

• Create the shape you want as a PNG.
  • For example, you could create the shape in PowerPoint or in drawing software, and then save it as a PNG picture. 
  • Be sure the background of this picture is transparent. 
• In Storyline, insert a standard rectangle.
• Right-click the rectangle and open the Format Shape window. 
• Fill the rectangle with the picture of the shape. 

Note: When you fill a shape with a picture, Storyline expand the picture to the dimensions of the shape. So if you want your custom shape in various aspect ratios, you may need to create multiple versions of the image to ensure a proper fit.