RT2679: Fix error if keysize too short

In keygen, return KEY_SIZE_TOO_SMALL not INVALID_KEYBITS.

** I also increased the minimum from 256 to 512, which is now
documented in CHANGES file. **

Reviewed-by: Matt Caswell <matt@openssl.org>