Fix an endless loop in rsa_builtin_keygen.

Cherry-picked by Matt Caswell from 69795831

.

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4670)