Don't use an enum in the return type for a public API function

We use an int instead. That means SSL_key_update() also should use an int.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)