Don't error with -1 for BIGNUM exp operations

The man pages say that BIGNUM arithmetic operations fail with a 0 return.
However some functions were returning -1 on error. In master and 1.1.0 they
already return 0, so this brings 1.0.2 in line.

Reviewed-by: Bernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/4682)