Fix errstr error code parsing

Error codes are printed in hex, and previous OpenSSL versions expected
the error codes to be provided to errstr in hex. In 1.1.0, for some
reason, it was expecting them to be decimal.

Reviewed-by: Richard Levitte <levitte@openssl.org>