Check ASN1_INTEGER_get for errors.

Check return value when calling ASN1_INTEGER_get to retrieve a certificate
serial number. If an error occurs (which will be caused by the value being
out of range) revert to hex dump of serial number.

Reviewed-by: Rich Salz <rsalz@openssl.org>