Return error when trying to print invalid ASN1 integer

GH: #1322
(cherry picked from commit 5e3553c2

)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Stephen Henson <steve@openssl.org>