Return error when trying to print invalid ASN1 integer

GH: #1322

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Stephen Henson <steve@openssl.org>
(cherry picked from commit 32baafb2)