Coverity fix in some crypto/asn1 code

Call to i2d method returns an int value.

Fix:
CID 1338183 (#1 of 1): Improper use of negative value (NEGATIVE_RETURNS)
CID 1371691 (#1 of 1): Improper use of negative value (NEGATIVE_RETURNS)
CID 1371692 (#1 of 1): Improper use of negative value (NEGATIVE_RETURNS)

[extended tests]

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7359)

(cherry picked from commit da84249be6492ccfc5ecad32ac367fd06e9bdbef)