Check for overflows in i2d_ASN1_SET()

Thanks to Shi Lei for reporting this issue.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit af601b83)