Avoid overflow issues in X509_cmp.

The length is a long, so returning the difference does not quite work.

Thanks to Torbjörn Granlund for noticing.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>