Fix hostname validation in the command-line tool to honour negative return values.

Specifically, an ASN.1 NumericString in the certificate CN will fail UTF-8 conversion
and result in a negative return value, which the "x509 -checkhost" command-line option
incorrectly interpreted as success.

Also update X509_check_host docs to reflect reality.

Thanks to Sean Burford (Google) for reporting this issue.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 0923e7df)