Check md_size isn't negative before we use it

Issue found by Coverity

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6592)