EVP_MD_size() can return an error

Fix some instances where we weren't checking the error return.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6373)