Fix some Coverity issues in sm2_encrypt()

Check for a negative EVP_MD_size().
Don't dereference group until we've checked if it is NULL.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6592)