add 'unsupported cipher mode' diagnostics to evp_lib.c and genpkey.c

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6217)