[test] ECC: check the bounds for auto computing cofactor

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/9821)

(cherry picked from commit 1d3cd983f56e0a580ee4216692ee3c9c7bf14de9)