Convert modular exponentiation tests to new framework

Updated due to test framework changes
Updates after code review
Missed some checks

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3269)