Implement length checks as a macro

Replace the various length checks in the extension code with a macro to
simplify the logic.

Reviewed-by: Rich Salz <rsalz@openssl.org>