Ensure that rc5 doesn't try to use a key longer than 2040 bits

The maximum key length for rc5 is 2040 bits so we should not attempt to
use keys longer than this.

Issue found by OSS-Fuzz and Guido Vranken.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/8834)

(cherry picked from commit 792cb4ee8d82e4b063f707fc9f4992271ffd65ab)