Error if memory limit exceeded.

Set a specific error if the parameters are otherwise valid but exceed the
memory limit.

Reviewed-by: Rich Salz <rsalz@openssl.org>