Commit 8a85c341 authored by Geoff Thorpe's avatar Geoff Thorpe
Browse files

The problem of rsa key-generation getting stuck in a loop for (pointlessly) 

small key sizes seems to result from the code continually regenerating the
same prime value once the range is small enough. From my tests, this change
fixes the problem by setting an escape velocity of 3 repeats for the second
of the two primes.

PR: 874
parent bcfea9fb

crypto/rsa/rsa_gen.c

+15 −2
Original line number Diff line number Diff line
@@ -128,12 +128,25 @@ static int rsa_builtin_keygen(RSA *rsa, int bits, BIGNUM *e_value, BN_GENCB *cb) 
	if(!BN_GENCB_call(cb, 3, 0))

		goto err;

	for (;;)

		{

		/* When generating ridiculously small keys, we can get stuck

		 * continually regenerating the same prime values. Check for

		 * this and bail if it happens 3 times. */

		unsigned int degenerate = 0;

		do

			{

			if(!BN_generate_prime_ex(rsa->q, bitsq, 0, NULL, NULL, cb))

				goto err;

			} while((BN_cmp(rsa->p, rsa->q) == 0) && (++degenerate < 3));

		if(degenerate == 3)

			{

			ok = 0; /* we set our own err */

			RSAerr(RSA_F_RSA_GENERATE_KEY,RSA_R_KEY_SIZE_TOO_SMALL);

			goto err;

			}

		if (!BN_sub(r2,rsa->q,BN_value_one())) goto err;

		if (!BN_gcd(r1,r2,rsa->e,ctx)) goto err;

		if (BN_is_one(r1) && (BN_cmp(rsa->p,rsa->q) != 0))

		if (BN_is_one(r1))

			break;

		if(!BN_GENCB_call(cb, 2, n++))

			goto err;