Segmentation Fault Generating RSA Key Using OpenSSL

Question

I am trying to generate RSA keys using OpenSSL with the following function:

RSA *genRSA() {
  clear();
  mvprintw(0, 0, "Generating RSA key...\n");
  RAND_load_file("/dev/random", 4096);
  BIGNUM *e = BN_new();
  BN_set_word(e, RSA_F4);
  RSA *rsa;
  while (getch() != '\n'); // the program does reach this point
  if (!RSA_generate_key_ex(rsa, 4096, e, 0)) { // seg fault must occur on this line
    while (getch() != '\n'); // never gets here
    printw("ERROR: Failed to create RSA key\n");
    return NULL;
  }
  while (getch() != '\n'); // or here
  BN_free(e);
  if (!RSA_check_key(rsa)) {
    printw("ERROR: Key failed validation\n");
    return NULL;
  }
  printw("Key generation completed successfully\n");
  return rsa;
}

I'm not receiving any compiler warnings other than some deprecated on OS X ones (could that be causing an issue?). Why am I getting a seg fault?

If you're certain it's faulting in the RSA_generate_key_ex() function, then its likely your input parameters. Verify that each one is as expected for the function. — Ender
– Ender, Commented Aug 29, 2014 at 5:32
This could be a problem, too: RAND_load_file("/dev/random", 4096);. You are asking for bytes, not bits. And a lot of them. It could deplete the device and you could block a long time. To achieve the equivalent of 4096-bit key, you need about 140-bit security level. 140/8 = 17.5 bytes: RAND_load_file("/dev/random", 18);. — jww
– jww, Commented Aug 29, 2014 at 6:48
@jww Fortunatly for me it didn't take too long, but I will take your suggestion into thought. — carloabelli
– carloabelli, Commented Aug 29, 2014 at 6:49
@cabellicar123 - in that case, RAND_load_file("/dev/random", 4096); probably failed. You should check the return value and fall back to "/dev/urandom". — jww
– jww, Commented Aug 29, 2014 at 6:52

PaulMcKenzie · Accepted Answer · 2014-08-29 05:25:35Z

2

Without knowing anything about the library you're using, this is incorrect:

RSA *rsa;
while (getch() != '\n'); // the program does reach this point
  if (!RSA_generate_key_ex(rsa, 4096, e, 0))

You are calling RSA_generate_key_ex with an uninitialized pointer rsa. There is no way that RSA_generate_key_ex function is going to be able to do anything with it except attempt to use it and as you see, crash.

So read the docs on that function as to what the first parameter is supposed to be. Maybe it should be this:

RSA rsa;
while (getch() != '\n'); // the program does reach this point
  if (!RSA_generate_key_ex(&rsa, 4096, e, 0))

If this is the case, then you need to change your return type to RSA and not RSA* (I'm assuming that RSA is a struct or typedef of a type that can be returned safely by value).

answered Aug 29, 2014 at 5:25

PaulMcKenzie

35.6k4 gold badges26 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

carloabelli Over a year ago

I figured out I needed to first initialize RSA by changing the line to RSA *rsa = RSA_new();. Thanks for the help!

Collectives™ on Stack Overflow

Segmentation Fault Generating RSA Key Using OpenSSL

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related