What is difference between below two commands?
1. openssl genpkey -algorithm RSA
2. openssl genrsa
In document difference is "Private Key" and "RSA Private Key".
Then..
What is diference between "Private Key with algorithm RSA" and "RSA Private Key"?
The genpkey command can create other types of private keys - DSA, DH, EC and maybe GOST - whereas the genrsa, as it's name implies, only generates RSA keys. There are equivalent gendh and gendsa commands.
However, the OpenSSL documentation states that these gen* commands have been superseded by the generic genpkey command.
In the case of your examples, both generate RSA private keys.
openssl genrsa -out genrsa.key 2048
and
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out genpkey.key
will generate a 2048 bit RSA key with the exponent set to 65537.
Simply cat the resulting files to see that they are both PEM format private keys; although openssl rsa encloses them in BEGIN RSA PRIVATE KEY and END RSA PRIVATE KEY while openssl genpkey omits the RSA. The former is PKCS#1 format, while the latter is PKCS#8.
Running openssl rsa text -in <filename> against both shows that they are RSA private keys with the same publicExponent. The newer genpkey command has the option to change this using -pkeyopt rsa_keygen_pubexp:value while the genrsa command doesn't have this option.
genrsa (and also rsa) and 'new' (since about 2000) PKCS8 format written by genpkey (and also pkey and pkcs8 -topk8); although they contain semantically the same information they are not the same and the PEM label RSA PRIVATE KEY vs PRIVATE KEY is very important. Also if they are password-encrypted (your example is not) the PBE used is very different (PKCS8 is better). There is no commandline gendh -- and EC-specific gen is inconsistently ecparam -genkey !
gendh is listed as an option to openssl help. That's as far as I've tried it though!
gendh and gendsa generate parameters (aka groups) not keys, equivalent to new-style genpkey -genparam ; dsaparam -genkey and ecparam -genkey generate keys for given parameters but there is no old-style way to generate a DH key (for any parameters), only new-style genpkey (without -genparam) can do a DH key.
Be aware that Exim with DKIM support does not directly accept RSA private keys generated by the openssl genpkey -algorithm rsa ... command. Exim expects the private key to use the BEGIN RSA PRIVATE KEY and END RSA PRIVATE KEY delimiter lines, as generated by openssl genrsa ..., and not BEGIN PRIVATE KEY/END PRIVATE KEY as generated with openssl genpkey ....
Exim will fail with the message DKIM: signing failed (RC -101) in the panic log when sending mail if the delimiter lines are wrong.
genpkey DKIM key with BEGIN PRIVATE KEY / END PRIVATE KEY delimiters. Exim is using this key to sign outgoing SMTP without any problems.