java.lang.IllegalArgumentException: Invalid point encoding 0x30

Verification with the Java code fails for the following reasons:

• The public key is imported incorrectly. ECCurve#decodePoint() returns the public key as org.bouncycastle.math.ec.ECPoint.ECPoint and requires the uncompressed/compressed key. A possible fix is to make the following changes to import the DER encoded public SPKI key and get the required ECPoint:

  import java.security.KeyFactory;
  import java.security.spec.X509EncodedKeySpec;
  import org.bouncycastle.jce.interfaces.ECPublicKey;
  import org.bouncycastle.crypto.signers.ECDSASigner;
  import org.bouncycastle.jce.provider.BouncyCastleProvider;
  ...
  Security.addProvider(new BouncyCastleProvider());
  ...
  byte[] publicKey = Base64.getDecoder().decode(pkey);      
  KeyFactory keyFactory = KeyFactory.getInstance("EC", "BC");
  X509EncodedKeySpec x509EncodedKeySpec = new X509EncodedKeySpec(publicKey);
  ECPublicKey ecPublicKey = (ECPublicKey)keyFactory.generatePublic(x509EncodedKeySpec); // note: org.bouncycastle.jce.interfaces.ECPublicKey
  ...
  signer.init (false, new ECPublicKeyParameters(ecPublicKey.getQ(), domain)); // note: public key is imported as ECPoint
  ...
  

• When verifying, the hashing is missing. This must be done explicitly, as ECDSASigner#verifySignature() does not hash automatically:

  import java.security.MessageDigest;
  ...
  MessageDigest messageDigest = MessageDigest.getInstance("SHA-256");
  messageDigest.update(message.getBytes(StandardCharsets.UTF_8));
  byte[] hash = messageDigest.digest();
  ...
  String R = "AMt4LZPxkiWu47DcqbRMcrzLFww9Lm584eq6zdj+/Jgy";
  String S = "AKgoAIV50yXYGIARdNE4bEt+eaujKFwX2cX6pf/y681p";
  boolean verified = signer.verifySignature(hash, new BigInteger(Base64.getDecoder().decode(R)), new BigInteger(Base64.getDecoder().decode(S)));
  

With these changes, the verification of the test data (posted at the end of the question) is successful.

As already noted in the comment, ECDSA is supported in Java 11 in both formats (ASN.1/DER and IEEE P1363), so that no BouncyCastle is required. Since sign.sign() in the NodeJS code generates the signature in ASN.1/DER format by default (IEEE P1363 is also supported as of v13.2.0, see here), the signature can be verified directly in this format on the Java side:

import java.security.KeyFactory;
import java.security.spec.X509EncodedKeySpec;
import java.security.interfaces.ECPublicKey;
import java.security.Signature;
...
KeyFactory keyFactory = KeyFactory.getInstance("EC");
X509EncodedKeySpec x509EncodedKeySpec = new X509EncodedKeySpec(Base64.getDecoder().decode(pkey));
ECPublicKey ecPublicKey = (ECPublicKey) keyFactory.generatePublic(x509EncodedKeySpec); // note: java.security.interfaces.ECPublicKey

Signature verifier = Signature.getInstance("SHA256withEcdsa");
verifier.initVerify(ecPublicKey);
verifier.update(message.getBytes(StandardCharsets.UTF_8));
String signature = "MEYCIQDLeC2T8ZIlruOw3Km0THK8yxcMPS5ufOHqus3Y/vyYMgIhAKgoAIV50yXYGIARdNE4bEt+eaujKFwX2cX6pf/y681p";
boolean verified = verifier.verify(Base64.getDecoder().decode(signature));
...

which also successfully verifies the signature.

For the sake of completeness: The specifier for IEEE P1363 is SHA256withECDSAinP1363Format, s. The SunEC Provider. In this case, the signature is to be passed as a concatenation of the byte arrays r and s, where r and s are fixed sized, unsigned big endian. r and s each have the length of the order of the generator point (if necessary with leading 0x00 values), e.g. for your example (hex encoded):

cb782d93f19225aee3b0dca9b44c72bccb170c3d2e6e7ce1eabacdd8fefc9832 a828008579d325d818801174d1386c4b7e79aba3285c17d9c5faa5fff2ebcd69

Spaces only for display purposes! Note the difference to the ASN.1/DER format: There r and s are minimally sized, signed big endian (which is why in the case of a leading byte > 0x7f a 0x00 must be prefixed so that the values are positive):

30460221 00cb782d93f19225aee3b0dca9b44c72bccb170c3d2e6e7ce1eabacdd8fefc9832 0221 00a828008579d325d818801174d1386c4b7e79aba3285c17d9c5faa5fff2ebcd69

Based on these definitions, IEEE P1363 provides a fixed size signature (64 bytes for P-256), while ASN.1/DER has no fixed size, but only a maximum size (72 bytes for P-256). See also this post for more details on both formats.

When creating BigInteger values from unsigned r and s, use the constructor BigInteger(int, byte[]) instead of BigInteger(byte[]). In your test data, r and s are signed, so the BigInteger conversion is correct.

Just as an addition to Topaco's explanation.... I was still getting issues aroud the point, either because it wasn't found on the line or with the Invalid point encoding 0x30.

For me the solution was not to load the curve via NISTNamedCurves.getByName("P-256"), but instead, get it from the public key itself.

public boolean signatureMatchesMessages(String message, BigInteger r, BigInteger s) {
        try {

            Security.addProvider(new BouncyCastleProvider());
            String pkey = "MFkwEw...Kug=="; // the entire base64 string above

            byte[] publicKeyDecoded = Base64.decode(pkey);
            KeyFactory keyFactory = KeyFactory.getInstance("EC", "BC");
            X509EncodedKeySpec x509EncodedKeySpec = new X509EncodedKeySpec(publicKeyDecoded);

            ECPublicKey publicKey = (ECPublicKey)keyFactory.generatePublic(x509EncodedKeySpec);
            ECDSASigner verifier = new ECDSASigner();
            ECCurve curve = publicKey.getQ().getCurve();
            ECDomainParameters domain = new ECDomainParameters(curve, publicKey.getParameters().getG(), publicKey.getParameters().getN(), publicKey.getParameters().getH());

            MessageDigest messageDigest = MessageDigest.getInstance("SHA-256");
            messageDigest.update(message.getBytes(StandardCharsets.UTF_8));
            byte[] hash = messageDigest.digest();

            verifier.init (
                    false,
                    new ECPublicKeyParameters(
                            publicKey.getQ(),
                            domain
                    )
            );

            return verifier.verifySignature(hash, r, s);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }