How to roll back a transaction on error in PostgreSQL?

I'm writing a script for PostgreSQL and since I want it to be executed atomically, I'm wrapping it inside a transaction.
I expected the script to look something like this:

BEGIN
-- 1) Execute some valid actions;
-- 2) Execute some action that causes an error.
EXCEPTION
    WHEN OTHERS THEN
        ROLLBACK;
END; -- A.k.a. COMMIT;

However, in this case pgAdmin warns me about a syntax error right after the initial BEGIN. If I terminate the command there by appending a semicolon like so: BEGIN; it instead informs me about error near EXCEPTION.
I realize that perhaps I'm mixing up syntax for control structures and transactions, however I couldn't find any mention of how to roll back a failed transaction in the docs (nor in SO for that matter).

I also considered that perhaps the transaction is rolled back automatically on error, but it doesn't seem to be the case since the following script:

BEGIN;
-- 1) Execute some valid actions;
-- 2) Execute some action that causes an error.
COMMIT;

warns me that: ERROR: current transaction is aborted, commands ignored until end of transaction block and I have to then manually ROLLBACK; the transaction.

It seems I'm missing something fundamental here, but what?

EDIT:
I tried using DO as well like so:

DO $$
BEGIN
-- 1) Execute some valid actions;
-- 2) Execute some action that causes an error.
EXCEPTION
    WHEN OTHERS THEN
        ROLLBACK;
END; $$

pgAdmin hits me back with a: ERROR: cannot begin/end transactions in PL/pgSQL. HINT: Use a BEGIN block with an EXCEPTION clause instead. which confuses me to no end, because that is exactly what I am (I think) doing.

POST-ACCEPT EDIT: Regarding Laurenz's comment: "Your SQL script would contain a COMMIT. That ends the transaction and rolls it back." - this is not the behavior that I observe. Please consider the following example (which is just a concrete version of an example I already provided in my original question):

BEGIN;

-- Just a simple, self-referencing table.
CREATE TABLE "Dummy" (
    "Id" INT GENERATED ALWAYS AS IDENTITY,
    "ParentId" INT NULL,
    CONSTRAINT "PK_Dummy" PRIMARY KEY ("Id"),
    CONSTRAINT "FK_Dummy_Dummy" FOREIGN KEY ("ParentId") REFERENCES "Dummy" ("Id")
);

-- Foreign key violation terminates the transaction.
INSERT INTO "Dummy" ("ParentId")
VALUES (99);

COMMIT;

When I execute the script above, I'm greeted with: ERROR: insert or update on table "Dummy" violates foreign key constraint "FK_Dummy_Dummy". DETAIL: Key (ParentId)=(99) is not present in table "Dummy". which is as expected. However, if I then try to check whether my Dummy table was created or rolled back like so:

SELECT EXISTS (
    SELECT FROM information_schema."tables"
    WHERE "table_name" = 'Dummy');

instead of a simple false, I get the same error that I already mentioned twice: ERROR: current transaction is aborted, commands ignored until end of transaction block. Then I have to manually terminate the transaction via issuing ROLLBACK;.

So to me it seems that either the comment mentioned above is false or at least I'm heavily misinterpreting something here.