As long as the file is correctly named and in the right place, e.g. /pages/404.tsx, that should be enough. Could you provide a minimal reproducible example?
@juliomalves I just created a new project with create-next-app, using TS, but the issue is not there, the custom 404 works just fine, while in the existing project it doesn't. Sadly the existing project is huge and way beyond my understanding, so I wouldn't know which parts would be worth putting in an example, much less which parts I can actually share here, since it's a company project.
So as it turns out the next.config.js configured to project to only accept the file extensions page.tsx, page.ts, page.jsx and page.js. I changed my filename to 404.page.tsx and it's working perfectly now.
/pages/404.tsx, that should be enough. Could you provide a minimal reproducible example?