make allyesconfig builds as many drivers into the main kernel as possible, which makes for a huge kernel file. The <kernel source>/Documentation/admin-guide/README.rst says:
Having unnecessary drivers will make the kernel bigger, and can under some circumstances lead to problems: probing for a nonexistent controller card may confuse your other controllers.
Also:
The "kernel hacking" configuration details usually result in a bigger or slower kernel (or both), and can even make the kernel less stable by configuring some routines to actively try to break bad code to find kernel problems (kmalloc()). Thus you should probably answer 'n' to the questions for "development", "experimental", or "debugging" features.
make allmodconfig builds as many drivers as loadable modules as possible, so it might be a better starting point than make allyesconfig.
You might consider grabbing a linux-config package from Debian Testing or from Stretch-backports and picking the kernel configuration from it. That's only a few kernel releases away from 5.1, so it should not be too hard to run make oldconfig with that configuration, and reading the help text before deciding on any kernel features added in versions 4.20, 5.0 and 5.1.
Note that the kernel packages from Debian should be already configured to be as generic as possible, and using that as a basis for your new kernel configuration might allow you to avoid some pitfalls with debugging features or settings related to system console. Debugging a kernel configuration that fails with no viable system console drivers is a special kind of nuisance.