It's a combination of color management and perception. Textures that are not smooth numerically (i.e., that lead to non-smooth displacements) may appear quite smooth when viewed as colors.
In your case, the Voronoi texture with F1/Minkowski distance outputs a decidedly non-smooth texture, with diagonal seams and sharp points at the maximums. A quick fix is to switch from "F1" to "Smooth F1" which reduces these sharp edges. Then, instead of using "Power" and "Brightness/Contrast" nodes which are hard to control, try mapping the Voronoi Distance into a reasonable range with a Map Range node and fine tuning the result with the Float Curve. You'll probably need to adjust the Displacement Scale as well. With a little fiddling, I came up with this modification to your network:

which looks like it's in the ballpark:

Here's the modified Blend file.
