I found an answer here about using labels with the points but that only works for 1 column with the label, whereas in order to plot (x, y) along with the point I have to use 2.
So I need something along the lines of plot "data.txt" using ($1):($2):1:2 with labels, "data.txt" using 1:2 with ($1):($2):1:2 being (x):(y) the coordinate for the point on the image and 1:2 being the label, except that doesn't work since it isn't a valid syntax and so only the first value is plotted at the correct location.
Well I guess I can use something like plot "data.txt" using ($1+2):($2+2):1 with labels, "data.txt" using ($1+10):($2+2):2 with labels, "data.txt" using 1:2 to manually set the spacing but damn is that ugly/low-level/hackish/bad/etc.