How to apply a transformation to a shape in JavaFX?

If you only want to do rotations, simply add the values up. rotate is the "absolute" rotation, not a relative one. If you want to concatenate different transformations, I recommend using Transforms and adding them to the transforms list of the node.

As for finding finding the coordinates: You're proably be interested in the coordinates in the parent node. You can get those from the local coordinates using Node.localToParent:

@Override
public void start(Stage stage) throws IOException {
    Polygon p =  new Polygon(0,0,0,100,50,50); 
    p.getPoints(); // output before : 0,0,0,100,50,50

    p.getPoints(); // output after: 0,0,0,100,50,50

    StackPane root = new StackPane(p);
    Scene scene = new Scene(root, 500.0, 400.0);

    scene.setOnMouseClicked(evt -> {
        // add rotation and determine resulting position of (0, 0)
        Rotate r = new Rotate(90, (0+0+50) / 3d, (0+100+50) / 3d);
        p.getTransforms().add(r);
        System.out.println("(0, 0) is now at " + p.localToParent(0, 0));
    });

    stage.setScene(scene);
    stage.show();
}