From the documentation of Node#scaleXProperty():

Defines the factor by which coordinates are scaled about the center [emphasis added] of the object along the X axis of this Node. This is used to stretch or shrink the node either manually or by using an animation.

This scale factor is not included in layoutBounds by default, which makes it ideal for scaling the entire node after all effects and transforms have been taken into account.

The pivot point about which the scale occurs is the center of the untransformed layoutBounds [emphasis added].

The pivot point is the point that doesn’t move. Since the x-coordinate of the pivot point is not at 0 or width, any change in the horizontal scale means both “ends” have to move (and since it’s the center, both ends move equally). That’s why you’re seeing the origin (top left corner) move. Note the documentation of Node#scaleYProperty() is basically the same.

If you want to scale using a different pivot point then you need to use a Scale transform and add it to the Node#getTransforms() list. By default, the pivot point of a Scale will be (0, 0) (well, technically (0, 0, 0), but we don’t care about the third dimension in this case).

As for your bonus question:

Bonus question, why are the lines outside of the border in both of these examples?

That has to do with the StrokeType of the border. Apparently, the default stroke type of a border, at least for one created via CSS, is INSIDE. From what I can tell, you want a border with a stroke type of CENTERED. However, the corners of the ends of the lines will still be outside the pane’s border due to simple geometry. If you really want things to align, consider rounding both the ends of the lines and the corners of the border.

Example

Code:

import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.control.ToggleButton;
import javafx.scene.layout.Border;
import javafx.scene.layout.BorderPane;
import javafx.scene.layout.BorderStroke;
import javafx.scene.layout.BorderStrokeStyle;
import javafx.scene.layout.BorderWidths;
import javafx.scene.layout.Pane;
import javafx.scene.paint.Color;
import javafx.scene.shape.Line;
import javafx.scene.shape.StrokeLineCap;
import javafx.scene.shape.StrokeLineJoin;
import javafx.scene.shape.StrokeType;
import javafx.scene.transform.Scale;
import javafx.stage.Stage;

public class Main extends Application {

  public static void main(String[] args) {
    Application.launch(Main.class);
  }

  @Override
  public void start(Stage primaryStage) throws Exception {
    var scale = new Scale();
    var scaleBtn = new ToggleButton("Toggle scale");
    scaleBtn.setOnAction(_ -> {
      scale.setX(scaleBtn.isSelected() ? 0.5 : 1.0);
      scale.setY(scaleBtn.isSelected() ? 0.8 : 1.0);
    });

    var root = new BorderPane();
    root.setTop(scaleBtn);
    root.setCenter(createPane(scale, 150, 300));

    primaryStage.setScene(new Scene(root, 600, 400));
    primaryStage.show();
  }

  private Pane createPane(Scale scale, double width, double height) {
    var pane = new Pane();
    pane.setMinSize(width, height);
    pane.setPrefSize(width, height);
    pane.setMaxSize(width, height);
    pane.getTransforms().add(scale);
    pane.getChildren().addAll(
      createLine(Color.BLUE, 0, 0, width, height),
      createLine(Color.RED, 0, height, width, 0));

    // CSS approach
    pane.setStyle(
        """
        -fx-border-color: black;
        -fx-border-width: 5;
        -fx-border-style: solid centered line-join round;
        """);

    // programmatic approach
    // pane.setBorder(new Border(new BorderStroke(
    //   Color.BLACK,
    //   new BorderStrokeStyle(
    //     StrokeType.CENTERED,  // strokeType
    //     StrokeLineJoin.ROUND, // lineJoin (ROUND the corners)
    //     null,                 // lineCap
    //     1,                    // miterLimit
    //     0,                    // dashOffset
    //     null),                // dashArray
    //   null,
    //   new BorderWidths(5))));
  
    return pane;
  }

  private Line createLine(Color stroke, double startX, double startY, double endX, double endY) {
    var line = new Line(startX, startY, endX, endY);
    line.setStroke(stroke);
    line.setStrokeLineCap(StrokeLineCap.ROUND); // round the ends
    line.setStrokeWidth(5);
    return line;
  }
}

Output:

Supplemental answer if desiring more explanation, otherwise just refer to Slaw’s answer.

The shift is just a mathematical property of how scaling works, you are scaling about the center of the node, not the origin of the coordinate system.

See these notes on scaling for some background info:

• A scaling example.

When scaling is applied to a shape that is not centered at (0,0), then in addition to being stretched or shrunk, the shape will be moved away from 0 or towards 0. In fact, the true description of a scaling operation is that it pushes every point away from (0,0) or pulls every point towards (0,0). If you want to scale about a point other than (0,0), you can use a sequence of three transforms, similar to what was done in the case of rotation.

• The quick math is in notes on 2D transforms. Refer to the sections on scaling and composition.

To scale about a point, these transformations are applied:

1. Translate so that P1 is at the origin.
2. Scale.
3. Translate so that the point at the origin is at P1.

In your code, the point scaled about is the pane center, and you scale the entire pane:

pane.setScaleX(.5); 
pane.setScaleY(.8);

The pane is placed in the scene at (0,0) and is 100×100 wide, so centered at (50,50). This is important, as the scale methods on node scale about the center of the node.

When the pane is then scaled by .5x.8, the pane size changes from 100×100 -> 50×80, remaining centered at (50,50).

Its new display coordinates are top left -> 50 – (100 * .5) / 2, 50 – (100 * .8) / 2 = (25, 10) and size of 50×80.

You end up with a smaller node centered on the same position, so its bounds are now top left (25,10) and bottom right (75,90).

Stuff in the top left of the scaled pane looks like it shifts down and to the right, and anything in the bottom right of the pane would look like it shifted up and to the left.

You can see this in this simpler example that just draws a box.

import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.layout.Pane;
import javafx.stage.Stage;

public class SimpleScalingExample extends Application {

  @Override
  public void start(Stage stage) {
    Pane p = getPane();
    Scene scene = new Scene(new Pane(p), 720, 540);

    stage.setScene(scene);
    stage.setTitle("Example");
    stage.show();

    System.out.println(p.getBoundsInParent());
  }

  public Pane getPane() {
    Pane pane = new Pane();

    double width = 100;
    double height = 100;
    pane.setPrefSize(width, height);
    pane.setStyle("-fx-border-color: black; -fx-border-width: 5; -fx-border-style: solid inside;");
    pane.setScaleX(.5);
    pane.setScaleY(.8);

    return pane;
  }
}

It outputs:

BoundingBox [
    minX:25.0, minY:10.0, minZ:0.0, 
    width:50.0, height:80.0, depth:0.0, 
    maxX:75.0, maxY:90.0, maxZ:0.0
]

Which is exactly what the math calculated it would be.