setting X/Y axis with motion tracking

I have tracked a video clip and have a number of tracking points. When playing back, the tracking appear to be in good position along the face of a building. Tracking error is about 0.48 - not that great, but workable for now.

I selected a tracker on the face of the building and set that as the origin. I then selected another tracker along a horizontal line and set that as the X axis. The grid re-aligns, but not where I expected - it is no where near the horizontal line of the building. Same with Y if I select a tracker vertical to the origin.

You can see the origin in the center of the wireframe plane, directly below it is the tracker I selected for the Y Axis (green) - it obviously didn’t align that way. What am I doing wrong?