I finally got to run a SQL trace on what was going on. A thing that I probably should have tried a lot earlier. From the trace I could see that the cross reference for some reason was in an infinite loop on a particular form. I checked over a few hours and it never did seem to get past that form.
I killed the process and started a new cross reference agin, but this time I deleted the existing cross reference first:
When that was finished, I started a new cross reference update with the "Update all" option checked and this time cross reference was updated within the expected time frame.