The AMD RX 480 is a recent generation mid-range GPU that is good for 1080p gaming and entry-level VR. After a bit more googling, I came across an excellent tutorial on installing an AMD RX 480. I made further google searches and there were a few things I could try, but, frankly, I think that this Mac Pro is just too old to properly power the 1080 ti. I powered off the Mac Pro and put the old GPU back in and it booted up without issue. The Mac Pro powered on, but it didn't make the familiar Mac 'bing' on start up. I plugged in the two 6-pin power cables anyway and closed the side panel and fired up the Mac Pro and I waited for the login screen… and waited. No matter! It's only missing a measly 25W. Each 6-pin power cable provides 75W plus the PCI connector bus provides another 75W for a total of 225W. Unscrew the expansion card retainer plate.
Disconnect the 6-pin power cable from the old GPU.I shut down the Mac Pro and proceeded with the hardware installation. With my new macOS Sierra freshly installed, I was able to successfully install the NVIDIA beta drivers without issue.