Royal brothers began to drift apart years before Meghan