Following in his father's footsteps, Elgin was made Viceroy of India in 1894. His viceroyalty was not a particularly notable one. Elgin himself did not enjoy the pomp and ceremony associated with the viceroyalty, and his conservative instincts were not well suited to a time of economic and social unrest. He returned to England in 1899 and was made a Knight of the Garter.
From 1902 to 1903, Elgin was made chairman of the commission that investigated the conduct of the Boer War. When the Liberals returned to power in 1905, Elgin became Secretary of State for the Colonies. As colonial secretary, he pursued a conservative policy, and opposed the generous settlement of the South African question proposed by Prime Minister Campbell-Bannerman, which was enacted more in spite of the Colonial Secretary's opposition than due to his efforts. Elgin retired from public life in 1908, dying nine years later at the family estate in Dunfermline.