The press conference officially marked the end of his time as a special government employee heading up the Department of Government Efficiency