Sir Graham Brady has become the latest senior Conservative MP to announce they will not contest their seat at the next election.