Speaking to enthusiastic supporters at the rally after nightfall, Sharif said "this mammoth crowd in Rawalpindi is in fact the start of a revolution".
The march is taking place amid tight security, with large numbers of police officers and paramilitary soldiers deployed both in Islamabad and along the Grand Trunk Road, the main highway connecting the capital with Lahore.
Mr Sharif was seen off in Islamabad by Mr Abbasi, along with the new cabinet and other party officials, Mr Sharif's political adviser Asif Kirmani said.
"Does Pakistan's elected prime minister deserve to be removed in such an insulting manner?" he asked.
Sharif was the 15th prime minister in Pakistan's 70-year history-roughly half of which has been spent under military rule-to be ousted before completing a full term.
Prime Minister Shahid Khaqan Abbasi on Tuesday said that Article 62 (1)(f) of the Constitution may be scrapped with the consultation of all parties. The timing and venue of gathering has not been announced, as it is not known how long he would take to reach Lahore.
Sharif criticized the investigators in the probe against him and his family, saying "our worst opponents were part of the JIT (Joint Investigation Team)" which was appointed by the Supreme Court earlier this year. The PMLN initially named his younger brother Shahbaz as his eventual successor and Shahbaz was set to contest an upcoming by-election to enter Parliament.
The PML-N has planned a "million march" to protest against Sharif's disqualification in the Panama Papers case.
Canada-based cleric of Pakistani-origin Tahir-ul-Qadri returned to Lahore ahead of the rally. He wants justice for eight of his supporters killed in anti-government rallies in 2014.
Former interior minister Chaudhry Nisar Ali Khan, through a statement issued by his spokesman, denied reports that he did not attend Nawaz Sharif's GT Road rally due to back pain.