Nelms, 65, has been on the job since August 2007. His last day is Aug. 6.
During his tenure, Nelms saw NCCU increase online course offerings, add the university's first Ph.D. program in 50 years and move from Division II to Division I athletics.
University of North Carolina President Tom Ross named Charles Becton as interim chancellor while the N.C. Central Board of Trustees searches for a permanent successor.
Becton is a Durham lawyer, a law professor at UNC-Chapel Hill and a former judge on the North Carolina Court of Appeals.
“Chancellor Nelms has made extraordinary contributions to the academic quality and stature of N.C. Central University that will hold it in good stead as we search for his successor," Ross said in a statement. "Judge Charles Becton is known and respected throughout the Durham community and the state of North Carolina for his sound judgment and ability to get things done, so I’m confident that NCCU won’t lose any momentum as the search for a permanent chancellor gets under way."