Colorectal cancer starts in the colon or the rectum. These cancers can also be called colon cancer or rectal cancer.  Most colorectal cancers start as a growth (polyp) on the inner lining of the colon or rectum.