In this paper, an algorithm for classifying five different varieties of rice, using the color and texture features is presented. The proposed algorithm consists of several steps: image acquisition, segmentation, feature extraction, feature selection, and classification. Sixty color and texture features were extracted from rice kernels. The Set of features contained redundant, noisy or even irrelevant information so features were examined by four different algorithms. Finally twenty-two features were selected as the superior ones. A back propagation neural network-based classifier was developed to classify rice varieties. The overall classification accuracy was achieved as 96.67%.