Image processing has been widely used for automation purposes in modern agriculture. The algorithm development for the image segmentation is the most controversial and challenging issue in orchard environment which researchers encounter. This paper describes a robust algorithm based on artificial neural network (ANN) and genetic algorithm (GA) for segmenting grape clusters from leaves and background using color features near to harvest. GA was employed for optimizing of ANN structure and selecting supreme color features simultaneously. The results showed that GA specifies the 8 color features as supreme features and define 8–15-35–3 as the best structure of the ANN. The overall accuracy of the algorithm was 99.40%. The promising results in algorithm development described in this study lead to introduce it as a practical sensing tool in precision agriculture as well as those industrial facilities dealing with image analysis.