Pepper is widely planted and used all over the world as fresh vegetable and spice. Genetic and morphological information of pepper are stored through seeds. Determination of seed variety is crucial for correctly identifying genetic materials. Pepper varieties cannot be easily classified even by an expert eye due to the very small size of seeds and visual similarities. Hence, more advanced technologies are required to determine the variety of a pepper seed. A classification method was proposed to discriminate pepper seed based on neural networks and computer vision. Image acquisition was conducted using an office scanner at a resolution of 1200 dpi. Image features representing color, shape, and texture were extracted and used to classify pepper seeds. By calculating features from different color components, a feature database was constructed. Effective features were selected using sequential feature selection with different criterion functions. As a result of the feature selection procedure, the number of the features was significantly reduced from 257 to 10. Cross validation rules were applied to obtain a reliable classification model by preventing overfitting. Different numbers of neurons in the hidden layer and various training algorithms were investigated to determine the best multilayer perceptron model. The best classification performance was obtained using 30 neurons in the hidden layer of the network. With this network, an accuracy rate of 84.94% was achieved using the sequential feature selection and the training algorithm of resilient back propagation in classifying eight pepper seed varieties.