This paper presents the development and application of image analysis and computer vision system in defect detection of fruit surface in the agricultural field. Computer vision is a rapid, consistent inspection technique, which has expanded to varied industries. Monitoring and detecting defect is becoming a very important issue in fruit management since ripeness is perceived by customers as main quality indicator. In this paper we present a method for automatic defect detection of various fruits based on image processing techniques. The method was implemented, and tested on sample of different fruit images. Segmentation is one of the basic techniques in computer vision. Color is often thought as a property of an individual object and the color of this object comes from the visible light that reflects off the object surface. In this experiment we have implemented a method to quantify the standard color of fruit in HSV(Hue, saturation and Value) color spaces in order to achieve fruit image segmentation.HSV system is suggested as the best color space for quantification in fruit defect detection. In this article we shall give the results of the experiments we have carried out. We have made a comparative study between HSV and RGB color space and the results so formed demonstrate the feasibility of our proposed method in color segmentation for various fruits.