Ear recognition is one of the most relevant applications of image analysis. It’s a true challenge to build an automated system which exceeds human ability to recognize ears. Humans do not identify the ears ordinarily, so we are not skilled when we must deal with a large number of unknown ears. The modern computers, with an almost limitless memory and computational speed, should overcome humans' limitations. This work uses Ear classification problem to improve Deviance Information Criterion- Structural Hidden Markov Model (DIC-SHMM) by Convolutional Neural Network (CNN). HMM is a strong model for large size of features. While the CNN, as a most important technology for deep learning, used for image classification, to recognize persons by their ears as a one of unique biometric physiological characteristics. Three systems will be used to classify ear images, deep learning for the original image directly, deep learning for eigenvector as Principle Components Analysis (PCA) of the original image to compare them with proposed combining convolution layers of CNN with improved HMM for the original Image. The proposed system shows the best correction rate as 97.5%.