图片验证码识别之卷积神经网络CNN

系统管理员 2023-07-07 08:41 40阅读 0赞

最近新冠肺炎肆虐，已在家办公三周左右，闲来无聊，来写一篇最近用到的卷积神经网络训练模型来识别图片验证码的blog。

最近在做一个网络爬虫的项目，爬取官方网站上的信息，需要输入查询内容及图片验证码上边的1-9及A-Z（除去O和I）字符进行验证，验证通过方可获取到返回的数据内容。  
起初用到了python推荐的tesserocr lib,但识别效率不是很高，因为我需要识别的验证码是倾斜的，并且带有干扰线  
![在这里插入图片描述][20200222162217836.jpg]  
这种就不太好验证了，一般tesserocr可以验证无干扰线、不黏连且无倾斜角度的字符验证码图片

故考虑到了CNN，通过训练样本训练模型来识别个性化的图片验证码，这里工作量最大的就是前期的训练集的获取了  
1、首先用爬虫的方式爬取网站上的原始图片验证码，大概获取了七八千张，注意，其中有许多是无法预处理的，所以原始图片越多越好。  
2、将原始图片转换成二值图片，0-黑，255-白  
处理逻辑即将那些干扰线处理成黑色，否则为白色（将值set为1），下一步会将这些孤立点作为噪声去掉（干扰线周围均为白色（1））

gray = img.convert('L')
    gray = gray.point(lambda x: 0 if x < 100 else 255, '1')

3、用9领域方法将孤立黑点去掉，去除原理是判断当前黑色点（0）周围的白色值（1）是否多于阈值，若是则认为是孤立点将其去除。

# 降噪
     noise_point_list = collect_noise_point(gray)
     remove_noise_pixel(gray, noise_point_list)

sum\_9\_region\_new(img,x,y) 函数是用9领域方法判断该点周围是否为1（白点），若超过阈值（0 < res\_9 < 3）并且该点为黑点，则remove\_noise\_pixel去除

def collect_noise_point(img):
        '''收集所有的噪点'''
        noise_point_list = []
        for x in range(img.width):
            for y in range(img.height):
                res_9 = sum_9_region_new(img, x, y)
                if (0 < res_9 < 3) and img.getpixel((x, y)) == 0:  # 找到孤立点
                    pos = (x, y)
                    noise_point_list.append(pos)
        return noise_point_list

def sum_9_region_new(img, x, y):
        '''确定噪点 '''
        cur_pixel = img.getpixel((x, y))  # 当前像素点的值
        width = img.width
        height = img.height
    
        if cur_pixel == 1:  # 如果当前点为白色区域,则不统计邻域值
            return 0
    
        # 因当前图片的四周都有黑点，所以周围的黑点可以去除
        if y < 2:  # 本例中，前两行的黑点都可以去除
            return 1
        elif y == height - 1:  # 最下面一行
            if x < 1 or x == width - 1:
                return 1
            else:
                sum = img.getpixel((x - 1, y - 1)) \
                      + img.getpixel((x - 1, y)) \
                      + img.getpixel((x, y - 1)) \
                      + cur_pixel \
                      + img.getpixel((x + 1, y - 1)) \
                      + img.getpixel((x + 1, y))
                return 6 - sum
        else:  # y不在边界
            if x < 2:  # 前两列
                return 1
            elif x == width - 1:  # 右边非顶点
                sum = img.getpixel((x - 1, y - 1)) \
                      + img.getpixel((x - 1, y)) \
                      + img.getpixel((x - 1, y + 1)) \
                      + img.getpixel((x, y - 1)) \
                      + cur_pixel \
                      + img.getpixel((x, y + 1))
                return 6 - sum
            else:  # 具备9领域条件的
                sum = img.getpixel((x - 1, y - 1)) \
                      + img.getpixel((x - 1, y)) \
                      + img.getpixel((x - 1, y + 1)) \
                      + img.getpixel((x, y - 1)) \
                      + cur_pixel \
                      + img.getpixel((x, y + 1)) \
                      + img.getpixel((x + 1, y - 1)) \
                      + img.getpixel((x + 1, y)) \
                      + img.getpixel((x + 1, y + 1))
                return 9 - sum

去除孤立黑色点的方法

def remove_noise_pixel(img, noise_point_list):
        '''根据噪点的位置信息，消除二值图片的黑点噪声，即将值set为1'''
        for item in noise_point_list:
            img.putpixel((item[0], item[1]), 1)

4、此时的原始图片已经经过预处理，将部分干扰线去掉了，当然由于部分干扰线和字符粘连在一起，或者较粗（这两种情况9领域判断失效），去除效果不是很好，这种情况的预处理图片不进行下一步的再处理。

5、用垂直投影法来对图片进行切割，图片验证码包括五个字符，原则上应该是切割成五个单独的字符，投影切割的原理是：以水平方向为方向轴，从左向右依次判断竖直方向上黑点的数目（（x,y）=(x,0)）,将y轴上的黑点总数放入list中（list.add(sum(0))）。然后从左向右依次找sum(0)>0的（x,y）为字符的左端点，sum(0)=0的(x,y)为字符的右端点，将图片切割成五个单字符，当然会有部分字符粘连在一起无法分割，这种情况和4中的干扰线无法去除一样，摒弃不进行下一步处理。

#竖直方向投影法切割
     cutpoints = vertical(gray)
    
     for i, item in enumerate(cutpoints):
                if(abs(item[0]-item[1]) < 5):
                    i-=1
                    continue
                box = (item[0], 0, item[1]+1, gray.height)
                region = gray.crop(box)
                region.save("laji" + "/" + os.path.basename(file) + str(i) + '.png')

# 竖直方向投影法
    def vertical(img):
        """传入二值化后的图片进行垂直投影"""
        pixdata = img.load()
        w, h = img.size
        ver_list = []
        # 开始投影
        for x in range(w):
            black = 0
            for y in range(h):
                if pixdata[x, y] == 0:
                    black += 1
            ver_list.append(black)
        # 判断边界
        l, r = 0, 0
        flag = False
        cuts = []
        for i, count in enumerate(ver_list):
            # 阈值这里为0
            if flag is False and count > 0:
                l = i
                flag = True
            if flag and count == 0:
                r = i
                flag = False
                if r-l >= 15:
                    cuts.append((l, (l+r)/2))
                    cuts.append(((l+r)/2, r))
                else:
                    cuts.append((l, r))
        return cuts

6、经过上边的处理，已将大部分图片验证码切割成了单字符图片，接下来就是无聊繁琐耗时的人肉分类工作了。将这个单字符分类到不同的文件夹

![文件夹以字符命名][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpbmZ1amlhbjE5OTk_size_16_color_FFFFFF_t_70]

7、CNN卷积神经网络训练模型

import cv2
    import pickle
    import os.path
    import numpy as np
    from imutils import paths
    from sklearn.preprocessing import LabelBinarizer
    from sklearn.model_selection import train_test_split
    from keras.models import Sequential
    from keras.layers.convolutional import Conv2D, MaxPooling2D
    from keras.layers.core import Flatten, Dense
    from helpers import resize_to_fit
    
    
    LETTER_IMAGES_FOLDER = "C:\\Users\\xxxx\\Desktop\\category"
    MODEL_FILENAME = "captcha_model.hdf5"
    MODEL_LABELS_FILENAME = "model_labels.dat"
    
    
    # initialize the data and labels
    data = []
    labels = []
    
    # loop over the input images
    for image_file in paths.list_images(LETTER_IMAGES_FOLDER):
        print(image_file)
        if "副本" in image_file:
            continue
        # Load the image and convert it to grayscale
        image = cv2.imread(image_file)
        image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    
        # Resize the letter so it fits in a 20x20 pixel box
        image = resize_to_fit(image, 20, 20)
    
        # Add a third channel dimension to the image to make Keras happy
        image = np.expand_dims(image, axis=2)
    
        # Grab the name of the letter based on the folder it was in
        label = image_file.split(os.path.sep)[-2]
    
        # Add the letter image and it's label to our training data
        data.append(image)
        labels.append(label)
    
    
    # scale the raw pixel intensities to the range [0, 1] (this improves training)
    data = np.array(data, dtype="float") / 255.0
    labels = np.array(labels)
    
    # Split the training data into separate train and test sets
    (X_train, X_test, Y_train, Y_test) = train_test_split(data, labels, test_size=0.25, random_state=0)
    
    # Convert the labels (letters) into one-hot encodings that Keras can work with
    lb = LabelBinarizer().fit(Y_train)
    Y_train = lb.transform(Y_train)
    Y_test = lb.transform(Y_test)
    
    # Save the mapping from labels to one-hot encodings.
    # We'll need this later when we use the model to decode what it's predictions mean
    with open(MODEL_LABELS_FILENAME, "wb") as f:
        pickle.dump(lb, f)
    
    # Build the neural network!
    model = Sequential()
    
    # First convolutional layer with max pooling
    model.add(Conv2D(20, (5, 5), padding="same", input_shape=(20, 20, 1), activation="relu"))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    
    # Second convolutional layer with max pooling
    model.add(Conv2D(50, (5, 5), padding="same", activation="relu"))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    
    # Hidden layer with 500 nodes
    model.add(Flatten())
    model.add(Dense(500, activation="relu"))
    
    # Output layer with 32 nodes (one for each possible letter/number we predict)
    model.add(Dense(32, activation="softmax"))
    
    # Ask Keras to build the TensorFlow model behind the scenes
    model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
    
    # Train the neural network
    model.fit(X_train, Y_train, validation_data=(X_test, Y_test), batch_size=32, epochs=10, verbose=1)
    
    # Save the trained model to disk
    model.save(MODEL_FILENAME)

通过如上的训练，得到训练模型

8、根据训练得到的模型和标签，来预测未知的单字符图片

from keras.models import load_model
    from helpers import resize_to_fit
    import numpy as np
    import cv2
    import pickle
    
    
    MODEL_FILENAME = "captcha_model.hdf5"
    MODEL_LABELS_FILENAME = "model_labels.dat"
    
    def parse_code(image):
    
        # Load up the model labels (so we can translate model predictions to actual letters)
        with open(MODEL_LABELS_FILENAME, "rb") as f:
            lb = pickle.load(f)
    
        # Load the trained neural network
        model = load_model(MODEL_FILENAME)
        image = cv2.imread(image)
        image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    
        letter_image = resize_to_fit(image, 20, 20)
    
        # Turn the single image into a 4d list of images to make Keras happy
        letter_image = np.expand_dims(letter_image, axis=2)
        letter_image = np.expand_dims(letter_image, axis=0)
    
        # Ask the neural network to make a prediction
        prediction = model.predict(letter_image)
    
        # Convert the one-hot-encoded prediction back to a normal letter
        letter = lb.inverse_transform(prediction)[0]
        print("CAPTCHA text is: {}".format(letter))
    
        return letter

将单字符图片作为入参传给parse\_code(image)方法即可。

--------------------

如上是如何使用CNN来训练得到可以识别单字符的模型。工作量比较大的是前期训练集的获取。卷积神经网络这块是通用的训练方法。希望能够给需要的童鞋带来帮助。

[20200222162217836.jpg]: https://img-blog.csdnimg.cn/20200222162217836.jpg
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpbmZ1amlhbjE5OTk_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200222193847533.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpbmZ1amlhbjE5OTk=,size_16,color_FFFFFF,t_70