012利用tensorflow接口打包数据生成tfrecords文件

女爷i 2023-07-07 11:25 77阅读 0赞

下载好的models中有目标检测相应的脚本文件，能进行模型的训练、数据的打包、模型的测试、将模型转成pb文件等。

models下载官网：[https://github.com/tensorflow/models][https_github.com_tensorflow_models]

也可以下载我的百度云分享：  
链接:https://pan.baidu.com/s/1TNeg20BvkgxruqyTH\_ClxA 密码:rky0

models及目标检测环境配置：  
[https://blog.csdn.net/ayangann915/article/details/104357520][https_blog.csdn.net_ayangann915_article_details_104357520]

1.  进入models/research/object\_detection/dataset\_tools文件，里面有有create\_pascal\_tf\_record.py文件，是数据打包的脚本，主要是对VOC格式进行打包的数据脚本

SETS = ['train', 'val', 'trainval', 'test']：数据集的类别，测试集训练集等
    YEARS = ['VOC2007', 'VOC2012', 'merged']:数据集文件

要打包的内容：

example = tf.train.Example(features=tf.train.Features(feature={ 
          'image/height': dataset_util.int64_feature(height),
          'image/width': dataset_util.int64_feature(width),
          'image/filename': dataset_util.bytes_feature(
              data['filename'].encode('utf8')),
          'image/source_id': dataset_util.bytes_feature(
              data['filename'].encode('utf8')),
          'image/key/sha256': dataset_util.bytes_feature(key.encode('utf8')),
          'image/encoded': dataset_util.bytes_feature(encoded_jpg),
          'image/format': dataset_util.bytes_feature('jpeg'.encode('utf8')),
          'image/object/bbox/xmin': dataset_util.float_list_feature(xmin),
          'image/object/bbox/xmax': dataset_util.float_list_feature(xmax),
          'image/object/bbox/ymin': dataset_util.float_list_feature(ymin),
          'image/object/bbox/ymax': dataset_util.float_list_feature(ymax),
          'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
          'image/object/class/label': dataset_util.int64_list_feature(classes),
          'image/object/difficult': dataset_util.int64_list_feature(difficult_obj),
          'image/object/truncated': dataset_util.int64_list_feature(truncated),
          'image/object/view': dataset_util.bytes_list_feature(poses),
      }))

因为使用的是widerface数据集，所以要修改YEARS

YEARS = ['widerface']

文件路径：

examples_path = os.path.join(data_dir, year, 'ImageSets', 'Main',
                                      + FLAGS.set + '.txt')
    
    flags.DEFINE_string('label_map_path', '/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/models/research/object_detection/data/face_label_map.pbtxt',
                        'Path to label map proto')

1.  进入models/research/object\_detection/data文件夹下，有face\_label\_map.pbtxt文件，只有一个类别face，而pascal\_label\_map.pbtxt有20个类别，包含背景有21个类别
2.  使用官方给的脚本

Example usage:
        python object_detection/dataset_tools/create_pascal_tf_record.py \
            --data_dir=/home/user/VOCdevkit \
            --year=VOC2012 \
            --output_path=/home/user/pascal.record

就可以对VOC格式的数据进行打包

1.  进入models/research/object\_detection/sample/config文件夹，里面有配置文件，修改ssd\_mobilenet\_v1\_coco.config文件，也可新创建ssd\_mobilenet\_v1\_face.config在里面进行修改参数配置

ssd { 
        num_classes: 2#修改类别为2，人脸or非人脸两个类别，二分类问题
        box_coder { 
          faster_rcnn_box_coder { 
            y_scale: 10.0
            x_scale: 10.0
            height_scale: 5.0
            width_scale: 5.0
          }
        }
    
    #定义标准，50%概率为分界线
    argmax_matcher { 
            matched_threshold: 0.5
            unmatched_threshold: 0.5
            ignore_thresholds: false
            negatives_lower_than_unmatched: true
            force_match_for_each_row: true
          }
    #定义anchor的比例和尺寸
    ssd_anchor_generator { 
            num_layers: 6
            min_scale: 0.2
            max_scale: 0.95
            aspect_ratios: 1.0
            aspect_ratios: 2.0
            aspect_ratios: 0.5
            aspect_ratios: 3.0
            aspect_ratios: 0.3333
          }
    #输入图片大小
    image_resizer { 
          fixed_shape_resizer { 
            height: 300
            width: 300
          }
        }
    
    #bdnbox层，对应的卷积参数和预测的一些初始化参数
    box_predictor { 
          convolutional_box_predictor { 
            min_depth: 0
            max_depth: 0
            num_layers_before_predictor: 0
            use_dropout: false
            dropout_keep_probability: 0.8
            kernel_size: 1
            box_code_size: 4
            apply_sigmoid_to_scores: false
            conv_hyperparams { 
              activation: RELU_6,
              regularizer { 
                l2_regularizer { 
                  weight: 0.00004
                }
              }
              initializer { 
                truncated_normal_initializer { 
                  stddev: 0.03
                  mean: 0.0
                }
              }
              batch_norm { 
                train: true,
                scale: true,
                center: true,
                decay: 0.9997,
                epsilon: 0.001,
              }
            }
          }
        }
    #特征提取层，主干网络
    feature_extractor { 
          type: 'ssd_mobilenet_v1'#定义了主干网络所用的type
          min_depth: 16
          depth_multiplier: 1.0#网络宽度，channel的数量
          conv_hyperparams { 
            activation: RELU_6,
            regularizer { 
              l2_regularizer { 
                weight: 0.00004
              }
            }
            initializer { 
              truncated_normal_initializer { 
                stddev: 0.03
                mean: 0.0
              }
            }
            batch_norm { 
              train: true,
              scale: true,
              center: true,
              decay: 0.9997,
              epsilon: 0.001,
            }
          }
        }
    #网络优化的参数
    train_config: { 
      batch_size: 24
      optimizer { 
        rms_prop_optimizer: { 
          learning_rate: { 
            exponential_decay_learning_rate { 
              initial_learning_rate: 0.004
              decay_steps: 800720
              decay_factor: 0.95
            }
          }
          momentum_optimizer_value: 0.9
          decay: 0.9
          epsilon: 1.0
        }
      }
    #最后要定义训练样本的输入和验证样本的输入，这里需要修改路径，最后生成record文件
    train_input_reader: { 
      tf_record_input_reader { 
        input_path: "PATH_TO_BE_CONFIGURED/mscoco_train.record-?????-of-00100"
      }
      label_map_path: "PATH_TO_BE_CONFIGURED/mscoco_label_map.pbtxt"
    }
    
    eval_config: { 
      num_examples: 8000
      # Note: The below line limits the evaluation process to 10 evaluations.
      # Remove the below line to evaluate indefinitely.
      max_evals: 10
    }
    
    eval_input_reader: { 
      tf_record_input_reader { 
        input_path: "PATH_TO_BE_CONFIGURED/mscoco_val.record-?????-of-00010"
      }
      label_map_path: "PATH_TO_BE_CONFIGURED/mscoco_label_map.pbtxt"
      shuffle: false
      num_readers: 1
    }
    
    #修改后：
    train_input_reader: { 
      tf_record_input_reader { 
        input_path: "/Users/apple/Downloads/11人脸识别/数据集/widerface/tf-data/train.record"
      }
      label_map_path: "/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/models/research/object_detection/data/face_label_map.pbtxt"
    }
    
    eval_config: { 
      num_examples: 8000
      # Note: The below line limits the evaluation process to 10 evaluations.
      # Remove the below line to evaluate indefinitely.
      max_evals: 10
    }
    
    eval_input_reader: { 
      tf_record_input_reader { 
        input_path: "/Users/apple/Downloads/11人脸识别/数据集/widerface/tf-data/test.record"
      }
      label_map_path: "/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/models/research/object_detection/data/face_label_map.pbtxt"
      shuffle: false
      num_readers: 1
    }

1.  执行create\_face\_tf\_record.py文件

#test:
    (base) appledeMBP:~ apple$ cd /anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/models/research 
    (base) appledeMBP:research apple$ python3 object_detection/dataset_tools/create_face_tf_record.py --data_dir=/Users/apple/Downloads/11人脸识别/数据集 --year=widerface --output_path=/Users/apple/Downloads/11人脸识别/数据集/widerface/tf-data/test.record --set=test
    
    #train
    (base) appledeMBP:~ apple$ cd /anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/models/research 
    (base) appledeMBP:research apple$ python3 object_detection/dataset_tools/create_face_tf_record.py --data_dir=/Users/apple/Downloads/11人脸识别/数据集 --year=widerface --output_path=/Users/apple/Downloads/11人脸识别/数据集/widerface/tf-data/train.record --set=train

最后生成两个record文件

在这段代码中：

python3 object_detection/dataset_tools/create_face_tf_record.py --data_dir=/Users/apple/Downloads/11人脸识别/数据集 --year=widerface --output_path=/Users/apple/Downloads/11人脸识别/数据集/widerface/tf-data/train.record --set=train

–data\_dir= 要指向widerface数据集的上层文件夹  
–year= 是widerface数据集所在的文件夹名称  
–output\_path= 是生成的record文件所在的路径  
–set= 是选择用训练样本还是测试样本生成record文件，要和前面的xx.record文件保持一致

[https_github.com_tensorflow_models]: https://github.com/tensorflow/models
[https_blog.csdn.net_ayangann915_article_details_104357520]: https://blog.csdn.net/ayangann915/article/details/104357520