Python：分割all_data.txt----＞train_data.txt、val_data.txt、test_data.txt

迷南。 2023-09-25 14:56 94阅读 0赞

all\_data.txt

小儿珠蛋白生成障碍性贫血	acompany_with	腹水
    病毒性感冒	acompany_with	鼻炎
    其他类精神活性物质依赖	acompany_with	梅毒
    炎性乳腺癌	acompany_with	败血症
    小儿选择性免疫球蛋白A缺乏症	acompany_with	萎缩性胃炎
    烟雾病	acompany_with	脑梗死
    异位急性阑尾炎	acompany_with	败血症
    老年人高血压	acompany_with	心肌梗死
    ......

共312215行，选80%作为训练集、10%作为验证集、10%作为测试集。

## 方法一：太慢 ##

import random
    
    train_val_id_list = random.sample(range(312215), 31221 * 9)
    val_id_list = random.sample(train_val_id_list, 31221)
    
    idx = 0
    with open('all_data.txt', 'r', encoding='utf-8') as fr:
        lines = fr.readlines()
        for line in lines:
            if idx not in train_val_id_list:
                with open('test.txt', 'a', encoding='utf-8') as fw:
                    fw.write(line)
            elif idx in val_id_list:
                with open('val.txt', 'a', encoding='utf-8') as fw:
                    fw.write(line)
            else:
                with open('train.txt', 'a', encoding='utf-8') as fw:
                    fw.write(line)
            idx += 1

## 方法二： 很快 ##

# 使用train_test_split()将选取的固定大小的数据集，按照一定的比例，如9：1随机划分为训练集，测试集。
    # 并分别将划分好的数据集进行写入到固定目录下
     
    import pandas as pd
    from sklearn.model_selection import train_test_split
    
    
    
    fr = open('all_data.txt', 'r', encoding='utf-8')
    all_lines = fr.readlines()
    train_data, val_test_data = train_test_split(all_lines, train_size=0.8, test_size=0.2)
    val_data, test_data = train_test_split(val_test_data, train_size=0.5, test_size=0.5)
    print("len(train_data) = ", len(train_data))
    print("len(val_data) = ", len(val_data))
    print("len(test_data) = ", len(test_data))
    fr.close()
    
    for line in train_data:
        with open('train.txt', 'a', encoding='utf-8') as fw:
            fw.write(line)
    
    for line in val_data:
        with open('val.txt', 'a', encoding='utf-8') as fw:
            fw.write(line)
    
    for line in test_data:
        with open('test.txt', 'a', encoding='utf-8') as fw:
            fw.write(line)
    
    
    # all_data=pd.read_csv("D:/shujuji/MobiAct_Dataset_v2.0/kalman/train/train12.csv")
    # train_data, test_data = train_test_split(all_data, train_size=0.9, test_size=0.1)
    
    # train_data.to_csv("D:/shujuji/MobiAct_Dataset_v2.0/kalman/train/train12_9.csv",index=False)
    # test_data.to_csv("D:/shujuji/MobiAct_Dataset_v2.0/kalman/train/train12_1.csv",index=False)

![16a80c2ae47f7304045a382c35bbbe88.jpeg][]

[python随机划分数据集\_weixin\_51443598的博客-CSDN博客\_python随机划分数据集][python_weixin_51443598_-CSDN_python]

[Python数据分析中的 训练集、验证集、测试集 - 知乎][Python_ _ -]

[16a80c2ae47f7304045a382c35bbbe88.jpeg]: https://img-blog.csdnimg.cn/img_convert/16a80c2ae47f7304045a382c35bbbe88.jpeg
[python_weixin_51443598_-CSDN_python]: https://blog.csdn.net/weixin_51443598/article/details/123412000
[Python_ _ -]: https://zhuanlan.zhihu.com/p/474941218