Shell命令数据迁移实践

客官°小女子只卖身不卖艺 2021-09-12 05:28 799阅读 0赞

中奖纪录侧

数据量：千万级

实施方案：数据库层面迁移，新数据一一匹配

实施脚本：

1、sql整理出需要导出的数据：

select a,b,c,d,e,f,g
     from table_1 
                    into outfile '/data/aaa/haha.csv'
    fields terminated by ',' optionally enclosed by '\''
    lines terminated by '\r\n';

2、使用shell分割记录结果，分批导入

split -a 3 -l 20000 haha.csv target/table2_
      
    for i in `ls -1 target `
    do
        mysql -s -N -p'xxxxxxx' DPEmidas -e "load data infile '/data/aaa/target/$i'
    IGNORE into table table2
    fields terminated by ','  optionally enclosed by '\''
    lines terminated by '\r\n'"
        mv /data/aaa/target/$i /data/aaa/complete/
        sleep 1
    done

3、

获取 所有项目的git地址：
    git/config文件如下：
    [core]
    	repositoryformatversion = 0
    	filemode = false
    	bare = false
    	logallrefupdates = true
    	symlinks = false
    	ignorecase = true
    [remote "origin"]
    	url = ssh://git@git.dianpingoa.com/emidas/emidas-decorate.git
    	fetch = +refs/heads/*:refs/remotes/origin/*
    [branch "master"]
    	remote = origin
    	merge = refs/heads/master
    [branch "feature/search"]
    	remote = origin
    	merge = refs/heads/feature/search
    [branch "beta"]
    	remote = origin
    	merge = refs/heads/beta
    
    
    需要获取每一个url 那一行后面的地址，正则需要匹配  空格符  url = xxxx
    具体正则 (^\s*)url = (.*)，sed命令中需要转义  \ $ * + ( )
    
    sed -n 's/$^\s*$url = $.*$/\2/p' */.git/config > test.txt

[https://blog.csdn.net/z69183787/article/details/53693739][https_blog.csdn.net_z69183787_article_details_53693739]

[https_blog.csdn.net_z69183787_article_details_53693739]: https://blog.csdn.net/z69183787/article/details/53693739