canal快速监听数据库变化-docker版

太过爱你忘了你带给我的痛 2024-03-24 09:59 135阅读 0赞

**目录**

一、拉取镜像

二、运行镜像

三、可能会出现的问题：镜像容器自动停止。

四、进入canal容器（目的是拷贝配置文件到宿主主机）

五、拷贝canal配置文件

六、开启数据库 Binlog 写入功能并建立一个canal账号

七、修改配置文件（instance.properties）

八、关闭之前的容器，重启新容器

九、python客户端

十、保证canal服务、mysql服务、客户端服务三方网络畅通

十一、测试

--------------------

#### 一、拉取镜像 ####

docker pull canal/canal-server

#### 二、运行镜像 ####

docker run --name canal -d canal/canal-server

docker logs canal

![e1c12b21e98040189410b50ee3dc755b.png][]

#### 三、可能会出现的问题：镜像容器自动停止。 ####

**原因：主机内存太小。**

**解决：内存搞大点。**

#### 四、进入canal容器（目的是拷贝配置文件到宿主主机） ####

docker exec -it canal bash

![84fb197aec894aa19eed74a2944082ec.png][]

#### 五、拷贝canal配置文件 ####

**在主机下新建文件夹/home/canal/conf**

![634de1d351ce467d9a60a36ed0db187d.png][]

docker cp canal:/home/admin/canal-server/conf/canal.properties /home/canal/conf
    docker cp canal:/home/admin/canal-server/conf/example/instance.properties /home/canal/conf

![14ed10fd2b42481ab129f85b58e4e5a7.png][]

#### 六、开启数据库 [Binlog][] 写入功能并建立一个canal账号 ####

修改[mysql配置文件][mysql]my.cnf或者my.ini![c6e300d4af8d411bb3a26e5b4dc18fa1.png][]

[mysqld]
    log-bin=mysql-bin # 开启 binlog
    binlog-format=ROW # 选择 ROW 模式
    server_id=1 # 配置 MySQL replaction 需要定义，不要和 canal 的 slaveId 重复

![5c3faf3d0d8e4a129da993bacdcb16f3.png][]

CREATE USER canal IDENTIFIED BY 'canal';  
    GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
    -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
    FLUSH PRIVILEGES;

#### 七、修改配置文件（instance.properties） ####

![5402639dcdc648b597f99009768efbbb.png][]

#### 八、关闭之前的容器，重启新容器 ####

docker stop canal

docker rm canal

#启动新的 这里-v是将外部的文件挂载到容器内部 这样就不用每次启动都要配置参数了
    docker run --name canal -p 11111:11111 -d \
    -v /home/canal/conf/instance.properties:/home/admin/canal-server/conf/example/instance.properties \
    -v /home/canal/conf/canal.properties:/home/admin/canal-server/conf/canal.properties \
    -v /home/canal/log/:/home/admin/canal-server/logs/ \
    canal/canal-server

![a8e2f34ca5564aca975ad9c031e007cd.png][]

![966d09749c414109b42da7bc2e4cdf4e.png][]

#### 九、python客户端 ####

pip install canal-python

import time
    
    from canal.client import Client
    from canal.protocol import EntryProtocol_pb2
    from canal.protocol import CanalProtocol_pb2
    
    client: Client = Client()
    client.connect(host='192.168.1.71', port=11111)
    # client.connect(host='127.0.0.1', port=11111)
    client.check_valid(username=b'', password=b'')
    client.subscribe(client_id=b'1001', destination=b'example', filter=b'.*\\..*')
    
    if __name__ == '__main__':
        while True:
            message = client.get(100)
            entries = message['entries']
            for entry in entries:
                entry_type = entry.entryType
                if entry_type in [EntryProtocol_pb2.EntryType.TRANSACTIONBEGIN, EntryProtocol_pb2.EntryType.TRANSACTIONEND]:
                    continue
                row_change = EntryProtocol_pb2.RowChange()
                row_change.MergeFromString(entry.storeValue)
                event_type = row_change.eventType
                header = entry.header
                database = header.schemaName
                table = header.tableName
                event_type = header.eventType
                for row in row_change.rowDatas:
                    format_data = dict()
                    if event_type == EntryProtocol_pb2.EventType.DELETE:
                        format_data['event_type_name'] = '删除'
                        for column in row.beforeColumns:
                            format_data[column.name] = column.value
                    elif event_type == EntryProtocol_pb2.EventType.INSERT:
                        format_data['event_type_name'] = '新增'
                        for column in row.afterColumns:
                            format_data[column.name] = column.value
                    else:
                        format_data['event_type_name'] = '更新'
                        format_data['before'] = format_data['after'] = dict()
                        for column in row.beforeColumns:
                            format_data['before'][column.name] = column.value
                        for column in row.afterColumns:
                            format_data['after'][column.name] = column.value
                    data = dict(
                        db=database,
                        table=table,
                        event_type=event_type,
                        data=format_data,
                    )
                    print(data)
            time.sleep(1)
        client.disconnect()

#### 十、保证canal服务、mysql服务、客户端服务三方网络畅通 ####

①canal与mysql畅通（在canal主机上ping mysql主机）

![5d4bf54fdafb434f9f4a17cd6295a704.png][]

②canal与客户端畅通（在mysql主机上ping客户端主机）

![0ba208b792dd490a992fce11031550a5.png][]![acb702b633f249a1887bdedda9488696.png][]

#### 十一、测试 ####

数据库新增一条数据、修改一条数据

![17e5ae1db23246318fb8282458cb3ae4.png][]

[e1c12b21e98040189410b50ee3dc755b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/6d94b46deb5845aaada4f7b32edd20ad.png
[84fb197aec894aa19eed74a2944082ec.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/8baf954c67d44f5792ad466cc6fadd10.png
[634de1d351ce467d9a60a36ed0db187d.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/ebd6796a8ada4503bf686b622927e1c4.png
[14ed10fd2b42481ab129f85b58e4e5a7.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/3f4971442aeb4048878f99357cfb1b59.png
[Binlog]: https://so.csdn.net/so/search?q=Binlog&spm=1001.2101.3001.7020
[mysql]: https://so.csdn.net/so/search?q=mysql%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6&spm=1001.2101.3001.7020
[c6e300d4af8d411bb3a26e5b4dc18fa1.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/8bf573252ec94383af2ff50f1694f458.png
[5c3faf3d0d8e4a129da993bacdcb16f3.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/134107a900014149a40f1621b18181a8.png
[5402639dcdc648b597f99009768efbbb.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/c0af0fa001cf4bd6a023150ccf58f1cf.png
[a8e2f34ca5564aca975ad9c031e007cd.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/55504a26dc7d4e25bf7c246365604d02.png
[966d09749c414109b42da7bc2e4cdf4e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/a92ec9a63ce84cd09628a999dc258137.png
[5d4bf54fdafb434f9f4a17cd6295a704.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/e2cb8bffb6eb4f378bafa3cad19ed724.png
[0ba208b792dd490a992fce11031550a5.png]: https://img-blog.csdnimg.cn/0ba208b792dd490a992fce11031550a5.png
[acb702b633f249a1887bdedda9488696.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/adb29067fda247e19f52e34bf8ebdf74.png
[17e5ae1db23246318fb8282458cb3ae4.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/b2297dcd033040a3bcbf6fb1baf9c30d.png