OpenStack记录

一时失言乱红尘 2023-07-10 14:14 44阅读 0赞

官网文档：https://docs.openstack.org/mitaka/zh\_CN/install-guide-ubuntu/nova-compute-i

官网文档：https://docs.openstack.org/mitaka/zh\_CN/install-guide-ubuntu/nova-compute-install.html

OpenStack项目主要提供：计算服务、存储服务、镜像服务、网络服务，均依赖于身份认证keystone的支撑。其中的每个项目可以拆开部署，同一项目也可以部署在多台=物理机上，并且每个服务都提供了应用接口程序（API），方便与第三方集成调用资源。

### https://app.yinxiang.com/fx/26611639-2495-4bf0-9233-74c1ad98f64a                   参考链接 ###

（结合自己的文档：注意配置文件修改的权限）

### 环境准备 ###

#### 1.安装OpenStack环境得硬件需求 ####

centos7.4

*  CPU 支持intel 64或AMD 64 CPU扩展，并启用AMD-H或intel VT硬件虚拟化支持的64位x86处理器
 *  内存 >=2G
 *  磁盘空间 >=50G

#### 2.虚拟机分配 ####

<table> 
 <thead> 
  <tr> 
   <th>主机名</th> 
   <th>操作系统</th> 
   <th>IP地址</th> 
   <th>备注</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>controller</td> 
   <td>CentOS-7.4-x86_64</td> 
   <td>192.168.66.197</td> 
   <td>控制节点</td> 
  </tr> 
  <tr> 
   <td>compute</td> 
   <td>CentOS-7.4-x86_64</td> 
   <td>192.168.66.147</td> 
   <td>计算节点</td> 
  </tr> 
  <tr> 
   <td>cinder</td> 
   <td>CentOS-7.4-x86_64</td> 
   <td>192.168.66.128</td> 
   <td>块存储节点</td> 
  </tr> 
 </tbody> 
</table>

#### 3.三台主机关闭虚拟机防火墙SElinux ####

都要操作如下：

\[root@controller ~\]\# systemctl disable firewalld.service

\[root@controller ~\]\# systemctl stop firewalld.service

\[root@controller ~\]\# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

#### 4.配置域名解析 ####

都要操作：修改所有主机名

\[root@controller ~\]\# hostnamectl set-hostname controller

\[root@controller ~\]\# hostnamectl set-hostname compute

\[root@controller ~\]\# hostnamectl set-hostname cinder  
修改后，重启主机

都要操作：修改所有主机hosts文件

\[root@controller ~\]\# cat /etc/hosts  
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4  
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6  
192.168.66.197 controller  
192.168.66.147 compute  
192.168.66.128 cinder

测试连通性  
都要操作，每台主机ping另外两台主机  
![cef432f952d88fd2d528968e97c28abd-32048][]

### 开始搭建OpenStack ###

#### 1.配置阿里云的yum源 ####

都要操作：配置阿里云yum源，备份默认yum源

\[root@controller ~\]\# mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

都要操作：下载最新的yum源

\[root@controller ~\]\# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo  
![d2deca82a9f9d29a843a9c049bdb7c99-24493][]

#### 2.安装配置NTP服务 ####

在controller节点安装配置chrony服务

\[root@controller ~\]\# yum -y install chrony  
![2358ae6eadc988010252aa81502bd22f-12224][]  
启动服务，设置开机自启

\[root@controller ~\]\# systemctl enable chronyd

\[root@controller ~\]\# systemctl restart chronyd

在compute节点安装配置chrony服务

\[root@compute ~\]\# yum -y install chrony  
![43c073fdabc59720a6128f9b433e6371-15695][]  
启动服务，设置开机自启

\[root@compute ~\]\# systemctl enable chronyd

\[root@compute ~\]\# systemctl restart chronyd

在cinder节点安装配置chrony服务  
![3039b8e48e59479fa22d180b6d8fe231-14037][]  
启动服务，设置开机自启

\[root@cinder ~\]\# systemctl enable chronyd

\[root@cinder ~\]\# systemctl restart chronyd

验证时钟同步服务

\[root@controller ~\]\# chronyc sources（每台主机可以执行此命令）  
![5c00d626f153e227fa089a7ebcbaf161-15368][]

#### 3.启用OpenStack库 ####

管理节点部署

\[root@controller ~\]\#yum install centos-release-openstack-queens -y

\[root@controller ~\]\#yum upgrade -y //在主机上升级包

\[root@controller ~\]\#yum install python-openstackclient -y //安装openstack客户端

\[root@controller ~\]\#yum install openstack-selinux -y //安装openstack-selinux，便于自动管理openstack的安全策略

#### 4.MySQL数据库部署 ####

管理节点部署

\[root@controller ~\]\#yum -y install mariadb mariadb-server python2-PyMySQL

配置文件修改  
vim /etc/my.cnf.d/mariadb-server.cnf

\[mysqld\]  
datadir=/var/lib/mysql  
socket=/var/lib/mysql/mysql.sock  
log-error=/var/log/mariadb/mariadb.log  
pid-file=/var/run/mariadb/mariadb.pid  
bind-address = 192.168.66.197 //修改为控制节点IP，使其他节点可以通过管理网络访问数据库  
default-storage-engine = innodb  
innodb\_file\_per\_table = on  
max\_connections = 4096  
collation-server = utf8\_general\_ci  
character-set-server = utf8  
启动服务，设置为开机自启

\[root@controller ~\]\#systemctl enable mariadb.service

\[root@controller ~\]\#systemctl start mariadb.service

对数据库进行安全加固

\[root@controller ~\]\# mysql\_secure\_installation  
![1fa38b35cf049cfd441034b6057e1e83-594281][]

#### 5.0 安装配置Messaging server-RabbitMQOpenStack ####

使用message queue协调操作和各服务器的状态信息。消息队列服务一般运行在控制节点上。  
在管理节点安装RabbitMQ

\[root@controller ~\]\# yum -y install rabbitmq-server  
开启服务并设置为开机自启

\[root@controller ~\]\# systemctl enable rabbitmq-server.service

\[root@controller ~\]\# systemctl start rabbitmq-server.service  
查看服务

\[root@controller ~\]\# netstat -ntap | grep 5672  
![8158d7641b4d48f51a585fc16e40dd94-20499][]

#### 6. 添加OpenStack用户 ####

管理节点

\[root@controller ~\]\# rabbitmqctl add\_user openstack 123456  
//创建用户openstack，密码为123456  
![d4aa986409ae92459ad1d48e71473247-4495][]

\[root@controller ~\]\# rabbitmqctl set\_permissions openstack ".*" ".*" ".\*" //授予新建用户权限  
![b6aa46669216208e43b5e39af576efb5-4911][]

#### 7.部署memcached服务 ####

管理节点  
安装软件

\[root@controller ~\]\# yum -y install memcached python-memcached  
修改配置文件  
![866ea74c7d71819d363c69e493eb15ab-10038][]  
开启服务并设置为开机自启

\[root@controller ~\]\# systemctl enable memcached.service

\[root@controller ~\]\# systemctl start memcached.service

#### 8. 部署etcd服务 ####

etcd是一个分布式，一致的键值存储，用于共享配置和服务发现，特点是，安全，具有可选客户端证书身份验证的自动TLS；快速，基准测试10,000次/秒；可靠，使用Raft正确分发。  
管理节点  
安装软件

\[root@controller ~\]\# yum -y install etcd  
修改配置文件，改为管理节点的IP  
![2090ed8ff4fef442c94e2843581404f4-24748][]

> ETCD\_INITIAL\_CLUSTER  
> ETCD\_INITIAL\_ADVERTISE\_PEER\_URLS  
> ETCD\_ADVERTISE\_CLIENT\_URLS  
> ETCD\_LISTEN\_CLIENT\_URLS  
> \#\[Member\]  
> ETCD\_DATA\_DIR="/var/lib/etcd/default.etcd"  
> ETCD\_LISTEN\_PEER\_URLS="http://192.168.66.197:2380"  
> ETCD\_LISTEN\_CLIENT\_URLS="http://192.168.66.197:2379"  
> ETCD\_NAME="controller"  
> \#\[Clustering\]  
> ETCD\_INITIAL\_ADVERTISE\_PEER\_URLS="http://192.168.66.197:2380"  
> ETCD\_ADVERTISE\_CLIENT\_URLS="http://192.168.66.197:2379"  
> ETCD\_INITIAL\_CLUSTER="controller=http://192.168.66.197:2380"  
> ETCD\_INITIAL\_CLUSTER\_TOKEN="etcd-cluster-01"  
> ETCD\_INITIAL\_CLUSTER\_STATE="new"

开启服务并设置为开机自启

\[root@controller ~\]\# systemctl enable etcd.service

\[root@controller ~\]\# systemctl start etcd.service

#### 9. 部署keystone认证服务 ####

Identity服务为其他OpenStack服务提供验证和授权服务，为所有服务提供终端目录，其他OpenStack服务将身份认证当作通用统一API来使用。此外，提供用户信息但是不在OpenStack项目中的服务（如LDAP服务）可被整合进先前存在的基础设施中。  
为了从identify服务中获益，其他的OpenStack服务需要与他合作。当某个OpenStack服务需要与他合作。当某个OpenStack服务收到来自用户的请求时，该服务询问identify服务，验证该用户是否有权限进行此次请求，身份验证服务包括以下组件服务器：一个中心化的服务器使用RESTful接口来提供认证和授权服务驱动：驱动或服务后端被整合进集中式服务器中。它们被用来访问OpenStack外部仓库的身份信息，并且它们可能已经存在于OpenStack被部署在的基础设施中，如SQL数据库模块：中间件模块运行于使用身份验证服务的OpenStack组件的地址空间中。这些模块拦截服务请求，取出用户凭据，并将它们送入中央服务器寻求授权。中间件模块和OpenStack组件间的整合使用python web服务器网关接口。  
当安装OpenStack自身服务时，用户必须将之注册到其OpenStack安装环境的每个服务。身份服务才可以追踪到哪些OpenStack服务已经安装，以及在网络中定位它们。

Keystone服务的安装配置  
管理节点操作  
配置Mysql数据库及授权  
![4da716f169f1a1c53c64d7d7e94485f6-29629][]  
\[root@controller ~\]\# mysql -uroot -pcdp12345

Welcome to the MariaDB monitor. Commands end with ; or \\g.  
Your MariaDB connection id is 12  
Server version: 10.1.20-MariaDB MariaDB Server

Type 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.

MariaDB \[(none)\]> create database keystone;  
Query OK, 1 row affected (0.00 sec)

MariaDB \[(none)\]> *grant all privileges on keyston. to 'keystone'@'localhost' identified by '123456';*\*  
Query OK, 0 rows affected (0.00 sec)

MariaDB \[(none)\]> *grant all privileges on keyston. to 'keystone'@'%' identified by '123456';*\*  
Query OK, 0 rows affected (0.00 sec)

MariaDB \[(none)\]> flush privileges;  
Query OK, 0 rows affected (0.00 sec)

MariaDB \[(none)\]> exit

安装软件包

\[root@controller ~\]\# yum install openstack-keystone httpd mod\_wsgi -y  
修改配置文件（keyston.conf）

vim /etc/keystone/keystone.conf

\[database\]  
connection = mysql+pymysql://keystone:123456@controller/keystone

\[token\]  
provider = fernet //2922行，安全消息传递算法

同步数据库

\[root@controller ~\]\# su -s /bin/sh -c "keystone-manage db\_sync" keystone  
此时keystone库中已经有许多的表

（注：如果看不到表，修改配置文件：![a2895ef4406abab6aa1194614dcd3411-4236][]改为IP地址，看看日志报错：\[root@controller ~\]\# tail -f -n 20 /var/log/keystone/keystone.log）

正常如下：  
![e41474cd4a25eb299080b871a08f1f24-36097][]  
初始化数据库

\[root@controller ~\]\# keystone-manage fernet\_setup --keystone-user keystone --keystone-group keystone  
\*\*\[root@controller ~\]\#keystone-manage credential\_setup --keystone-user keystone --keystone-group keystone  
\*\[root@controller ~\]\# \*keystone-manage bootstrap --bootstrap-password 123456 \\

> \--bootstrap-admin-url http://controller:35357/v3/  
> \--bootstrap-internal-url http://controller:5000/v3/  
> \--bootstrap-public-url http://controller:5000/v3/  
> \--bootstrap-region-id RegionOne

admin用户创建完成  
![55756e73831882a5c5a60e01b7422570-7509][]

#### 10. 配置Apache服务 ####

修改主机名  
![d9fce64a0f37fd319e8e8cf5ee3b879d-5890][]

\[root@controller ~\]\# vim /etc/httpd/conf/httpd.conf

\[root@controller ~\]\# ln -s /usr/share/keystone/wsgi- keystone.conf /etc/httpd/conf.d/ \\创建软链接  
启动服务，设置为开机自启

\[root@controller ~\]\# systemctl enable httpd.service

\[root@controller ~\]\# systemctl start httpd.service

设置环境变量脚本  
export OS\_PROJECT\_DOMAIN\_NAME=Default  
export OS\_USER\_DOMAIN\_NAME=Default  
export OS\_PROJECT\_NAME=admin  
export OS\_USERNAME=admin  
export OS\_PASSWORD=123456export OS\_AUTH\_URL=http://controller:5000/v3export OS\_IDENTITY\_API\_VERSION=3export OS\_IMAGE\_API\_VERSION=2  
![192e1b749abdc0a9e099a48760452cdf-16503][]

#### 11. 创建域，项目用户和角色 ####

创建域

\[root@controller ~\]\# openstack domain create --description "Domain" example  
![c810e26d0719a40f4c4b89bf2539f84a-9462][]

创建项目

\[root@controller ~\]\# openstack project create --domain default --description "Service Project" service  
![7350d5d58fc92d4882c2011d634cc176-13404][]

创建平台demo项目

\[root@controller ~\]\# openstack project create --domain default --description "Demo Project" demo  
![e139da66ff29c3b1447c939122fea934-13354][]

创建demo用户

\[root@controller ~\]\# openstack user create --domain default --password-prompt demo  
![e325384ec6fbb0da02a260d02f501890-22698][]

创建用户角色

\[root@controller ~\]\# openstack role create user  
![4ba0c70ffe9c4d8b3fed0c1bf55baa73-6550][]

添加用户角色到demo项目和用户

\[root@controller ~\]\# openstack role add --project demo --user demo user \\该没有返回值

验证keystone  
取消环境变量

\[root@controller ~\]\# unset OS\_AUTH\_URL OS\_PASSWORD

admin用户返回的认证token

\[root@controller ~\]\# openstack --os-auth-url http://controller:35357/v3 \\

> \--os-project-domain-name Default --os-user-domain-name Default  
> \--os-project-name admin --os-username admin token issue  
> ![f9b1d83ed14783aaf2433f8f1438b7b4-30437][]

demo用户返回的认证token

\[root@controller ~\]\# openstack --os-auth-url http://controller:5000/v3 \\

> \--os-project-domain-name Default --os-user-domain-name Default  
> \--os-project-name demo --os-username demo token issue  
> ![7f414371ccf6bb5888ba493af0b3feb4-26869][]

#### 12. 创建openstack客户端环境脚本 ####

创建admin-openrc脚本  
export OS\_PROJECT\_DOMAIN\_NAME=Default  
export OS\_USER\_DOMAIN\_NAME=Default  
export OS\_PROJECT\_NAME=admin  
export OS\_USERNAME=admin  
export OS\_PASSWORD=123456export OS\_AUTH\_URL=http://controller:5000/v3export OS\_IDENTITY\_API\_VERSION=3export OS\_IMAGE\_API\_VERSION=2  
![397b05329f4e204553aa8819b50881ab-13988][]

创建demo-openrc脚本  
export OS\_PROJECT\_DOMAIN\_NAME=Default  
export OS\_USER\_DOMAIN\_NAME=Default  
export OS\_PROJECT\_NAME=demo  
export OS\_USERNAME=demo  
export OS\_PASSWORD=123456export OS\_AUTH\_URL=http://controller:5000/v3export OS\_IDENTITY\_API\_VERSION=3export OS\_IMAGE\_API\_VERSION=2  
![5b663299c05c17862aadaa44e4c07d9e-14763][]

使用脚本验证返回值，查看admin用户的token信息

\[root@controller ~\]\# source ~/admin-openrc

\[root@controller ~\]\# openstack token issue  
![9c26cbcb54160162a7e575969abd1650-19792][]

#### 13. 镜像服务 ####

管理节点操作  
配置MySQL数据库及授权  
![9e4ffacb5ddb7e7c5dc6ddecf9d93ee8-40919][]  
MariaDB \[(none)\]> create database glance;  
MariaDB \[(none)\]> *grant all privileges on glance. to 'glance'@'localhost' identified by '123456';*\*  
MariaDB \[(none)\]> *grant all privileges on glance.* to 'glance'@'%' identified by '123456';\*  
MariaDB \[(none)\]> flush privileges;

获取admin用户的环境变量

\[root@controller ~\]\# source admin-openrc

\[root@controller ~\]\# export | grep OS\_  
![ecc6a39825f436d4c9d8db81e5d7606b-15738][]

创建glance用户

\[root@controller ~\]\# openstack user create --domain default --password-prompt glance  
![53a4e796d2e413ff648d9b091cf584f6-15254][]

admin用户添加到glance用户和项目中

\[root@controller ~\]\# openstack role add --project service --user glance admin \\没有返回值

创建glance服务

\[root@controller ~\]\# openstack service create --name glance --description "OpenStack lmage" image  
![f56963d2f1cd685dd82721ec910a6068-10785][]

创建镜像服务API端点  
OpenStack使用三种API端点变种代表每种服务：admin、internal、public。

\[root@controller ~\]\# openstack endpoint create --region RegionOne image public http://controller:9292

\[root@controller ~\]\# openstack endpoint create --region RegionOne image internal http://controller:9292

\[root@controller ~\]\# openstack endpoint create --region RegionOne image admin http://controller:9292  
![b9095f5fc5d64250b1e5134510f337af-28511][]  
![b4449a8339f40764d8dec7323c56be7b-15172][]

安装glance包

\[root@controller ~\]\# yum -y install openstack-glance

创建images文件夹，并修改属性

\[root@controller ~\]\# mkdir /var/lib/glance/images

\[root@controller ~\]\# cd /var/lib/

\[root@controller lib\]\# chown -hR glance:glance glance

修改glance-api.conf配置文件  
![76b2fcb980b26e00b6d55a9937f5c9a9-34487][]

修改glance-registry.conf配置文件  
![fb049f15ad32585e5d6d3f3716112715-25728][]

同步镜像数据库

\[root@controller ~\]\# su -s /bin/sh -c "glance-manage db\_sync" glance  
![e6a5acdfc1d7d3ffaf881f5198f4e653-20310][]

启动服务

\[root@controller ~\]\# systemctl enable openstack-glance-api.service

\[root@controller ~\]\# systemctl start openstack-glance-api.service

\[root@controller ~\]\# systemctl enable openstack-glance-registry.service

\[root@controller ~\]\# systemctl start openstack-glance-registry.service  
![d1b726a3ee9a8ba4d25833387bbb8441-19312][]

验证上传镜像  
获取admin用户的环境变量并下载镜像

\[root@controller ~\]\# source ~/admin-openrc

\[root@controller ~\]\# wget http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86\_64-disk.img //下载一个小型linux镜像进行测试  
![085ef441570f9e44adf99bf4705580c9-24327][]

上传镜像  
使用QCOW2磁盘格式，裸容器格式和公开可见性将图像上传到Image服务，以便所有项目都可以访问它

\[root@controller ~\]\# openstack image create "cirros" --file cirros-0.3.5-x86\_64-disk.img --disk-format qcow2 --container-format bare --public  
![a758a71844d6c2362a9cafbbc22c58c1-28751][]

查看上传的镜像

\[root@controller ~\]\# openstack image list  
![8ce26b3e90c09126c1197a257897776a-6371][]

#### 14.部署compute服务 ####

在controller节点上操作  
安装与配置  
配置MySQL数据库及授权  
CREATE DATABASE nova\_api;  
CREATE DATABASE nova;  
CREATE DATABASE nova\_cell0;  
GRANT ALL PRIVILEGES ON nova\_api.\* TO 'nova'@'localhost' IDENTIFIED BY '123456';  
GRANT ALL PRIVILEGES ON nova\_api.\* TO 'nova'@'%' IDENTIFIED BY '123456';  
GRANT ALL PRIVILEGES ON nova.\* TO 'nova'@'localhost' IDENTIFIED BY '123456';  
GRANT ALL PRIVILEGES ON nova.\* TO 'nova'@'%' IDENTIFIED BY '123456';  
GRANT ALL PRIVILEGES ON nova\_cell0.\* TO 'nova'@'localhost' IDENTIFIED BY '123456';  
GRANT ALL PRIVILEGES ON nova\_cell0.\* TO 'nova'@'%' IDENTIFIED BY '123456';  
![a5ab992c2b3d7bcfc48b208957c44263-58557][]

创建nova用户

\[root@controller ~\]\# source ~/admin-openrc

\[root@controller ~\]\# openstack user create --domain default --password-prompt nova  
![c7281e0beee5c12c80feeb095564987b-17053][]

添加admin用户为nova用户

\[root@controller ~\]\# openstack role add --project service --user nova admin

创建nova服务端点

\[root@controller ~\]\# openstack service create --name nova --description "OpenStack Compute" compute  
![101fef3890957851eabe20e63aee3091-10391][]

创建compute API 服务端点  
\[root@controller ~\]\# openstack endpoint create --region RegionOne compute public http://controller:8774/v2.1  
\[root@controller ~\]\# openstack endpoint create --region RegionOne compute internal http://controller:8774/v2.1  
\[root@controller ~\]\# openstack endpoint create --region RegionOne compute admin http://controller:8774/v2.1  
![55bd4f9cee6524dced4f9bcaf0be2385-42275][]

创建一个placement服务用户

\[root@controller ~\]\# openstack user create --domain default --password-prompt placement  
![f8eae57c58d36c1a6e1cb4e5e0dbe8b1-15222][]

添加placement用户为项目服务admin角色

\[root@controller ~\]\# openstack role add --project service --user placement admin

在服务目录创建Placement API服务

\[root@controller ~\]\# openstack service create --name placement --description "Placement API" placement  
![60396ed4aa449f1b60d8991e89625c28-10530][]

创建Placement API服务端点

\[root@controller ~\]\# openstack endpoint create --region RegionOne placement public http://controller:8778

\[root@controller ~\]\# openstack endpoint create --region RegionOne placement internal http://controller:8778

\[root@controller ~\]\# openstack endpoint create --region RegionOne placement admin http://controller:8778  
![c5b910ccda8664a13f872d8c0b1af8b0-43992][]

安装软件包

\[root@controller ~\]\# yum install openstack-nova-api openstack-nova-conductor openstack-nova-console openstack-nova-novncproxy openstack-nova-scheduler openstack-nova-placement-api -y  
![cbe6fc31c2b5b77b149ba73cf9c91aee-31295][]

修改nova.conf配置文件

\[root@controller ~\]\#vim /etc/nova/nova.conf

\[DEFAULT\]  
enabled\_apis=osapi\_compute,metadata //2766行  
transport\_url=rabbit://openstack:123456@controller //3156行  
my\_ip=172.16.10.33 //1291行  
use\_neutron=true //1755行  
firewall\_driver=nova.virt.firewall.NoopFirewallDriver //2417行

\[api\_database\]  
connection=mysql+pymysql://nova:123456@controller/nova\_api //3513行

\[database\]  
connection=mysql+pymysql://nova:123456@controller/nova //4588行

\[api\]  
auth\_strategy=keystone //3221行

\[keystone\_authtoken\]  
auth\_uri=http://controller:5000  
auth\_url=http://controller:35357 //6073行  
memcached\_servers=controller:11211 //6124行  
auth\_type=password //6231行  
project\_domain\_name = default  
user\_domain\_name = default  
project\_name = service  
username = nova  
password = 123456

\[vnc\]  
enabled=true //10213行  
server\_listen=my\_ip //10237行 server\_proxyclient\_address=my\_ip //10250行

\[glance\]  
api\_servers=http://controller:9292 //5266行

\[oslo\_concurrency\]  
lock\_path=/var/lib/nova/tmp //7841行

\[placement\]  
os\_region\_name=RegionOne //8740行  
auth\_type=password //8780行  
auth\_url=http://controller:35357/v3 //8786行  
project\_name=service //8801行  
project\_domain\_name=Default //8807行  
username=placement //8827行  
user\_domain\_name=Default //8833行  
password=123456 //8836行

启用placement API访问  
由于软件包错误，必须启用对Placement API的访问，在配置文件末尾添加即可。  
<Directory /usr/bin>  
<IfVersion >= 2.4>  
Require all granted  
</IfVersion>  
<IfVersion < 2.4>  
Order allow,deny  
Allow from all  
</IfVersion>  
</Directory>  
![3629fedc6c7fdfe29b85c2ba4f382b2d-41010][]

重启httpd服务

\[root@controller ~\]\# systemctl restart httpd.service

同步nova-api数据库

\[root@controller ~\]\# su -s /bin/sh -c "nova-manage api\_db sync" nova

注册cell0数据库

\[root@controller ~\]\# su -s /bin/sh -c "nova-manage cell\_v2 map\_cell0" nova

创建cell1 cell

\[root@controller ~\]\# su -s /bin/sh -c "nova-manage cell\_v2 create\_cell --name=cell1 --verbose" nova

同步nova数据库

\[root@controller ~\]\# su -s /bin/sh -c "nova-manage db sync" nova

验证数据库是否注册正确

\[root@controller ~\]\# nova-manage cell\_v2 list\_cells  
![a7056fdb3898719668e87812b2690e8e-292507][]

启动并将服务添加为开机自启  
systemctl enable openstack-nova-api.service  
systemctl enable openstack-nova-consoleauth.service  
systemctl enable openstack-nova-scheduler.service  
systemctl enable openstack-nova-conductor.service  
systemctl enable openstack-nova-novncproxy.service  
systemctl start openstack-nova-api.service  
systemctl start openstack-nova-consoleauth.service  
systemctl start openstack-nova-scheduler.service  
systemctl start openstack-nova-conductor.service  
systemctl start openstack-nova-novncproxy.service

#### 15. 安装和配置compute节点 ####

安装软件包  
yum -y install openstack-nova-compute

修改nova.conf配置文件

vim /etc/nova/nova.conf

\[DEFAULT\]  
my\_ip = 172.16.10.35 //1291，输入compute节点IP  
use\_neutron=true //1755  
firewall\_driver=nova.virt.firewall.NoopFirewallDriver //2417  
enabled\_apis = osapi\_compute,metadata //2756  
transport\_url = rabbit://openstack:123456@controller //3156

\[api\]  
auth\_strategy=keystone //3221

\[keystone\_authtoken\]  
auth\_uri = http://172.16.10.33:5000 //6073controller节点IP  
auth\_url = http://controller:35357  
memcached\_servers=controller:11211 //6124  
auth\_type=password //6231  
project\_domain\_name=default  
user\_domain\_name=default  
project\_name=service  
username=nova  
password=123456

\[vnc\]  
enabled=true //10213  
server\_listen=0.0.0.0 //10237  
server\_proxyclient\_address=$my\_ip //10250  
novncproxy\_base\_url=http://controller:6080/vnc\_auto.html //10268

\[glance\]  
api\_servers=http://controller:9292 //5266

\[oslo\_concurrency\]  
lock\_path=/var/lib/nova/tmp //7841

\[placement\]  
os\_region\_name=RegionOne //8740  
auth\_type = password //8780  
auth\_url=http://controller:35357/v3 //8786  
project\_name = service //8801  
project\_domain\_name = Default //8807  
user\_domain\_name = Default //8833  
username = placement //8827  
password = 123456 //8836

启动服务同时添加为开机自启  
systemctl enable libvirtd.service  
systemctl restart libvirtd  
systemctl enable openstack-nova-compute.service  
systemctl start openstack-nova-compute.service

#### 16. 添加compute节点到cell数据库 ####

在controller节点上进行操作  
验证在数据库中的计算节点

\[root@controller ~\]\# source ~/admin-openrc //在重启虚拟机时需重新加载环境变量

\[root@controller ~\]\# openstack compute service list --service nova-compute

发现计算节点

\[root@controller ~\]\# su -s /bin/sh -c "nova-manage cell\_v2 discover\_hosts --verbose" nova  
![196d0517da28cd601dd1a19f7b72635c-10686][]

在controller节点验证计算服务操作

\[root@controller ~\]\# openstack compute service list  
![94aad5d9fa722630ef6a8249ff494fcf-15165][]

列出身份服务中的API端点以验证与身份服务的连接

\[root@controller ~\]\# openstack catalog list  
![27384122199bf176cab5e6dc73b9e9ff-29821][]

检查cells和placement API是否正常

\[root@controller ~\]\# nova-status upgrade check  
![67a15bc0134463f57675dbf5fd65ab07-30902][]

#### 17. Networking服务 ####

安装和配置controller节点neutron网络配置  
创建nuetron数据库并授权  
MariaDB \[(none)\]> CREATE DATABASE neutron;  
MariaDB \[(none)\]> GRANT ALL PRIVILEGES ON neutron.\* TO 'neutron'@'localhost' IDENTIFIED BY '123456';  
MariaDB \[(none)\]> GRANT ALL PRIVILEGES ON neutron.\* TO 'neutron'@'%' IDENTIFIED BY '123456';  
![319aa73ff13f400b8d722d9d3dfed401-27353][]

创建用户

\[root@controller ~\]\# source ~/admin-openrc

\[root@controller ~\]\# openstack user create --domain default --password-prompt neutron  
![76211e40e60deeec4dbf9c9373ee1bda-16179][]

创建网络服务端点

\[root@controller ~\]\# openstack service create --name neutron --description "OpenStack Networking" network  
![4623c260c787ac9ebf6385ac22c2de95-10541][]

创建网络服务端点

\[root@controller ~\]\#openstack endpoint create --region RegionOne network public http://controller:9696  
![869f47f55f69525b0721b55e9f6c8a28-15631][]

\[root@controller ~\]\#openstack endpoint create --region RegionOne network internal http://controller:9696  
![afb19e3e4b56552eefb3110224d0299a-15909][]

\[root@controller ~\]\#openstack endpoint create --region RegionOne network admin http://controller:9696  
![12b00ca5c864a1273c1d4dcc861c1791-15783][]

安装软件包

\[root@controller ~\]\# yum install -y openstack-neutron openstack-neutron-ml2 openstack-neutron-linuxbridge ebtables

修改配置文件

vim /etc/neutron/neutron.conf

\[database\]  
connection = mysql+pymysql://neutron:123456@controller/neutron //729

\[DEFAULT\]  
auth\_strategy = keystone //27  
core\_plugin = ml2 //30  
service\_plugins = //33 不写代表禁用其他插件  
transport\_url = rabbit://openstack:123456@controller //570  
notify\_nova\_on\_port\_status\_changes = true //98  
notify\_nova\_on\_port\_data\_changes = true //102

\[keystone\_authtoken\]  
auth\_uri = http://controller:5000 //847  
auth\_url = http://controller:35357  
memcached\_servers = controller:11211 //898  
auth\_type = password //1005  
project\_domain\_name = default  
user\_domain\_name = default  
project\_name = service  
username = neutron  
password = 123456

\[nova\]  
auth\_url = http://controller:35357 //1085  
auth\_type = password //1089  
project\_domain\_name = default //1127  
user\_domain\_name = default //1156  
region\_name = RegionOne //1069  
project\_name = service //1135  
username = nova //1163  
password = 123456 //1121

\[oslo\_concurrency\]  
lock\_path = /var/lib/neutron/tmp //1179

配置网络二层插件

vim /etc/neutron/plugins/ml2/ml2\_conf.ini

\[ml2\]  
type\_drivers = flat,vlan //136  
tenant\_network\_types = //141 设置空是禁用本地网络  
mechanism\_drivers = linuxbridge //145  
extension\_drivers = port\_security //150

\[ml2\_type\_flat\]  
flat\_networks = provider //186

\[securitygroup\]  
enable\_ipset = true //263

配置Linux网桥

vim /etc/neutron/plugins/ml2/linuxbridge\_agent.ini

\[linux\_bridge\]  
physical\_interface\_mappings = provider:ens33 //157

\[vxlan\]  
enable\_vxlan = false //208

\[securitygroup\]  
enable\_security\_group = true //193  
firewall\_driver = neutron.agent.linux.iptables\_firewall.IptablesFirewallDriver //188

配置DHCP

vim /etc/neutron/dhcp\_agent.ini

interface\_driver = linuxbridge //16  
dhcp\_driver = neutron.agent.linux.dhcp.Dnsmasq //28  
enable\_isolated\_metadata = true //37

配置metadata

vim /etc/neutron/metadata\_agent.ini

\[DEFAULT\]  
nova\_metadata\_host = controller //22  
metadata\_proxy\_shared\_secret = 123456 //34

配置计算服务使用网络服务

vim /etc/nova/nova.conf

\[neutron\]  
url = http://controller:9696 //7534  
auth\_url = http://controller:35357 //7610  
auth\_type = password //7604  
project\_domain\_name = default //7631  
user\_domain\_name = default //7657  
region\_name = RegionOne //7678  
project\_name = service //7625  
username = neutron //7651  
password = 123456 //7660  
service\_metadata\_proxy = true //7573  
metadata\_proxy\_shared\_secret = 123456 //7584

建立服务软连接

\[root@controller ~\]\# ln -s /etc/neutron/plugins/ml2/ml2\_conf.ini /etc/neutron/plugin.ini

同步数据库

\[root@controller ~\]\# su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2\_conf.ini upgrade head" neutron

重启compute API服务

\[root@controller ~\]\# systemctl restart openstack-nova-api.service

启动neutron服务并添加为开机自启

systemctl enable neutron-server.service  
systemctl enable neutron-linuxbridge-agent.service  
systemctl enable neutron-dhcp-agent.service  
systemctl enable neutron-metadata-agent.service  
systemctl start neutron-server.service  
systemctl start neutron-linuxbridge-agent.service  
systemctl start neutron-dhcp-agent.service  
systemctl start neutron-metadata-agent.service  
![538c0bb7f1a64ed8f25e15da81252132-31384][]

#### 18. 配置compute节点网络服务 ####

安装软件包

\[root@compute ~\]\# yum install -y openstack-neutron-linuxbridge ebtables ipset

配置公共组件

vim /etc/neutron/neutron.conf

\[DEFAULT\]  
auth\_strategy = keystone //27  
transport\_url = rabbit://openstack:123456@controller //570

\[oslo\_concurrency\]  
lock\_path = /var/lib/neutron/tmp //1180

配置Linux网桥

vim /etc/neutron/plugins/ml2/linuxbridge\_agent.ini

\[linux\_bridge\]  
physical\_interface\_mappings = provider:ens33 //157

\[vxlan\]  
enable\_vxlan = false //208

\[securitygroup\]  
enable\_security\_group = true //193  
firewall\_driver = neutron.agent.linux.iptables\_firewall.IptablesFirewallDriver //188

配置计算节点网络服务

vim /etc/nova/nova.conf

\[neutron\]  
url = http://controller:9696 //7534  
auth\_url = http://controller:35357 //7610  
auth\_type = password //7640  
project\_domain\_name = default //7631  
user\_domain\_name = default //7657  
region\_name = RegionOne //7678  
project\_name = service //7625  
username = neutron //7651  
password = 123456 //7660

启动服务  
systemctl restart openstack-nova-compute.service  
systemctl enable neutron-linuxbridge-agent.service  
systemctl start neutron-linuxbridge-agent.service

#### 19. 部署Horizon服务 ####

在controller节点安装Horizon服务  
安装软件包

\[root@controller ~\]\# yum install openstack-dashboard -y

修改配置文件

vim /etc/openstack-dashboard/local\_settings

OPENSTACK\_HOST = "controller" //189  
OPENSTACK\_KEYSTONE\_DEFAULT\_ROLE = "admin" //191  
ALLOWED\_HOSTS = \['\*'\] //38  
SESSION\_ENGINE = 'django.contrib.sessions.backends.file' //51

配置memcache会话存储

SESSION\_ENGINE = 'django.contrib.sessions.backends.cache' //50，添加  
CACHES = \{ //注释166-170 去掉注释159-164  
'default': \{  
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',  
'LOCATION': 'controller:11211',  
\}  
\}  
OPENSTACK\_KEYSTONE\_URL = "http://%s:5000/v3" % OPENSTACK\_HOST //开启身份认证API版本v3 190行  
OPENSTACK\_KEYSTONE\_MULTIDOMAIN\_SUPPORT = True //开启domains版本支持 76行

OPENSTACK\_API\_VERSIONS = \{ //配置API版本 65行  
"identity": 3,  
"image": 2,  
"volume": 2,  
\}  
OPENSTACK\_KEYSTONE\_DEFAULT\_DOMAIN = "Default" //98

OPENSTACK\_NEUTRON\_NETWORK = \{ //324

'enable_router': False,
    'enable_quotas': False,
    'enable_distributed_router': False,
    'enable_ha_router': False,
    'enable_lb': False,
    'enable_firewall': False,
    'enable_***': False,
    'enable_fip_topology_check': False,

解决网页无法打开检查

vim /etc/httpd/conf.d/openstack-dashboard.conf

WSGISocketPrefix run/wsgi  
WSGIApplicationGroup %\{GLOBAL\} //添加

重启web服务和会话存储  
systemctl restart httpd.service  
systemctl restart memcached.service

#### 20. 登陆测试 ####

http://192.168.66.197/dashboard

domain： default  
用户名：admin  
密码：123456  
![d2c8592454822eb96360b7efa017d9b3-67896][]

保存到我的笔记

本内容不代表印象笔记立场

[cef432f952d88fd2d528968e97c28abd-32048]: https://app.yinxiang.com/FileSharing.action?hash=1/cef432f952d88fd2d528968e97c28abd-32048
[d2deca82a9f9d29a843a9c049bdb7c99-24493]: https://app.yinxiang.com/FileSharing.action?hash=1/d2deca82a9f9d29a843a9c049bdb7c99-24493
[2358ae6eadc988010252aa81502bd22f-12224]: https://app.yinxiang.com/FileSharing.action?hash=1/2358ae6eadc988010252aa81502bd22f-12224
[43c073fdabc59720a6128f9b433e6371-15695]: https://app.yinxiang.com/FileSharing.action?hash=1/43c073fdabc59720a6128f9b433e6371-15695
[3039b8e48e59479fa22d180b6d8fe231-14037]: https://app.yinxiang.com/FileSharing.action?hash=1/3039b8e48e59479fa22d180b6d8fe231-14037
[5c00d626f153e227fa089a7ebcbaf161-15368]: https://app.yinxiang.com/FileSharing.action?hash=1/5c00d626f153e227fa089a7ebcbaf161-15368
[1fa38b35cf049cfd441034b6057e1e83-594281]: https://app.yinxiang.com/FileSharing.action?hash=1/1fa38b35cf049cfd441034b6057e1e83-594281
[8158d7641b4d48f51a585fc16e40dd94-20499]: https://app.yinxiang.com/FileSharing.action?hash=1/8158d7641b4d48f51a585fc16e40dd94-20499
[d4aa986409ae92459ad1d48e71473247-4495]: https://app.yinxiang.com/FileSharing.action?hash=1/d4aa986409ae92459ad1d48e71473247-4495
[b6aa46669216208e43b5e39af576efb5-4911]: https://app.yinxiang.com/FileSharing.action?hash=1/b6aa46669216208e43b5e39af576efb5-4911
[866ea74c7d71819d363c69e493eb15ab-10038]: https://app.yinxiang.com/FileSharing.action?hash=1/866ea74c7d71819d363c69e493eb15ab-10038
[2090ed8ff4fef442c94e2843581404f4-24748]: https://app.yinxiang.com/FileSharing.action?hash=1/2090ed8ff4fef442c94e2843581404f4-24748
[4da716f169f1a1c53c64d7d7e94485f6-29629]: https://app.yinxiang.com/FileSharing.action?hash=1/4da716f169f1a1c53c64d7d7e94485f6-29629
[a2895ef4406abab6aa1194614dcd3411-4236]: https://app.yinxiang.com/FileSharing.action?hash=1/a2895ef4406abab6aa1194614dcd3411-4236
[e41474cd4a25eb299080b871a08f1f24-36097]: https://app.yinxiang.com/FileSharing.action?hash=1/e41474cd4a25eb299080b871a08f1f24-36097
[55756e73831882a5c5a60e01b7422570-7509]: https://app.yinxiang.com/FileSharing.action?hash=1/55756e73831882a5c5a60e01b7422570-7509
[d9fce64a0f37fd319e8e8cf5ee3b879d-5890]: https://app.yinxiang.com/FileSharing.action?hash=1/d9fce64a0f37fd319e8e8cf5ee3b879d-5890
[192e1b749abdc0a9e099a48760452cdf-16503]: https://app.yinxiang.com/FileSharing.action?hash=1/192e1b749abdc0a9e099a48760452cdf-16503
[c810e26d0719a40f4c4b89bf2539f84a-9462]: https://app.yinxiang.com/FileSharing.action?hash=1/c810e26d0719a40f4c4b89bf2539f84a-9462
[7350d5d58fc92d4882c2011d634cc176-13404]: https://app.yinxiang.com/FileSharing.action?hash=1/7350d5d58fc92d4882c2011d634cc176-13404
[e139da66ff29c3b1447c939122fea934-13354]: https://app.yinxiang.com/FileSharing.action?hash=1/e139da66ff29c3b1447c939122fea934-13354
[e325384ec6fbb0da02a260d02f501890-22698]: https://app.yinxiang.com/FileSharing.action?hash=1/e325384ec6fbb0da02a260d02f501890-22698
[4ba0c70ffe9c4d8b3fed0c1bf55baa73-6550]: https://app.yinxiang.com/FileSharing.action?hash=1/4ba0c70ffe9c4d8b3fed0c1bf55baa73-6550
[f9b1d83ed14783aaf2433f8f1438b7b4-30437]: https://app.yinxiang.com/FileSharing.action?hash=1/f9b1d83ed14783aaf2433f8f1438b7b4-30437
[7f414371ccf6bb5888ba493af0b3feb4-26869]: https://app.yinxiang.com/FileSharing.action?hash=1/7f414371ccf6bb5888ba493af0b3feb4-26869
[397b05329f4e204553aa8819b50881ab-13988]: https://app.yinxiang.com/FileSharing.action?hash=1/397b05329f4e204553aa8819b50881ab-13988
[5b663299c05c17862aadaa44e4c07d9e-14763]: https://app.yinxiang.com/FileSharing.action?hash=1/5b663299c05c17862aadaa44e4c07d9e-14763
[9c26cbcb54160162a7e575969abd1650-19792]: https://app.yinxiang.com/FileSharing.action?hash=1/9c26cbcb54160162a7e575969abd1650-19792
[9e4ffacb5ddb7e7c5dc6ddecf9d93ee8-40919]: https://app.yinxiang.com/FileSharing.action?hash=1/9e4ffacb5ddb7e7c5dc6ddecf9d93ee8-40919
[ecc6a39825f436d4c9d8db81e5d7606b-15738]: https://app.yinxiang.com/FileSharing.action?hash=1/ecc6a39825f436d4c9d8db81e5d7606b-15738
[53a4e796d2e413ff648d9b091cf584f6-15254]: https://app.yinxiang.com/FileSharing.action?hash=1/53a4e796d2e413ff648d9b091cf584f6-15254
[f56963d2f1cd685dd82721ec910a6068-10785]: https://app.yinxiang.com/FileSharing.action?hash=1/f56963d2f1cd685dd82721ec910a6068-10785
[b9095f5fc5d64250b1e5134510f337af-28511]: https://app.yinxiang.com/FileSharing.action?hash=1/b9095f5fc5d64250b1e5134510f337af-28511
[b4449a8339f40764d8dec7323c56be7b-15172]: https://app.yinxiang.com/FileSharing.action?hash=1/b4449a8339f40764d8dec7323c56be7b-15172
[76b2fcb980b26e00b6d55a9937f5c9a9-34487]: https://app.yinxiang.com/FileSharing.action?hash=1/76b2fcb980b26e00b6d55a9937f5c9a9-34487
[fb049f15ad32585e5d6d3f3716112715-25728]: https://app.yinxiang.com/FileSharing.action?hash=1/fb049f15ad32585e5d6d3f3716112715-25728
[e6a5acdfc1d7d3ffaf881f5198f4e653-20310]: https://app.yinxiang.com/FileSharing.action?hash=1/e6a5acdfc1d7d3ffaf881f5198f4e653-20310
[d1b726a3ee9a8ba4d25833387bbb8441-19312]: https://app.yinxiang.com/FileSharing.action?hash=1/d1b726a3ee9a8ba4d25833387bbb8441-19312
[085ef441570f9e44adf99bf4705580c9-24327]: https://app.yinxiang.com/FileSharing.action?hash=1/085ef441570f9e44adf99bf4705580c9-24327
[a758a71844d6c2362a9cafbbc22c58c1-28751]: https://app.yinxiang.com/FileSharing.action?hash=1/a758a71844d6c2362a9cafbbc22c58c1-28751
[8ce26b3e90c09126c1197a257897776a-6371]: https://app.yinxiang.com/FileSharing.action?hash=1/8ce26b3e90c09126c1197a257897776a-6371
[a5ab992c2b3d7bcfc48b208957c44263-58557]: https://app.yinxiang.com/FileSharing.action?hash=1/a5ab992c2b3d7bcfc48b208957c44263-58557
[c7281e0beee5c12c80feeb095564987b-17053]: https://app.yinxiang.com/FileSharing.action?hash=1/c7281e0beee5c12c80feeb095564987b-17053
[101fef3890957851eabe20e63aee3091-10391]: https://app.yinxiang.com/FileSharing.action?hash=1/101fef3890957851eabe20e63aee3091-10391
[55bd4f9cee6524dced4f9bcaf0be2385-42275]: https://app.yinxiang.com/FileSharing.action?hash=1/55bd4f9cee6524dced4f9bcaf0be2385-42275
[f8eae57c58d36c1a6e1cb4e5e0dbe8b1-15222]: https://app.yinxiang.com/FileSharing.action?hash=1/f8eae57c58d36c1a6e1cb4e5e0dbe8b1-15222
[60396ed4aa449f1b60d8991e89625c28-10530]: https://app.yinxiang.com/FileSharing.action?hash=1/60396ed4aa449f1b60d8991e89625c28-10530
[c5b910ccda8664a13f872d8c0b1af8b0-43992]: https://app.yinxiang.com/FileSharing.action?hash=1/c5b910ccda8664a13f872d8c0b1af8b0-43992
[cbe6fc31c2b5b77b149ba73cf9c91aee-31295]: https://app.yinxiang.com/FileSharing.action?hash=1/cbe6fc31c2b5b77b149ba73cf9c91aee-31295
[3629fedc6c7fdfe29b85c2ba4f382b2d-41010]: https://app.yinxiang.com/FileSharing.action?hash=1/3629fedc6c7fdfe29b85c2ba4f382b2d-41010
[a7056fdb3898719668e87812b2690e8e-292507]: https://app.yinxiang.com/FileSharing.action?hash=1/a7056fdb3898719668e87812b2690e8e-292507
[196d0517da28cd601dd1a19f7b72635c-10686]: https://app.yinxiang.com/FileSharing.action?hash=1/196d0517da28cd601dd1a19f7b72635c-10686
[94aad5d9fa722630ef6a8249ff494fcf-15165]: https://app.yinxiang.com/FileSharing.action?hash=1/94aad5d9fa722630ef6a8249ff494fcf-15165
[27384122199bf176cab5e6dc73b9e9ff-29821]: https://app.yinxiang.com/FileSharing.action?hash=1/27384122199bf176cab5e6dc73b9e9ff-29821
[67a15bc0134463f57675dbf5fd65ab07-30902]: https://app.yinxiang.com/FileSharing.action?hash=1/67a15bc0134463f57675dbf5fd65ab07-30902
[319aa73ff13f400b8d722d9d3dfed401-27353]: https://app.yinxiang.com/FileSharing.action?hash=1/319aa73ff13f400b8d722d9d3dfed401-27353
[76211e40e60deeec4dbf9c9373ee1bda-16179]: https://app.yinxiang.com/FileSharing.action?hash=1/76211e40e60deeec4dbf9c9373ee1bda-16179
[4623c260c787ac9ebf6385ac22c2de95-10541]: https://app.yinxiang.com/FileSharing.action?hash=1/4623c260c787ac9ebf6385ac22c2de95-10541
[869f47f55f69525b0721b55e9f6c8a28-15631]: https://app.yinxiang.com/FileSharing.action?hash=1/869f47f55f69525b0721b55e9f6c8a28-15631
[afb19e3e4b56552eefb3110224d0299a-15909]: https://app.yinxiang.com/FileSharing.action?hash=1/afb19e3e4b56552eefb3110224d0299a-15909
[12b00ca5c864a1273c1d4dcc861c1791-15783]: https://app.yinxiang.com/FileSharing.action?hash=1/12b00ca5c864a1273c1d4dcc861c1791-15783
[538c0bb7f1a64ed8f25e15da81252132-31384]: https://app.yinxiang.com/FileSharing.action?hash=1/538c0bb7f1a64ed8f25e15da81252132-31384
[d2c8592454822eb96360b7efa017d9b3-67896]: https://app.yinxiang.com/FileSharing.action?hash=1/d2c8592454822eb96360b7efa017d9b3-67896