Manage Ceph Services on Proxmox VE Nodes

深藏阁楼爱情的钟 2024-04-19 09:09 141阅读 0赞

**Manage Ceph Services on Proxmox VE Nodes**

**Proxmox Server Solutions Gmbh**  
<[support@proxmox.com][support_proxmox.com]>  
version 6.0, Mon 15 Jul 2019 09:45:00 PM CEST

**Table of Contents**

[Precondition][]

[Initial Ceph installation & configuration][Initial Ceph installation _ configuration]

[Installation of Ceph Packages][]

[Creating initial Ceph configuration][]

[Creating Ceph Monitors][]

[Creating Ceph Manager][]

[Creating Ceph OSDs][]

[Ceph Bluestore][]

[Ceph Filestore][]

[Creating Ceph Pools][]

[Ceph CRUSH & device classes][Ceph CRUSH _ device classes]

[Ceph Client][]

[CephFS][]

[Metadata Server (MDS)][Metadata Server _MDS]

[Create a CephFS][]

[Destroy CephFS][]

[Ceph monitoring and troubleshooting][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70][]

Proxmox VE unifies your compute and storage systems, i.e. you can use the same physical nodes within a cluster for both computing (processing VMs and containers) and replicated storage. The traditional silos of compute and storage resources can be wrapped up into a single hyper-converged appliance. Separate storage networks (SANs) and connections via network attached storages (NAS) disappear. With the integration of Ceph, an open source software-defined storage platform, Proxmox VE has the ability to run and manage Ceph storage directly on the hypervisor nodes.

Ceph is a distributed object store and file system designed to provide excellent performance, reliability and scalability.

**Some advantages of Ceph on Proxmox VE are:**

*  Easy setup and management with CLI and GUI support
 *  Thin provisioning
 *  Snapshots support
 *  Self healing
 *  Scalable to the exabyte level
 *  Setup pools with different performance and redundancy characteristics
 *  Data is replicated, making it fault tolerant
 *  Runs on economical commodity hardware
 *  No need for hardware RAID controllers
 *  Open source

For small to mid sized deployments, it is possible to install a Ceph server for RADOS Block Devices (RBD) directly on your Proxmox VE cluster nodes, see [Ceph RADOS Block Devices (RBD)][Ceph RADOS Block Devices _RBD]. Recent hardware has plenty of CPU power and RAM, so running storage services and VMs on the same node is possible.

To simplify management, we provide *pveceph* \- a tool to install and manage [Ceph][] services on Proxmox VE nodes.

**Ceph consists of a couple of Daemons** **\[[1][]\]****, for use as a RBD storage:**

*  Ceph Monitor (ceph-mon)
 *  Ceph Manager (ceph-mgr)
 *  Ceph OSD (ceph-osd; Object Storage Daemon)

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">We highly recommend to get familiar with Ceph’s architecture&nbsp;[<a href="https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_2" rel="nofollow"><span style="color:#FF00FF;">2</span></a>]&nbsp;and vocabulary&nbsp;[<a href="https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_3" rel="nofollow"><span style="color:#FF00FF;">3</span></a>].</p> </td> 
  </tr> 
 </tbody> 
</table>

**Precondition**

To build a hyper-converged Proxmox + Ceph Cluster there should be at least three (preferably) identical servers for the setup.

Check also the recommendations from [Ceph’s website][Ceph_s website].

**CPU**

Higher CPU core frequency reduce latency and should be preferred. As a simple rule of thumb, you should assign a CPU core (or thread) to each Ceph service to provide enough resources for stable and durable Ceph performance.

**Memory**

Especially in a hyper-converged setup, the memory consumption needs to be carefully monitored. In addition to the intended workload from virtual machines and container, Ceph needs enough memory available to provide good and stable performance. As a rule of thumb, for roughly 1 TiB of data, 1 GiB of memory will be used by an OSD. OSD caching will use additional memory.

**Network**

We recommend a network bandwidth of at least 10 GbE or more, which is used exclusively for Ceph. A meshed network setup \[[4][]\] is also an option if there are no 10 GbE switches available.

The volume of traffic, especially during recovery, will interfere with other services on the same network and may even break the Proxmox VE cluster stack.

Further, estimate your bandwidth needs. While one HDD might not saturate a 1 Gb link, multiple HDD OSDs per node can, and modern NVMe SSDs will even saturate 10 Gbps of bandwidth quickly. Deploying a network capable of even more bandwith will ensure that it isn’t your bottleneck and won’t be anytime soon, 25, 40 or even 100 GBps are possible.

**Disks**

When planning the size of your Ceph cluster, it is important to take the recovery time into consideration. Especially with small clusters, the recovery might take long. It is recommended that you use SSDs instead of HDDs in small setups to reduce recovery time, minimizing the likelihood of a subsequent failure event during recovery.

In general SSDs will provide more IOPs than spinning disks. This fact and the higher cost may make a [class based][Ceph CRUSH _ device classes] separation of pools appealing. Another possibility to speedup OSDs is to use a faster disk as journal or DB/**W**rite-**A**head-**L**og device, see [creating Ceph OSDs][Creating Ceph OSDs]. If a faster disk is used for multiple OSDs, a proper balance between OSD and WAL / DB (or journal) disk must be selected, otherwise the faster disk becomes the bottleneck for all linked OSDs.

Aside from the disk type, Ceph best performs with an even sized and distributed amount of disks per node. For example, 4 x 500 GB disks with in each node is better than a mixed setup with a single 1 TB and three 250 GB disk.

One also need to balance OSD count and single OSD capacity. More capacity allows to increase storage density, but it also means that a single OSD failure forces ceph to recover more data at once.

**Avoid RAID**

As Ceph handles data object redundancy and multiple parallel writes to disks (OSDs) on its own, using a RAID controller normally doesn’t improve performance or availability. On the contrary, Ceph is designed to handle whole disks on it’s own, without any abstraction in between. RAID controller are not designed for the Ceph use case and may complicate things and sometimes even reduce performance, as their write and caching algorithms may interfere with the ones from Ceph.

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">Avoid RAID controller, use host bus adapter (HBA) instead.</p> </td> 
  </tr> 
 </tbody> 
</table>

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">Above recommendations should be seen as a rough guidance for choosing hardware. Therefore, it is still essential to adapt it to your specific needs, test your setup and monitor health and performance continuously.</p> </td> 
  </tr> 
 </tbody> 
</table>

**Initial Ceph installation & configuration**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 1][]

With Proxmox VE you have the benefit of an easy to use installation wizard for Ceph. Click on one of your cluster nodes and navigate to the Ceph section in the menu tree. If Ceph is not already installed you will be offered to do so now.

The wizard is divided into different sections, where each needs to be finished successfully in order to use Ceph. After starting the installation the wizard will download and install all required packages from Proxmox VE’s ceph repository.

After finishing the first step, you will need to create a configuration. This step is only needed once per cluster, as this configuration is distributed automatically to all remaining cluster members through Proxmox VE’s clustered [configuration file system (pmxcfs)][configuration file system _pmxcfs].

The configuration step includes the following settings:

*  **Public Network:** You should setup a dedicated network for Ceph, this setting is required. Separating your Ceph traffic is highly recommended, because it could lead to troubles with other latency dependent services, e.g., cluster communication may decrease Ceph’s performance, if not done.
 *  **Cluster Network:** As an optional step you can go even further and separate the [OSD][Creating Ceph OSDs] replication & heartbeat traffic as well. This will relieve the public network and could lead to significant performance improvements especially in big clusters.

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 2][]

You have two more options which are considered advanced and therefore should only changed if you are an expert.

*  **Number of replicas**: Defines the how often a object is replicated
 *  **Minimum replicas**: Defines the minimum number of required replicas for I/O to be marked as complete.

Additionally you need to choose your first monitor node, this is required.

That’s it, you should see a success page as the last step with further instructions on how to go on. You are now prepared to start using Ceph, even though you will need to create additional [monitors][Creating Ceph Monitors], create some [OSDs][Creating Ceph OSDs] and at least one [pool][Creating Ceph Pools].

The rest of this chapter will guide you on how to get the most out of your Proxmox VE based Ceph setup, this will include aforementioned and more like [CephFS][] which is a very handy addition to your new Ceph cluster.

**Installation of Ceph Packages**

Use Proxmox VE Ceph installation wizard (recommended) or run the following command on each node:

pveceph install

This sets up an apt package repository in /etc/apt/sources.list.d/ceph.list and installs the required software.

**Creating initial Ceph configuration**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 3][]

Use the Proxmox VE Ceph installation wizard (recommended) or run the following command on one node:

pveceph init --network 10.10.10.0/24

This creates an initial configuration at /etc/pve/ceph.conf with a dedicated network for ceph. That file is automatically distributed to all Proxmox VE nodes by using [pmxcfs][configuration file system _pmxcfs]. The command also creates a symbolic link from /etc/ceph/ceph.conf pointing to that file. So you can simply run Ceph commands without the need to specify a configuration file.

**Creating Ceph Monitors**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 4][]

The Ceph Monitor (MON) \[[5][]\] maintains a master copy of the cluster map. For high availability you need to have at least 3 monitors. One monitor will already be installed if you used the installation wizard. You wont need more than 3 monitors as long as your cluster is small to midsize, only really large clusters will need more than that.

On each node where you want to place a monitor (three monitors are recommended), create it by using the *Ceph* *→* *Monitor* tab in the GUI or run.

pveceph createmon

This will also install the needed Ceph Manager (*ceph-mgr*) by default. If you do not want to install a manager, specify the *-exclude-manager* option.

**Creating Ceph Manager**

The Manager daemon runs alongside the monitors, providing an interface for monitoring the cluster. Since the Ceph luminous release the ceph-mgr \[[6][]\] daemon is required. During monitor installation the ceph manager will be installed as well.

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">It is recommended to install the Ceph Manager on the monitor nodes. For high availability install more then one manager.</p> </td> 
  </tr> 
 </tbody> 
</table>

pveceph createmgr

**Creating Ceph OSDs**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 5][]

via GUI or via CLI as follows:

pveceph createosd /dev/sd\[X\]

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">We recommend a Ceph cluster size, starting with 12 OSDs, distributed evenly among your, at least three nodes (4 OSDs on each node).</p> </td> 
  </tr> 
 </tbody> 
</table>

If the disk was used before (eg. ZFS/RAID/OSD), to remove partition table, boot sector and any OSD leftover the following command should be sufficient.

ceph-volume lvm zap /dev/sd\[X\] \--destroy

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">The above command will destroy data on the disk!</p> </td> 
  </tr> 
 </tbody> 
</table>

**Ceph Bluestore**

Starting with the Ceph Kraken release, a new Ceph OSD storage type was introduced, the so called Bluestore \[[7][]\]. This is the default when creating OSDs since Ceph Luminous.

pveceph createosd /dev/sd\[X\]

**Block.db and block.wal**

If you want to use a separate DB/WAL device for your OSDs, you can specify it through the *-db\_dev* and *-wal\_dev* options. The WAL is placed with the DB, if not specified separately.

pveceph createosd /dev/sd\[X\] \-db\_dev /dev/sd\[Y\] \-wal\_dev /dev/sd\[Z\]

You can directly choose the size for those with the *-db\_size* and *-wal\_size* paremeters respectively. If they are not given the following values (in order) will be used:

*  bluestore\_block\_\{db,wal\}\_size from ceph configuration…
    
     *  … database, section *osd*
     *  … database, section *global*
     *  … file, section *osd*
     *  … file, section *global*
 *  10% (DB)/1% (WAL) of OSD size

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">The DB stores BlueStore’s internal metadata and the WAL is BlueStore’s internal journal or write-ahead log. It is recommended to use a fast SSD or NVRAM for better performance.</p> </td> 
  </tr> 
 </tbody> 
</table>

**Ceph Filestore**

Before Ceph Luminous, Filestore was used as default storage type for Ceph OSDs. Starting with Ceph Nautilus, Proxmox VE does not support creating such OSDs with *pveceph* anymore. If you still want to create filestore OSDs, use *ceph-volume* directly.

ceph-volume lvm create --filestore --data /dev/sd\[X\] \--journal /dev/sd\[Y\]

**Creating Ceph Pools**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 6][]

A pool is a logical group for storing objects. It holds **P**lacement **G**roups (PG, pg\_num), a collection of objects.

When no options are given, we set a default of **128 PGs**, a **size of 3 replicas** and a **min\_size of 2 replicas** for serving objects in a degraded state.

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">The default number of PGs works for 2-5 disks. Ceph throws a&nbsp;<em><span style="color:#000080;">HEALTH_WARNING</span></em>&nbsp;if you have too few or too many PGs in your cluster.</p> </td> 
  </tr> 
 </tbody> 
</table>

It is advised to calculate the PG number depending on your setup, you can find the formula and the PG calculator \[[8][]\] online. While PGs can be increased later on, they can never be decreased.

You can create pools through command line or on the GUI on each PVE host under **Ceph** **→** **Pools**.

pveceph createpool <name>

If you would like to automatically get also a storage definition for your pool, active the checkbox "Add storages" on the GUI or use the command line option *--add\_storages* on pool creation.

Further information on Ceph pool handling can be found in the Ceph pool operation \[[9][]\] manual.

**Ceph CRUSH & device classes**

The foundation of Ceph is its algorithm, **C**ontrolled **R**eplication **U**nder **S**calable **H**ashing (CRUSH \[[10][]\]).

CRUSH calculates where to store to and retrieve data from, this has the advantage that no central index service is needed. CRUSH works with a map of OSDs, buckets (device locations) and rulesets (data replication) for pools.

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">Further information can be found in the Ceph documentation, under the section CRUSH map&nbsp;[<a href="https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_11" rel="nofollow"><span style="color:#FF00FF;">11</span></a>].</p> </td> 
  </tr> 
 </tbody> 
</table>

This map can be altered to reflect different replication hierarchies. The object replicas can be separated (eg. failure domains), while maintaining the desired distribution.

A common use case is to use different classes of disks for different Ceph pools. For this reason, Ceph introduced the device classes with luminous, to accommodate the need for easy ruleset generation.

The device classes can be seen in the *ceph osd tree* output. These classes represent their own root bucket, which can be seen with the below command.

ceph osd crush tree --show-shadow

Example output form the above command:

ID CLASS WEIGHT TYPE NAME

\-16 nvme 2.18307 root default~nvme

\-13 nvme 0.72769 host sumi1~nvme

12 nvme 0.72769 osd.12

\-14 nvme 0.72769 host sumi2~nvme

13 nvme 0.72769 osd.13

\-15 nvme 0.72769 host sumi3~nvme

14 nvme 0.72769 osd.14

\-1 7.70544 root default

\-3 2.56848 host sumi1

12 nvme 0.72769 osd.12

\-5 2.56848 host sumi2

13 nvme 0.72769 osd.13

\-7 2.56848 host sumi3

14 nvme 0.72769 osd.14

To let a pool distribute its objects only on a specific device class, you need to create a ruleset with the specific class first.

ceph osd crush rule create-replicated <rule-name> <root> <failure-domain> <class>

<table> 
 <tbody> 
  <tr> 
   <td style="border-color:#527bbd;vertical-align:top;"> <p style="margin-left:0cm;">&lt;rule-name&gt;</p> </td> 
   <td style="border-color:#527bbd;vertical-align:top;"> <p style="margin-left:0cm;">name of the rule, to connect with a pool (seen in GUI &amp; CLI)</p> </td> 
  </tr> 
  <tr> 
   <td style="border-color:#527bbd;vertical-align:top;"> <p style="margin-left:0cm;">&lt;root&gt;</p> </td> 
   <td style="vertical-align:top;"> <p style="margin-left:0cm;">which crush root it should belong to (default ceph root "default")</p> </td> 
  </tr> 
  <tr> 
   <td style="border-color:#527bbd;vertical-align:top;"> <p style="margin-left:0cm;">&lt;failure-domain&gt;</p> </td> 
   <td style="vertical-align:top;"> <p style="margin-left:0cm;">at which failure-domain the objects should be distributed (usually host)</p> </td> 
  </tr> 
  <tr> 
   <td style="border-color:#527bbd;vertical-align:top;"> <p style="margin-left:0cm;">&lt;class&gt;</p> </td> 
   <td style="vertical-align:top;"> <p style="margin-left:0cm;">what type of OSD backing store to use (eg. nvme, ssd, hdd)</p> </td> 
  </tr> 
 </tbody> 
</table>

Once the rule is in the CRUSH map, you can tell a pool to use the ruleset.

ceph osd pool **set** <pool-name> crush\_rule <rule-name>

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">If the pool already contains objects, all of these have to be moved accordingly. Depending on your setup this may introduce a big performance hit on your cluster. As an alternative, you can create a new pool and move disks separately.</p> </td> 
  </tr> 
 </tbody> 
</table>

**Ceph Client**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 7][]

You can then configure Proxmox VE to use such pools to store VM or Container images. Simply use the GUI too add a new RBD storage (see section [Ceph RADOS Block Devices (RBD)][Ceph RADOS Block Devices _RBD]).

You also need to copy the keyring to a predefined location for a external Ceph cluster. If Ceph is installed on the Proxmox nodes itself, then this will be done automatically.

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">The file name needs to be&nbsp;<span style="color:#000080;">&lt;storage_id&gt; + `.keyring</span>&nbsp;-&nbsp;<span style="color:#000080;">&lt;storage_id&gt;</span>&nbsp;is the expression after&nbsp;<em><span style="color:#000080;">rbd:</span></em>&nbsp;in&nbsp;<span style="color:#000080;">/etc/pve/storage.cfg</span>&nbsp;which is&nbsp;<span style="color:#000080;">my-ceph-storage</span>&nbsp;in the following example:</p> </td> 
  </tr> 
 </tbody> 
</table>

mkdir /etc/pve/priv/ceph

cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/my-ceph-storage.keyring

**CephFS**

Ceph provides also a filesystem running on top of the same object storage as RADOS block devices do. A **M**eta**d**ata **S**erver (MDS) is used to map the RADOS backed objects to files and directories, allowing to provide a POSIX-compliant replicated filesystem. This allows one to have a clustered highly available shared filesystem in an easy way if ceph is already used. Its Metadata Servers guarantee that files get balanced out over the whole Ceph cluster, this way even high load will not overload a single host, which can be an issue with traditional shared filesystem approaches, like NFS, for example.

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 8][]

Proxmox VE supports both, using an existing [CephFS as storage][] to save backups, ISO files or container templates and creating a hyper-converged CephFS itself.

**Metadata Server (MDS)**

CephFS needs at least one Metadata Server to be configured and running to be able to work. One can simply create one through the Proxmox VE web GUI’s Node -> CephFS panel or on the command line with:

pveceph mds create

Multiple metadata servers can be created in a cluster. But with the default settings only one can be active at any time. If an MDS, or its node, becomes unresponsive (or crashes), another standby MDS will get promoted to active. One can speed up the hand-over between the active and a standby MDS up by using the *hotstandby* parameter option on create, or if you have already created it you may set/add:

mds standby replay = true

in the ceph.conf respective MDS section. With this enabled, this specific MDS will always poll the active one, so that it can take over faster as it is in a warm state. But naturally, the active polling will cause some additional performance impact on your system and active MDS.

**Multiple Active MDS**

Since Luminous (12.2.x) you can also have multiple active metadata servers running, but this is normally only useful for a high count on parallel clients, as else the MDS seldom is the bottleneck. If you want to set this up please refer to the ceph documentation. \[[12][]\]

**Create a CephFS**

With Proxmox VE’s CephFS integration into you can create a CephFS easily over the Web GUI, the CLI or an external API interface. Some prerequisites are required for this to work:

**Prerequisites for a successful CephFS setup:**

*  [Install Ceph packages][Installation of Ceph Packages], if this was already done some time ago you might want to rerun it on an up to date system to ensure that also all CephFS related packages get installed.
 *  [Setup Monitors][Creating Ceph Monitors]
 *  [Setup your OSDs][Creating Ceph Monitors]
 *  [Setup at least one MDS][Metadata Server _MDS]

After this got all checked and done you can simply create a CephFS through either the Web GUI’s Node -> CephFS panel or the command line tool pveceph, for example with:

pveceph fs create --pg\_num 128 --add-storage

This creates a CephFS named ‘'cephfs'' using a pool for its data named \`'cephfs\_data'' with \`128\` placement groups and a pool for its metadata named 'cephfs\_metadata'' with one quarter of the data pools placement groups (\`32). Check the [Proxmox VE managed Ceph pool chapter][Creating Ceph Pools] or visit the Ceph documentation for more information regarding a fitting placement group number (pg\_num) for your setup \[[13][]\]. Additionally, the \`*--add-storage*’ parameter will add the CephFS to the Proxmox VE storage configuration after it was created successfully.

**Destroy CephFS**

<table> 
 <tbody> 
  <tr> 
   <td style="vertical-align:top;">&nbsp;</td> 
   <td> <p style="margin-left:0cm;">Destroying a CephFS will render all its data unusable, this cannot be undone!</p> </td> 
  </tr> 
 </tbody> 
</table>

If you really want to destroy an existing CephFS you first need to stop, or destroy, all metadata server (M̀DS). You can destroy them either over the Web GUI or the command line interface, with:

pveceph mds destroy NAME

on each Proxmox VE node hosting a MDS daemon.

Then, you can remove (destroy) CephFS by issuing a:

ceph fs rm NAME --yes-i-really-mean-it

on a single node hosting Ceph. After this you may want to remove the created data and metadata pools, this can be done either over the Web GUI or the CLI with:

pveceph pool destroy NAME

**Ceph monitoring and troubleshooting**

A good start is to continuosly monitor the ceph health from the start of initial deployment. Either through the ceph tools itself, but also by accessing the status through the Proxmox VE [API][].

The following ceph commands below can be used to see if the cluster is healthy (*HEALTH\_OK*), if there are warnings (*HEALTH\_WARN*), or even errors (*HEALTH\_ERR*). If the cluster is in an unhealthy state the status commands below will also give you an overview on the current events and actions take.

\# single time output

pve\# ceph -s

\# continuously output status changes (press CTRL+C to stop)

pve\# ceph -w

To get a more detailed view, every ceph service has a log file under /var/log/ceph/ and if there is not enough detail, the log level can be adjusted \[[14][]\].

You can find more information about troubleshooting \[[15][]\] a Ceph cluster on its website.

--------------------

[1][1 1]. Ceph intro [http://docs.ceph.com/docs/luminous/start/intro/][http_docs.ceph.com_docs_luminous_start_intro]

[2][]. Ceph architecture [http://docs.ceph.com/docs/luminous/architecture/][http_docs.ceph.com_docs_luminous_architecture]

[3][]. Ceph glossary [http://docs.ceph.com/docs/luminous/glossary][http_docs.ceph.com_docs_luminous_glossary]

[4][4 1]. Full Mesh Network for Ceph [https://pve.proxmox.com/wiki/Full\_Mesh\_Network\_for\_Ceph\_Server][https_pve.proxmox.com_wiki_Full_Mesh_Network_for_Ceph_Server]

[5][5 1]. Ceph Monitor [http://docs.ceph.com/docs/luminous/start/intro/][http_docs.ceph.com_docs_luminous_start_intro]

[6][6 1]. Ceph Manager [http://docs.ceph.com/docs/luminous/mgr/][http_docs.ceph.com_docs_luminous_mgr]

[7][7 1]. Ceph Bluestore [http://ceph.com/community/new-luminous-bluestore/][http_ceph.com_community_new-luminous-bluestore]

[8][8 1]. PG calculator [http://ceph.com/pgcalc/][http_ceph.com_pgcalc]

[9][9 1]. Ceph pool operation [http://docs.ceph.com/docs/luminous/rados/operations/pools/][http_docs.ceph.com_docs_luminous_rados_operations_pools]

[10][10 1]. CRUSH [https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf][https_ceph.com_wp-content_uploads_2016_08_weil-crush-sc06.pdf]

[11][]. CRUSH map [http://docs.ceph.com/docs/luminous/rados/operations/crush-map/][http_docs.ceph.com_docs_luminous_rados_operations_crush-map]

[12][12 1]. Configuring multiple active MDS daemons [http://docs.ceph.com/docs/luminous/cephfs/multimds/][http_docs.ceph.com_docs_luminous_cephfs_multimds]

[13][13 1]. Ceph Placement Groups [http://docs.ceph.com/docs/luminous/rados/operations/placement-groups/][http_docs.ceph.com_docs_luminous_rados_operations_placement-groups]

[14][14 1]. Ceph log and debugging [http://docs.ceph.com/docs/luminous/rados/troubleshooting/log-and-debug/][http_docs.ceph.com_docs_luminous_rados_troubleshooting_log-and-debug]

[15][15 1]. Ceph troubleshooting [http://docs.ceph.com/docs/luminous/rados/troubleshooting/][http_docs.ceph.com_docs_luminous_rados_troubleshooting]

Version 6.0  
Last updated Mon 15 Jul 2019 09:45:00 PM CEST

[support_proxmox.com]: mailto:support@proxmox.com
[Precondition]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_precondition
[Initial Ceph installation _ configuration]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_install_wizard
[Installation of Ceph Packages]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_install
[Creating initial Ceph configuration]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_creating_initial_ceph_configuration
[Creating Ceph Monitors]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_monitors
[Creating Ceph Manager]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_manager
[Creating Ceph OSDs]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_osds
[Ceph Bluestore]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_ceph_bluestore
[Ceph Filestore]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_ceph_filestore
[Creating Ceph Pools]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_pools
[Ceph CRUSH _ device classes]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_device_classes
[Ceph Client]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_ceph_client
[CephFS]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pveceph_fs
[Metadata Server _MDS]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pveceph_fs_mds
[Create a CephFS]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pveceph_fs_create
[Destroy CephFS]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_destroy_cephfs
[Ceph monitoring and troubleshooting]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_ceph_monitoring_and_troubleshooting
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/332afaa2febf4335b7d9c0eada764506.png
[Ceph RADOS Block Devices _RBD]: https://pve.proxmox.com/pve-docs/chapter-pvesm.html#ceph_rados_block_devices
[Ceph]: http://ceph.com/
[1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_1
[Ceph_s website]: http://docs.ceph.com/docs/luminous/start/hardware-recommendations/
[4]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_4
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 1]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/24b5283968814616877d0d79fe76247c.png
[configuration file system _pmxcfs]: https://pve.proxmox.com/pve-docs/chapter-pmxcfs.html#chapter_pmxcfs
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 2]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/4fbf9cdb901e447496e71eeaa3447ecb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 3]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/3f5caf25c4db471892440a91c1bda288.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 4]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/bfe8ead947c94ed7af368dc4dcdda562.png
[5]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_5
[6]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_6
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 5]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/da60f78018c74aa8982c63a572bc852a.png
[7]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_7
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 6]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/53ad4a2fea14488a92ef1ac42a94554b.png
[8]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_8
[9]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_9
[10]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_10
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 7]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/d38a2e1927a64b17ab7bf7fc5130b6de.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FsbHdheTI_size_16_color_FFFFFF_t_70 8]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/18/db5ae8e3cf0c4304a3d25ded80a26d8c.png
[CephFS as storage]: https://pve.proxmox.com/pve-docs/chapter-pvesm.html#storage_cephfs
[12]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_12
[13]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_13
[API]: https://pve.proxmox.com/pve-docs/api-viewer/index.html
[14]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_14
[15]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnote_15
[1 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_1
[http_docs.ceph.com_docs_luminous_start_intro]: http://docs.ceph.com/docs/luminous/start/intro/
[2]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_2
[http_docs.ceph.com_docs_luminous_architecture]: http://docs.ceph.com/docs/luminous/architecture/
[3]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_3
[http_docs.ceph.com_docs_luminous_glossary]: http://docs.ceph.com/docs/luminous/glossary
[4 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_4
[https_pve.proxmox.com_wiki_Full_Mesh_Network_for_Ceph_Server]: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
[5 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_5
[6 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_6
[http_docs.ceph.com_docs_luminous_mgr]: http://docs.ceph.com/docs/luminous/mgr/
[7 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_7
[http_ceph.com_community_new-luminous-bluestore]: http://ceph.com/community/new-luminous-bluestore/
[8 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_8
[http_ceph.com_pgcalc]: http://ceph.com/pgcalc/
[9 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_9
[http_docs.ceph.com_docs_luminous_rados_operations_pools]: http://docs.ceph.com/docs/luminous/rados/operations/pools/
[10 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_10
[https_ceph.com_wp-content_uploads_2016_08_weil-crush-sc06.pdf]: https://ceph.com/wp-content/uploads/2016/08/weil-crush-sc06.pdf
[11]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_11
[http_docs.ceph.com_docs_luminous_rados_operations_crush-map]: http://docs.ceph.com/docs/luminous/rados/operations/crush-map/
[12 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_12
[http_docs.ceph.com_docs_luminous_cephfs_multimds]: http://docs.ceph.com/docs/luminous/cephfs/multimds/
[13 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_13
[http_docs.ceph.com_docs_luminous_rados_operations_placement-groups]: http://docs.ceph.com/docs/luminous/rados/operations/placement-groups/
[14 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_14
[http_docs.ceph.com_docs_luminous_rados_troubleshooting_log-and-debug]: http://docs.ceph.com/docs/luminous/rados/troubleshooting/log-and-debug/
[15 1]: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_footnoteref_15
[http_docs.ceph.com_docs_luminous_rados_troubleshooting]: http://docs.ceph.com/docs/luminous/rados/troubleshooting/