Ceph Geo-Replication Models
Overview
-
Ceph implements strong consistency between OSDs according to the replication factor, and reliability/availability according to the placement group (CRUSH MAP).
-
Clusters utilizing cross-site (a)sync will be performance-compromised and require fat, low-latency pipes, which strongly affects the Ceph RADOS protocol (both on RBD and object storage).
-
There is no recommendation/solution statement from Ceph upstream (or Red Hat) yet - https://tracker.ceph.com/projects/ceph/wiki/Can_Ceph_Support_Multiple_Data_Centers
-
It sounds like the team is developing/enhancing a ‘real’ stretch Ceph cluster. (Attachment 2)
Proposed Options: (Year 2018 update from Red Hat)
RBD - Stretch Pool w/ Tiers or Migration Status: You can deploy a stretch cluster today… But there’s a problem Stretch is sketch. Network latency is critical, as is bandwidth. Cannot “join” existing clusters; high level of coupling.
Low latency for performance. Requires nearby sites, limiting usefulness. Single (software) failure domain for all sites.
RBD - ASYNC MIRRORING Status: Available from Red Hat Ceph 3 (Note: Red Hat OpenStack 13 uses Ceph 2 by default. NSPCloud uses Ceph 3).
MISSING LINK: APPLICATION (k8s, etc.) ORCHESTRATION Hard for the IaaS layer to re-provision the app in a new site. Storage layer can’t solve it on its own either (e.g., how k8s maps/restructures the PV to blobs from the mirrored RBD). Need automated, declarative, structured specification for the entire app stack…
Object/CephFS - RGW or Federated GWs Federation/Zone/ZoneGroup, etc. RADOSGW metadata sync. Mirror/async, etc.