> For the complete documentation index, see [llms.txt](https://chain-dimension.gitbook.io/dimensionchain/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://chain-dimension.gitbook.io/dimensionchain/about-dimensionchain/core-technologies/dimension-star-data-storage-protocol.md). # Dimension Star Data Storage Protocol In terms of data storage, DimensionChain has designed the unique Dimension Starfish data storage protocol, using chain library separation technology and Starfish database to achieve complete decentralization, distribution, partition fault tolerance and extreme scalability. The Starfish data repository is a distributed peer-to-peer network node system. Small network clusters are formed between database nodes according to certain rules, and the seed routing information of the clusters exists on the chain, and the routing table itself is on top of the existing P2P system. dimension data repository is a data access system that takes into account both high frequency and low frequency, and the database itself provides statistics on the response of data through the block chain layer, and reconciles the request and response information at the time of checkpointing, and provides the ability to connect information across multiple physical IDCs. There are no centralized management nodes in the Starfish data repository, only data storage nodes. All storage nodes have exactly the same responsibilities and will provide the same services to the outside world, so there will never be a single point of failure in the whole system. The decentralized architecture makes horizontal scaling of the system very easy, and nodes can be added directly to the entire cluster of Dimension's Starfish data repository at any time, and will only cause a small amount of data migration in the cluster. ![](/files/huV1Xw10Z7yeOahcAP5j) The core concept of Dimension Starfish Data Repository includes the principle of Incremental Scalability, which requires a mechanism that can dynamically slice among a set of nodes, also known as DHT (distributed hash table), through which the TSDB can distribute the load to different storage nodes in a reasonable manner. The slicing strategy of Dimension data repository relies on Consistent hashing and is further adapted to the heterogeneity of the nodes, we use an algorithm called CCHDP algorithm, whose main steps are. (1) Firstly, a clustering algorithm is used to classify the set of devices such that the difference in weights of devices in each class is within a preset range; (2) After the clustering is completed, the inter-class layout mechanism divides the \[0, 1] interval into multiple subintervals according to the weights of the classes, assigns a subinterval to each class, and assigns the data that fall into a certain subinterval to the corresponding class; (3) The internal layout mechanism of each class uses a consistent hash method to redistribute data to specific devices. Each node in the Dimension data repository system is assigned a random location when it first joins the cluster. The randomness of the algorithm may cause different nodes to handle different ranges, and eventually the load on each node is not the same. To solve this problem, Dimension Data Repository uses a variant of the Consistent Hash algorithm to assign the same physical node to multiple locations in the ring, becoming multiple virtual nodes. Dimension Starfish database clustering and partitioning Clustering and partitioning capabilities are also provided in Dimension Data Repository, a distributed database system in which data files are broadcast to the entire network, and too large a single cluster may lead to network message propagation and congestion in extreme cases, so Starfish Data Repository provides global, location-based clustering and partitioning capabilities. Network failures can lead to partitioning of virtual rings, regardless of whether nodes are located within the same server room or in different server rooms. After partitioning, the nodes of Dimension Data Repository join the exit mechanism causing each partitioned node to form a new virtual ring. The process of forming a new virtual ring is completely automatic and requires no involvement of operations and maintenance personnel. Suppose there is a virtual ring whose nodes are partly in the European data center and partly in the Asian data center. If the network connection between the two data centers goes down, the Dimension data repository forms a new virtual ring in each data center. After some time, the communication between the European data center and the Asian data center is restored, and this time the two virtual rings need to be merged. We can specify some special landmark nodes. All nodes in the ring without landmark nodes first leave the ring they are in and then rejoin the ring where the landmark node is located. After a period of time, the whole system is again left with only one virtual ring. ![](/files/BmpzL8IXNs8QcIoC6W7r) In the Dimension Starfish Storage Protocol, the two nodes communicating each time will agree on the information of the nodes in the current system. By passing membership information between nodes to each other, eventually all nodes in the entire cluster of Dimension data repository will agree on the membership information. When a new node is added to the Dimension data repository, a slice transfer between nodes occurs. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://chain-dimension.gitbook.io/dimensionchain/about-dimensionchain/core-technologies/dimension-star-data-storage-protocol.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.