Giter VIP home page Giter VIP logo

polardb-filesystem's Introduction

You can refer to Readme-CN for introduction in Chinese.

What is PFS

The PolarDB File System (hereafter simplified as PFS or PolarFS) is a high-performance Distributed File System in User Space that is developed by Alibaba Cloud and used in PolarDB. PFS follows the standards of Portable Operating System Interface (POSIX).

Quick Start

PFS for PostgreSQL uses the background process pfsdaemon to provide services. PFS for PostgreSQL is developed and tested based on AliOS and CentOS 7.5. Theoretically, PFS for PostgreSQL can also be built based on other Linux versions.

Install Dependencies

In the following example, CentOS 7.5 is selected. Before you build PFS for PostgreSQL, install the following software:

  • CMake: The CMake version must be 2.8 or later.
  • GCC or G++: The GNU Compiler Collection (GCC) version or the GNU C++ Compiler (G++) version must be 4.8.5 or later.
  • zlog: The zlog version must be 1.2.12 or later.
  • libaio-devel

We recommend that you use yum or apt-get command to install CMake, GCC or G++, and libaio-devel. To install zlog, you must download the source code and run make && sudo make install command. zlog is installed in the /usr/local/lib directory. If the dynamic libraries cannot be located when pfsdaemon is running, you can run ldconfig command to add /usr/local/lib directory into the settings of dynamic libraries.

We also provide you installation by rpm package. If you use rpm package installation, skip the two steps of "Compile" and "Install pfsdaemon".

Compile

After the dependencies are installed, go to the root directory of PFS source code and run the autobuild.sh script to compile PFS.

./autobuild.sh

Install pfsdaemon

To install or uninstall pfsdaemon, you must be granted the root permissions. After you compile PFS, run the install.sh script to automatically install pfsdaemon.

sudo ./install.sh

Run pfsdaemon

1. Format the storage devices of PFS.

First, run the following command to find the existing block devices:

lsblk

​ Then, select the block device that you want to format, such as nvme1n1, and run the following command to format the device:

sudo pfs -C disk mkfs nvme1n1
2. Run the following command to start pfsdaemon:
sudo /usr/local/polarstore/pfsd/bin/start_pfsd.sh -p nvme1n1

-p nvme1n1 is a parameter that specifies the device name, and it is required. ​ The following parameters are optional in the command:

-f (not daemon mode)
-w #nworkers
-c log_config_file
-b (if bind cpuset)
-e db ins id
-a shm directory
-i #inode_list_size
3. Run the following command to stop pfsdaemon:
sudo /usr/local/polarstore/pfsd/bin/stop_pfsd.sh nvme1n1

nvme1n1 specifies the device name.

4. Clear the files that are generated when pfsdaemon is running.

After stopping pfsdaemon, run the following command to clear the temporary files, logs and shared memory files that are generated when pfsdaemon is running:

sudo /usr/local/polarstore/pfsd/bin/clean_pfsd.sh nvme1n1

nvme1n1 specifies the device name.

5. Use the PFS tool to check whether PFS is running as expected.

Perform common operations on files to verify that PFS is running as expected. For more information, see Instruction to the PFS tool . For example, you can run the following commands to view the new file hello.txt:

sudo pfs -C disk touch /nvme1n1/hello.txt
sudo pfs -C disk ls /nvme1n1/

nvme1n1 specifies the device name.

Uninstall pfsdaemon

To uninstall pfsdaemon, you must be granted the root permissions.

1. Run the following command to stop pfsdaemon:
sudo /usr/local/polarstore/pfsd/bin/stop_pfsd.sh nvme1n1

nvme1n1 specifies the device name.

2. Run the uninstall.sh script to uninstall pfsdaemon.
sudo ./uninstall.sh

Documentation

The doc folder includes the following file:

Software License

PFS is developed based on the open source software license Apache License 2.0.

Publications

  • PolarFS: An Ultra-low Latency and Failure Resilient Distributed File System for Shared Storage Cloud Database in VLDB 2018
  • POLARDB Meets Computational Storage: Efficiently Support Analytical Workloads in Cloud-Native Relational Database in FAST 2020

Contact us

  • For more information about the ApsaraDB PolarDB PostgreSQL-compatible edition, see PolarDB Official Site.
  • Use the DingTalk application to scan the following QR code and join the DingTalk group.

polardb-filesystem's People

Contributors

chaohi avatar koperjian avatar wulianqing avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

polardb-filesystem's Issues

源码中定义的Block Size与论文不同

您好,在源码的src/pfs_core/pfs_impl.h文件中发现:第39行定义Block Size大小为4M,这似乎与Polarfs论文3.2节中将Block Size设置为64KB不符,请问这里是如何考虑的?

help

[PFS_LOG] Mar 17 12:12:01.703979 ERR [2357] cp: can't find any pbdname
[PFS_LOG] Mar 17 12:12:01.704013 ERR [2357] cp failed: impl specific
cp failed: impl specific

文件系统扩展功能(pfs -C disk growfs)不可用?

在使用 pfs -C disk growfs -o 201 -n 300 nvme0n1对ceph 镜像进行扩展时报错:
[root@polardb01 ~]#pfs tool cmd record:growfs -o 201 -n 300 nvme0n1
[PFS_LOG] Oct 20 16:59:27.971611 INF [1097361] pfs build version:libpfs_version_("pfsd-build-desc-fb5114f-Tue Apr 19 15:18:10 CST 2022")
[PFS_LOG] Oct 20 16:59:27.971724 INF [1097361] pid: 1092823, caller: -bash
[PFS_LOG] Oct 20 16:59:27.971765 INF [1097361] pid: 1092820, caller: sudo -i
[PFS_LOG] Oct 20 16:59:27.971800 INF [1097361] pid: 804729, caller: -bash
[PFS_LOG] Oct 20 16:59:27.971833 INF [1097361] pid: 804718, caller: sshd: deveop@pts/1
[PFS_LOG] Oct 20 16:59:27.971867 INF [1097361] pid: 804699, caller: sshd: deveop [priv]
[PFS_LOG] Oct 20 16:59:27.971901 INF [1097361] pid: 1386, caller: /usr/sbin/sshd -D
pfs growfs failed!
growfs failed: impl specific

但是使用 pfs -C disk info nvme0n1 命令显示信息却能正常显示:
[root@polardb01 ~]#pfs -C disk info nvme0n1
Blktag Info:
(0)allocnode: id 0, shift 0, nchild=201, nall 514560, nfree 511670, next 0
Direntry Info:
(0)allocnode: id 0, shift 0, nchild=201, nall 411648, nfree 410066, next 0
Inode Info:
(0)allocnode: id 0, shift 0, nchild=201, nall 411648, nfree 410066, next 0

PolarFS Configuration for shared storage

Hi everyone,
I was looking polarFS configuration for shared storage but I am missing some details about connection of storage chunks.
Like how the RDMA or NVMe/TCP using SPDK is exporting the disks to Polar Switch and how the setup is taking place

polardb-file-system和PolarFS不同点?

您好,当前开源的介绍让人很容易与PolarFS混淆。但从实际的代码看只看到一个File System Layer,而不包含Cluster部分,与论文中描述的PolarFS差异非常大。
请问,能不能介绍一下当前开源版本的主要特性和功能,其与PolarFS的不同点?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.