Giter VIP home page Giter VIP logo

tikv / rocksdb Goto Github PK

View Code? Open in Web Editor NEW

This project forked from facebook/rocksdb

120.0 51.0 89.0 185.34 MB

A library that provides an embeddable, persistent key-value store for fast storage.

Home Page: http://rocksdb.org

License: GNU General Public License v2.0

CMake 0.39% Makefile 0.73% Python 1.66% Shell 1.15% PowerShell 0.08% C++ 82.01% C 2.56% Java 10.15% Perl 1.12% Dockerfile 0.01% JavaScript 0.01% Assembly 0.06% Starlark 0.01% BitBake 0.04%

rocksdb's Introduction

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

CircleCI Status TravisCI Status Appveyor Build status PPC64le Build Status

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat ([email protected]) and Jeff Dean ([email protected])

This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.

rocksdb's People

Contributors

adamretter avatar agiardullo avatar ajkr avatar akankshamahajan15 avatar al13n321 avatar dhruba avatar emayanke avatar fyrz avatar grooverdan avatar haoboxu avatar igorcanadi avatar islamabdelrahman avatar jay-zhuang avatar joelmarcey avatar jsteemann avatar lightmark avatar liukai avatar ltamasi avatar lth avatar mdcallag avatar miasantreble avatar mrambacher avatar pdillinger avatar riversand963 avatar rven1 avatar sagar0 avatar siying avatar yhchiang avatar yuslepukhin avatar zhichao-cao avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rocksdb's Issues

Fix ARM build in travis

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev

Expected behavior

Build success.

Actual behavior

When building in travis using ARM, we got the following error:

[ 72%] Building CXX object rocksdb/CMakeFiles/rocksdb.dir/utilities/blob_db/blob_db_impl.cc.o
In file included from /home/travis/build/tikv/titan/rocksdb/utilities/backupable/backupable_db.cc:19:
/home/travis/build/tikv/titan/rocksdb/util/channel.h:35:33: error: no matching constructor for initialization of 'std::lock_guard<std::mutex>'
    std::lock_guard<std::mutex> lk(lock_);
                                ^  ~~~~~
/usr/bin/../lib/gcc/aarch64-linux-gnu/9/../../../../include/c++/9/bits/std_mutex.h:158:16: note: candidate constructor not viable: 1st argument ('const std::mutex') would lose const qualifier
      explicit lock_guard(mutex_type& __m) : _M_device(__m)
               ^
/usr/bin/../lib/gcc/aarch64-linux-gnu/9/../../../../include/c++/9/bits/std_mutex.h:167:7: note: candidate constructor not viable: no known conversion from 'const std::mutex' to 'const std::lock_guard<std::mutex>' for 1st argument
      lock_guard(const lock_guard&) = delete;
      ^
/usr/bin/../lib/gcc/aarch64-linux-gnu/9/../../../../include/c++/9/bits/std_mutex.h:161:7: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
      lock_guard(mutex_type& __m, adopt_lock_t) noexcept : _M_device(__m)

It prevents us from adding ARM build in travis.

Steps to reproduce the behavior

Add ARM build in travis.

Rocksdb Background IO error: While pwrite to file at offset ***: Invalid argument

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev
When go-tpc tpcc runs the TiDB cluster created by operator way, TiKV goes into CrashLoopBackOff. The details of go-tpc is:
./bin/go-tpc tpcc --warehouses 2000 prepare -T 100

Expected behavior

Cluster should go on running. Or throwing some errors.

Actual behavior

2 TiKV nodes go into CrashLoopBackOff.

Steps to reproduce the behavior

As mentioned above. Or refer to facebook#5178

The more infor about this Error:
BCC Info (2022-05-07 18:23:47 +0800)
The COUNT and POS are not aligned by 512 bits.
image

Stack Info:(2022-05-07 10:23:47 +0000)
image

TiKV Version:
Release Version: 5.4.0
Edition: Community
Git Commit Hash: b5262299604df88711d9ed4b84d43e9c507749a2
Git Commit Branch: heads/refs/tags/v5.4.0
UTC Build Time: Unknown (env var does not exist when building)
Rust Version: rustc 1.56.0-nightly (2faabf579 2021-07-27)
Enable Features: jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure
Profile: dist_release

Cannot compile in g++ environment

Expected behavior

compile fail in g++-13, If it is g++-12, it still can't compiled.

Actual behavior

compile fail

Steps to reproduce the behavior

gcc & g++ version

gcc (Debian 13.2.0-4) 13.2.0
g++ (Debian 13.2.0-4) 13.2.0
gcc (Debian 12.2.0-9) 12.2.0
g++ (Debian 12.2.0-9) 12.2.0

error log:

  CC       options/customizable_test.o
In file included from /usr/include/c++/13/functional:59,
                 from ./db/db_impl/db_impl.h:13,
                 from ./db/db_test_util.h:24,
                 from options/customizable_test.cc:17:
In copy constructor ‘std::function<_Res(_ArgTypes ...)>::function(const std::function<_Res(_ArgTypes ...)>&) [with _Res = rocksdb::Status; _ArgTypes = {const rocksdb::ConfigOptions&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, void*}]’,
    inlined from ‘rocksdb::OptionTypeInfo::OptionTypeInfo(int, rocksdb::OptionType, rocksdb::OptionVerificationType, rocksdb::OptionTypeFlags, const rocksdb::ParseFunc&, const rocksdb::SerializeFunc&, const rocksdb::EqualsFunc&)’ at ./include/rocksdb/utilities/options_type.h:240:9,
    inlined from ‘static rocksdb::OptionTypeInfo rocksdb::OptionTypeInfo::AsCustomSharedPtr(int, rocksdb::OptionVerificationType, rocksdb::OptionTypeFlags, const rocksdb::SerializeFunc&, const rocksdb::EqualsFunc&) [with T = rocksdb::{anonymous}::TestCustomizable]’ at ./include/rocksdb/utilities/options_type.h:441:36,
    inlined from ‘static rocksdb::OptionTypeInfo rocksdb::OptionTypeInfo::AsCustomSharedPtr(int, rocksdb::OptionVerificationType, rocksdb::OptionTypeFlags) [with T = rocksdb::{anonymous}::TestCustomizable]’ at ./include/rocksdb/utilities/options_type.h:419:69:
/usr/include/c++/13/bits/std_function.h:391:17: error: ‘<anonymous>’ may be used uninitialized [-Werror=maybe-uninitialized]
  391 |             __x._M_manager(_M_functor, __x._M_functor, __clone_functor);
      |             ~~~~^~~~~~~~~~
/usr/include/c++/13/bits/std_function.h: In function ‘static rocksdb::OptionTypeInfo rocksdb::OptionTypeInfo::AsCustomSharedPtr(int, rocksdb::OptionVerificationType, rocksdb::OptionTypeFlags) [with T = rocksdb::{anonymous}::TestCustomizable]’:
/usr/include/c++/13/bits/std_function.h:267:7: note: by argument 2 of type ‘const std::_Any_data&’ to ‘static bool std::_Function_handler<_Res(_ArgTypes ...), _Functor>::_M_manager(std::_Any_data&, const std::_Any_data&, std::_Manager_operation) [with _Res = rocksdb::Status; _Functor = rocksdb::OptionTypeInfo::AsCustomSharedPtr<rocksdb::{anonymous}::TestCustomizable>(int, rocksdb::OptionVerificationType, rocksdb::OptionTypeFlags, const rocksdb::SerializeFunc&, const rocksdb::EqualsFunc&)::<lambda(const rocksdb::ConfigOptions&, const std::string&, const std::string&, void*)>; _ArgTypes = {const rocksdb::ConfigOptions&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, void*}]’ declared here
  267 |       _M_manager(_Any_data& __dest, const _Any_data& __source,
      |       ^~~~~~~~~~
In file included from ./monitoring/statistics_impl.h:19,
                 from ./monitoring/statistics.h:10,
                 from ./monitoring/perf_step_timer.h:9,
                 from ./monitoring/perf_context_imp.h:7,
                 from ./util/user_comparator_wrapper.h:11,
                 from ./db/dbformat.h:22,
                 from ./db/memtable.h:20,
                 from ./db/memtable_list.h:16,
                 from ./db/column_family.h:17,
                 from ./db/db_impl/db_impl.h:22:
./include/rocksdb/utilities/options_type.h:428:12: note: ‘<anonymous>’ declared here
  428 |     return OptionTypeInfo(
      |            ^~~~~~~~~~~~~~~
  429 |         offset, OptionType::kCustomizable, ovt,
      |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  430 |         flags | OptionTypeFlags::kShared,
      |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  431 |         [](const ConfigOptions& opts, const std::string& name,
      |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  432 |            const std::string& value, void* addr) {
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  433 |           auto* shared = static_cast<std::shared_ptr<T>*>(addr);
      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  434 |           if (name == kIdPropName() && value.empty()) {
      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  435 |             shared->reset();
      |             ~~~~~~~~~~~~~~~~
  436 |             return Status::OK();
      |             ~~~~~~~~~~~~~~~~~~~~
  437 |           } else {
      |           ~~~~~~~~
  438 |             return T::CreateFromString(opts, value, shared);
      |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  439 |           }
      |           ~
  440 |         },
      |         ~~
  441 |         serialize_func, equals_func);
      |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
make: *** [Makefile:2466: options/customizable_test.o] Error 1

About encryption

Hi,
I found tikv/rocksdb has enhanced the encryption feature than facebook/rocksdb, is there a plan to submmit the related patches to facebook/rocksdb?
I'm developing the apache/incubator-pegasus project which is using rocksdb as the storage engine, researching encryption feature.
Thanks!

Add coroutine (C++20) and io_uring support

This is a port of https://github.com/kennthhz/rocksdb. Eventually split the patch into its read and write components. This will help simply review and testing of the patch.

The final objective is to enable async execution of RocksDB API from TiKV. Allow asynchronous execution using C++ 20 coroutines. Leverage io_uring for asynchronous IO support.

Expected behavior

N/A

Actual behavior

N/A

Steps to reproduce the behavior

N/A

How does TIDB format logging for rocksdb?

tidb format: [2022/03/19 20:18:40.055 +08:00][5][INFO] [db/db_impl/db_impl.cc:704] ------- PERSISTING STATS -------

but in rocksdb: 2022/04/13-08:48:22.772257 7f4de9cde700 261993027858 [INFO] [SPECIAL]

what does tidb do in tikv? in .rs file or rewrite logging in rocksdb?

Upgrade to facebook 6.29

Reason for 6.29:

  • Version above 6.27 has superior performance (especially read) according to official benchmarks
  • Not sure about facebook's backport policy, but 6.29 is getting more minor releases than predecessors

List of TiKV patches in chronological order:

Bullet-ed items are not cherry-picked, for either one of these reasons:

  • Build related
  • Already deprecated in 6.4.tikv (e.g. multi batch write)
  • Rendered unnecessary by new improvements from 6.29.fb (consistency check and cache key collision)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.