This project forked from facebook/rocksdb
A library that provides an embeddable, persistent key-value store for fast storage.
Home Page: http://rocksdb.org
License: GNU General Public License v2.0
tidb format: [2022/03/19 20:18:40.055 +08:00][5][INFO] [db/db_impl/db_impl.cc:704] ------- PERSISTING STATS -------
but in rocksdb: 2022/04/13-08:48:22.772257 7f4de9cde700 261993027858 [INFO] [SPECIAL]
what does tidb do in tikv? in .rs file or rewrite logging in rocksdb?
Hi,
I found tikv/rocksdb has enhanced the encryption feature than facebook/rocksdb, is there a plan to submmit the related patches to facebook/rocksdb?
I'm developing the apache/incubator-pegasus project which is using rocksdb as the storage engine, researching encryption feature.
Thanks!
I read about the SPDK integration article https://www.pingcap.com/blog/tikv-and-spdk-pushing-the-limits-of-storage-performance/ and wondering about the current status of tikv with SPDK. Have you successfully integrated SPDK into tikv? If not, what problems you met, is there a schedule?
compile fail in g++-13, If it is g++-12, it still can't compiled.
compile fail
gcc & g++ version
gcc (Debian 13.2.0-4) 13.2.0
g++ (Debian 13.2.0-4) 13.2.0
gcc (Debian 12.2.0-9) 12.2.0
g++ (Debian 12.2.0-9) 12.2.0
error log:
CC options/customizable_test.o
In file included from /usr/include/c++/13/functional:59,
from ./db/db_impl/db_impl.h:13,
from ./db/db_test_util.h:24,
from options/customizable_test.cc:17:
In copy constructor ‘std::function<_Res(_ArgTypes ...)>::function(const std::function<_Res(_ArgTypes ...)>&) [with _Res = rocksdb::Status; _ArgTypes = {const rocksdb::ConfigOptions&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, void*}]’,
inlined from ‘rocksdb::OptionTypeInfo::OptionTypeInfo(int, rocksdb::OptionType, rocksdb::OptionVerificationType, rocksdb::OptionTypeFlags, const rocksdb::ParseFunc&, const rocksdb::SerializeFunc&, const rocksdb::EqualsFunc&)’ at ./include/rocksdb/utilities/options_type.h:240:9,
inlined from ‘static rocksdb::OptionTypeInfo rocksdb::OptionTypeInfo::AsCustomSharedPtr(int, rocksdb::OptionVerificationType, rocksdb::OptionTypeFlags, const rocksdb::SerializeFunc&, const rocksdb::EqualsFunc&) [with T = rocksdb::{anonymous}::TestCustomizable]’ at ./include/rocksdb/utilities/options_type.h:441:36,
inlined from ‘static rocksdb::OptionTypeInfo rocksdb::OptionTypeInfo::AsCustomSharedPtr(int, rocksdb::OptionVerificationType, rocksdb::OptionTypeFlags) [with T = rocksdb::{anonymous}::TestCustomizable]’ at ./include/rocksdb/utilities/options_type.h:419:69:
/usr/include/c++/13/bits/std_function.h:391:17: error: ‘<anonymous>’ may be used uninitialized [-Werror=maybe-uninitialized]
391 | __x._M_manager(_M_functor, __x._M_functor, __clone_functor);
| ~~~~^~~~~~~~~~
/usr/include/c++/13/bits/std_function.h: In function ‘static rocksdb::OptionTypeInfo rocksdb::OptionTypeInfo::AsCustomSharedPtr(int, rocksdb::OptionVerificationType, rocksdb::OptionTypeFlags) [with T = rocksdb::{anonymous}::TestCustomizable]’:
/usr/include/c++/13/bits/std_function.h:267:7: note: by argument 2 of type ‘const std::_Any_data&’ to ‘static bool std::_Function_handler<_Res(_ArgTypes ...), _Functor>::_M_manager(std::_Any_data&, const std::_Any_data&, std::_Manager_operation) [with _Res = rocksdb::Status; _Functor = rocksdb::OptionTypeInfo::AsCustomSharedPtr<rocksdb::{anonymous}::TestCustomizable>(int, rocksdb::OptionVerificationType, rocksdb::OptionTypeFlags, const rocksdb::SerializeFunc&, const rocksdb::EqualsFunc&)::<lambda(const rocksdb::ConfigOptions&, const std::string&, const std::string&, void*)>; _ArgTypes = {const rocksdb::ConfigOptions&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, void*}]’ declared here
267 | _M_manager(_Any_data& __dest, const _Any_data& __source,
| ^~~~~~~~~~
In file included from ./monitoring/statistics_impl.h:19,
from ./monitoring/statistics.h:10,
from ./monitoring/perf_step_timer.h:9,
from ./monitoring/perf_context_imp.h:7,
from ./util/user_comparator_wrapper.h:11,
from ./db/dbformat.h:22,
from ./db/memtable.h:20,
from ./db/memtable_list.h:16,
from ./db/column_family.h:17,
from ./db/db_impl/db_impl.h:22:
./include/rocksdb/utilities/options_type.h:428:12: note: ‘<anonymous>’ declared here
428 | return OptionTypeInfo(
| ^~~~~~~~~~~~~~~
429 | offset, OptionType::kCustomizable, ovt,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
430 | flags | OptionTypeFlags::kShared,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
431 | [](const ConfigOptions& opts, const std::string& name,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
432 | const std::string& value, void* addr) {
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
433 | auto* shared = static_cast<std::shared_ptr<T>*>(addr);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
434 | if (name == kIdPropName() && value.empty()) {
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
435 | shared->reset();
| ~~~~~~~~~~~~~~~~
436 | return Status::OK();
| ~~~~~~~~~~~~~~~~~~~~
437 | } else {
| ~~~~~~~~
438 | return T::CreateFromString(opts, value, shared);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
439 | }
| ~
440 | },
| ~~
441 | serialize_func, equals_func);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
make: *** [Makefile:2466: options/customizable_test.o] Error 1
Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev
When go-tpc tpcc runs the TiDB cluster created by operator way, TiKV goes into CrashLoopBackOff. The details of go-tpc is:
./bin/go-tpc tpcc --warehouses 2000 prepare -T 100
Cluster should go on running. Or throwing some errors.
2 TiKV nodes go into CrashLoopBackOff.
As mentioned above. Or refer to facebook#5178
The more infor about this Error:
BCC Info （2022-05-07 18:23:47 +0800）
The COUNT and POS are not aligned by 512 bits.
Stack Info:（2022-05-07 10:23:47 +0000）
TiKV Version:
Release Version: 5.4.0
Edition: Community
Git Commit Hash: b5262299604df88711d9ed4b84d43e9c507749a2
Git Commit Branch: heads/refs/tags/v5.4.0
UTC Build Time: Unknown (env var does not exist when building)
Rust Version: rustc 1.56.0-nightly (2faabf579 2021-07-27)
Enable Features: jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure
Profile: dist_release
We want to use Titan to some other storage system, is it still under developing? Is it stable enough to use it on production environment?
Thanks!
Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev
Build success.
When building in travis using ARM, we got the following error:
[ 72%] Building CXX object rocksdb/CMakeFiles/rocksdb.dir/utilities/blob_db/blob_db_impl.cc.o
In file included from /home/travis/build/tikv/titan/rocksdb/utilities/backupable/backupable_db.cc:19:
/home/travis/build/tikv/titan/rocksdb/util/channel.h:35:33: error: no matching constructor for initialization of 'std::lock_guard<std::mutex>'
std::lock_guard<std::mutex> lk(lock_);
^ ~~~~~
/usr/bin/../lib/gcc/aarch64-linux-gnu/9/../../../../include/c++/9/bits/std_mutex.h:158:16: note: candidate constructor not viable: 1st argument ('const std::mutex') would lose const qualifier
explicit lock_guard(mutex_type& __m) : _M_device(__m)
^
/usr/bin/../lib/gcc/aarch64-linux-gnu/9/../../../../include/c++/9/bits/std_mutex.h:167:7: note: candidate constructor not viable: no known conversion from 'const std::mutex' to 'const std::lock_guard<std::mutex>' for 1st argument
lock_guard(const lock_guard&) = delete;
^
/usr/bin/../lib/gcc/aarch64-linux-gnu/9/../../../../include/c++/9/bits/std_mutex.h:161:7: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
lock_guard(mutex_type& __m, adopt_lock_t) noexcept : _M_device(__m)
It prevents us from adding ARM build in travis.
Add ARM build in travis.
Reason for 6.29:
List of TiKV patches in chronological order:
Bullet-ed items are not cherry-picked, for either one of these reasons:
The task is to fix: facebook#6288
This is a port of https://github.com/kennthhz/rocksdb. Eventually split the patch into its read and write components. This will help simply review and testing of the patch.
The final objective is to enable async execution of RocksDB API from TiKV. Allow asynchronous execution using C++ 20 coroutines. Leverage io_uring for asynchronous IO support.
N/A
N/A
N/A
How about the performence on hdd, can you give some performence data about titan on hdd ? thanks
Be able to dynamically adjust max_subcompactions by via SetOptions()
Hi!
Recently I started RocksDB optimization process with PGO and have interesting results: tikv/tikv#13990 (comment) I think it would be interesting to consider building RocksDB with PGO as a TiKV dependency.
Further discussions about PGO and RocksDB are here (see the mentioned Discord chat in the discussion - all activity is in Discord): https://groups.google.com/g/rocksdb/c/j9iMFskUnpA
rocksdb/db/compaction/compaction.cc
Line 549 in ce0ea61
ExtractUserKey(*start) will assert false!
same to ExtractUserKey(*end)
Am I right?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.