figure out how scylla uses the fair queue mods per request.

We now have a priority_manager which stablishes system level priorities and weights - like scylla. However, I'm not sure how to push them through the futures returned for the RPC server.

mak with a number like 0,1 ala

\d?_smf\d+_(\d+).wal

from

smf\d+_(\d+).wal

allows us to embed status information on the filename.
maybefilename should be a

struct filename{
   char prefix[4];
   uint16_t core;
   uint32_t status
   uint64_t epoch;
  char padding___[4]; // or the gcc macros
};

this will effecitvely allows us to char* copy (srd, dest) as the filename and automagically extract all the status...

maybe... retrofit stuff to fix it. - could generate it w/ flatbuffers wihich will automagicall ypad like that too.

the WAL can be encrypted, compressed, split, etc.

Use a similar --> filters [ many ] -> save for input and

read --> undo_filteers [ many ] -> user for output of the WAL.

Allows for data at rest. Might need to come up w/ database header format per file. - maybe increase filesize?

Seastar is moving away from collectd and into the prometheus api.

https://github.com/scylladb/seastar/wiki/Metrics-report

Port your metrics over to the prometheus interface.

Write scripts for the ansible script to download and install prometheus.

Prevent reading more memory, if at limit.

Should not read more data. become stale. Clean up ALL state.

http://www.pcg-random.org/

TL;DR:

Mersene Twister, etc...

Seastar comes with a few scripts to run in high perf mode after setting NIC's IRQ balance and all that jazz.

Add Dan's ideas for IRQ balance, cpu pinning, etc - then measure posix latency

https://gist.github.com/93a76a18ea93ec29621d90e2abf79a23

Dan's latnecy was 40 micros after thread pinning, etc. copy that knowledge here on a python script

could stress a few things... mainly we need to ensure that we write a LARGE file... like 3 gigs. etc...
then read it back... iterate through the records... ensure the payload makes sense.

test on binary data and see how fast the checksums can be computed...

https://github.com/lemire/clhash

... not sure if worth changing at the moment.

Scylla builds on fedora but packages a debian binary.

Either finish the ldd work thing you started or create something like this for smurf

https://github.com/scylladb/scylla/blob/76628a7b0b55858621ab04f1ba1711eb5baf3dfa/dist/ubuntu/build_deb.sh

currently, cache & write are implemented. Read has the lazy file, but missing glue.
add that & file ticket for rstats, wstats, and cstats for read,write, and cache statistics during reads.

seastar has a set of services that are also helpful for us.

Currently we only support zstd.

We should let blosc manage the transitive compressor dependencies for us. We can simply ecompress,decompress...

https://github.com/Blosc/c-blosc/releases/tag/v1.11.2

We might loose tunning abilities of their compressors tho.

given multiple futures on a single wal_writer_node.*{h,cc}

and also assuming tha tyou have a big write payload i.e.: > max_file_size

and assuming that one of them is less than max_file_size you might interleave the writes to the wal file.

use a seastar::semaphore to serialize writes and yield other co-routines until ready.

https://github.com/fuchsia-mirror/ftl/blob/master/log_level.h

boost program options is verbose. create a couple of macros on a static object like the log... easy-boiiiiiii

http://llvm.org/docs/CommandGuide/lit.html#lit-infrastructure

effectively we have a bunch of main.cc files that need reporting and summarizing.

look for all the simple if () LOG... and change them to LOG_*_IF to capture the conditional

now that the rpc_server.h and rpc_server.cc have incoming& outgoing filters hooked up, we

should add it to the client too.

maybe @rblafford would like to take this?

during non-exception execution, you save the full directory scan? maybe? this is unsafe?

i.e.: what if you just link some ramdom file... you might override it ? no?

Instead of using a client connector, if the method is blocking then we should implement the
semaphore on the client side, so that only one request can call

input_stream<char>::read_exactly().then

Otherwise the reads interpolate.

clean up all the separate impl files into their own actual .cc files
& update the cmake file.

after prototype is finished.

DPDK is really hard to test as there is no localhost, however seastar has a
wrapper for localhost that effecitvely uses their future mechanism for testing.

Need to figure out how to run their unit tests.

https://github.com/senior7515/smf/blob/feature/wal_reader/src/filesystem/wal_writer_node.cc#L167

is a bug. fix it.

We already have the rpc_recv_typed_context which is the exact same type of template i would
want to use for the rpc_typed_envelope so all we have to do is make this a template of 2 types.

One for receive and one for send. and make aliases.

template<typename T>
struct rpc_typed_envelope : public rpc_typed<T, rpc_envelope> {} ;

struct rpc_recv_typed_context: public rpc_typed<MyType, rpc_recv_context> {} ;

#pragma once
#include "rpc/rpc_recv_context.h"
namespace smf {
template <typename T> struct rpc_recv_typed_context {
  using type = T;
  using opt_recv_ctx_t = std::experimental::optional<rpc_recv_context>;

  rpc_recv_typed_context() : ctx(std::experimental::nullopt) {}
  rpc_recv_typed_context(opt_recv_ctx_t t) : ctx(std::move(t)) {}
  rpc_recv_typed_context(rpc_recv_typed_context<T> &&o) noexcept
    : ctx(std::move(o.ctx)) {}

  T *get() {
    if(ctx) {
      return flatbuffers::GetMutableRoot<T>(
        ctx.value().payload->mutable_body()->Data());
    }
    return nullptr;
  }

  /// \brief returns true of this obj
  /// so that it works with
  /// \code
  ///     if(obj){}
  /// \endcode
  /// simply forward the bool operator to the option
  operator bool() const { return ctx.operator bool(); }
  opt_recv_ctx_t ctx;
};
}

I just added the american fuzzy lop to fuzz the inputs to the wal readers.

Need to fix the readers and writers of this stuff.

righ tnow the filename is

smf<core: uint16_t>_<epoch: uint64_t>.wal

with programmable prefix from smf to user input...

adds unnecessary param. remove.

if the wal is

smf0_1234.wal

without a time stamp, next time l-core 0 starts up at 1234 epoch it will override the same name.

This only happens If and only if, the file is empty. Otherwise it will move the offset to the correct byte.

The proposed change would label it like this:

run:<id-microsecond>:smf:0:1234.wal

rpc_codegen should take advatnage of this for known sequential int collections (i.e.: record sequences):

https://github.com/lemire/FastPFor.git

and the transitive links there.

We should add the number of times we would like to run a test - realy a main for the integration tests.

Currently i discovered a bug on the wal.h on second writes since it does directory scanning after. I think there is a raise condition on the cores it tries to do it.

In addition to we should change the number of cores from run to run.

{
"repeat_in_same_dir": true
"repeat_time": 3
}

https://github.com/tiny-dnn/tiny-dnn#examples

Effectively copy the script from scylla here and write a python wrapper that is more dynamic and doesn't leave your NIC all fucked up in case of error.

https://gist.github.com/49bb698cb4db2ea5831613eefa35c0f6

Need to add a way to reset - maybe even wrapped by configuration

the rpc_server.h

has a member var (rpc_limits)

these limits on mem need to be divided by the smp::count

Effectively, the wal-clock-pro, page-chunk, sorting, shuffling, etc. can be a generic cache.

it can even be templetized by async_clock_pro vs clock_pro

src/utils/caching/wal_clock.h

it can operate on a struct w/ public funcs
and the wrappers will simply use that.

Also uses a std::array behind the scenes.

the http server is for metrics similar to google /varz or .. any modern db... aka kudu.

Easy to see latency metrics of the RPC's as well as logs and ... all the seastar

engine() metrics.

I just discovered 2 bugs on the wal_requests.h by disabling copy and assignment.

Go through each file and ensure that you aren't paying any copies accidentally.

Mostly it involves having move constructors for all default types.

via

// A macro to disallow the copy constructor and operator= functions
// This is usually placed in the private: declarations for a class.
#define SMF_DISALLOW_COPY_AND_ASSIGN(TypeName) \
  TypeName(const TypeName &) = delete;         \
  void operator=(const TypeName &) = delete

#define SMF_DISALLOW_IMPLICIT_CONSTRUCTORS(TypeName) \
  TypeName() = delete;                               \
  SMF_DISALLOW_COPY_AND_ASSIGN(TypeName)

Lemire's work on fastmod improved tensor flow by 20% on % ops.

https://github.com/lemire/fastrange

tensorflow/tensorflow@a47a300

during the healing process just managet the files that map to your renaming core.

TL;DR:

https://gist.github.com/c7dd61cc7b84ab651e3ad438080b74bc

Turns out that seastar has a seastar.conf and a io.conf

under $home/.config/seastar/seastar.conf - in fact, that's how you get to run multiple apps under same user.

change the home directory.. kinda jenky .. maybe submit patch upstream

everything but the rpc.fbs is snake_case. somehow rpc.fbs is using CamelCase for it's datastructs...

we need to settle on one style for consistency.

rpc and wal.fbs

use big ints (uint) to store bit flags. Realistically, we can use smaller enums (byte) that have semantic meaning rather than bitfields... same size on the wire. i.e.: we can have 4 enums representing all the combinations we want. on the wire, it's padded to the largest struct via

std::align(... destructive) ... so we get aligned for the width of L1 - which on recent seastar mailing list it proved to be 64.

folly aligns to be 128... but that's no longer the case. PPL suggest to align 64. -

flatbuffers does all this for us, so no need to worry anyways.

Also need fully serialized types so that the users can simply push a serialized type. It's going to make writing easier.

https://github.com/ongardie/raft.tla/blob/master/raft.tla

see the when_all_succeed() commit for handling proper failures of futures on all for_each calls.

we're currently using boost dynamic bitmap which uses more memory than needed:

https://github.com/lemire/EWAHBoolArray

change deps to this... specially important w/ the file_lease+manager.h