Giter VIP home page Giter VIP logo

kno's Introduction

This is KNO

Kno is a platform for high-performance symbolic computing and qualitative data processing at scale, especially suited for development and deployment of symbolic AI services and solutions. The core platform is implemented in C for Unix platforms (including MacOS) and released under the open-source GPL license.

Kno is based on the general architecture and data model of http://www.framerd.org/ which has been used extensively in both experimental and commercial deployments supporting tens of thousands of users.

Kno databases readily include millions of searchable frames with multiple properties and relations. These databases can be distributed over multiple networked machines. Kno includes its own native database formats but also provides backends to libraries such as Google's LevelDB or RocksDB. Bundled drivers also allow Kno to connect to external databases such as MongoDB or MySQL.

The Kno runtime and language kernel provides:

  • a lockless reference-counting GC suited to real-time applications;
  • full UTF-8 support with conversion from external encodings (using libu8)
  • zero-cost object references for operating over large graphs using limited memory;
  • data structures (including the lockless GC) optimized for multiple threads and CPUs, enabling;
  • optimized data structures and algorithms for many set, lookup, and search operations;
  • a simple and powerful abstract storage layer for representing graphs and graph indexes, with drivers for both native databases and popular open-source database libraries or services;
  • an extensible type system available from both C and other hosted languages;
  • Knox, an extension, application, and scripting language (based on Scheme) which leverages all of the above features, and adds:
  • support for non-deterministic programming, a natural paradigm for many kinds of search and AI algorithms;
  • a mature module system and profiling facility to support programming in the large;
  • parseltongue, an FFI interface for Python which allows Knox programs to transparently access Python libraries and modules;
  • native high-performance modules for text processing using a structured pattern language, as well as have facilities for stemming and morphological analysis;
  • native modules for parsing and emitting various web formats including XML, JSON, MIME, and various URI formats, as well as native access to services such as LDNS;
  • native database drivers for connecting with MongoDB and SQL (through both ODBC and vendor-provided drivers);
  • Apache integration through a custom mod_knocgi module and a servlet engine supporting distributed processing and asynchronous I/O;
  • native wrappers for:
    • image processing (imagick, qrencode, exif, etc),
    • cryptographic functions,
    • text processing (tidy, markdown,hyphenation, etc),
    • databases including leveldb, rocksdb, sqlite, etc;
    • archival data files including ziptools, libarchive, and libzip,

Background

KNO was originally implemented as FramerD in the late 1990s at MIT's Media Laboratory. FramerD was designed to support large knowledge bases and practical semantic information systems. The laboratory's support, especially from its News In the Future program, is gratefully acknowledged. This version can be downloaded from SourceForge.

Starting in 2005, beingmeta began developing a new version of FramerD with a focus on scalability in two directions: down to low-powered devices and up to multi-core high performance servers and workstations. This included optimizing the underlying C code for modern cache and pipeline-focused CPU architectures and the introduction of finer-grained thread locking, implementation of a lock-free garbage collector, and a compiler to a SCHEME-like VM.

FramerD was renamed KNO in May 2019 and released under an open source license (the AGPLV2).

beingmeta's implementation of KNO provides a novel query optimization technique called iterated partial evaluation (patent) which can optimize complex high-latency queries by up to a factor of ten. The method uses progressively complete partial evaluations of a query to bundle together data references to both reduce the number of round trips and allow remote data sources to optimize retrieval.

Source modules

KNO comes bundled with modules written in it's native scripting language (a variant of Scheme) to provide additional functions:

  • access to many AWS (Amazon Web Services) APIs, including S3, SQS, SES, EC2 and others;
  • a facility for generating and using JWTs (Javascript Web Tokens) including both symmetric and asymmetric (public key) signatures;
  • libraries for DOM manipulation and processing, based on the representations generated by FramerD's native XML/HTML parser;
  • an extensible generic pathname facility (gpath) for working with a range of file-like data repositories (including S3, Dropbox, in-memory filesystems, zip files, and web resources);
  • many tools for accessing the BRICO semantic knowledge base;
  • modules for advanced parsing and rendering of times and numbers;
  • a lightweight implementation of structured records (ezrecords);
  • facilities for fine-grained logging of program activity;
  • various caching facilities for improving performance against resource-intensive functions or services;
  • support for OAuth2 authorization and API access;
  • HTTP access to CouchDB databases and RSS feeds;
  • facilities for integrating with SOAP-based web services;
  • interfaces to the APIs for Facebook, Gravatar, Dropbox, OpenLibrary, LibraryThing, Twilio (SMS), PayPal, and others.

kno's People

Contributors

bmdeploy avatar haase avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

kno's Issues

CONFIG documentation

Separate CONFIG use and CONFIG customization. Describe how delayed configuration works. Think about the one value/multiple value/init running piece.

Problem: for some configs, you want to run them in assertion order, for others, you want to run them in reverse assertion order. Some may be set-once? Some may be init-once? Some may be single valued? Use flags?

Reqdata documentation

Describe creation, modification, reader macros, req/call. Short servlet documentation.

Reqdata testing

Add testing for creating, modifying, accessing, binding req contexts.

knox documentation

Wiki page and man page.

Document no inits, how to get them anyway.

How arguments get parsed, passed to (main).

Document scheme parser extensions

Including:
choices {} and qchoices #{}
compounds #%
packets and secrets
@ (oids), including @? and others
Other # macros, and #:name: macros

knoc documentation

Wiki page and man page using ronn.

Startup: .kno.cfg, .kno.scm, local dir too.
History: macros, printout functions, etc.

Brief intro to knoc.el, readline.

XML output documentation

Rely on XML representation from XML parsing (or separate document). Document basic functions.

{} =?= #f

Decide on handling of {} for branches.

Proposal: for when, {} != true
for unless {} == false (???)
for loops, {} = exit

PRINTOUT testing

Try different data type outputs, combinations of writes and object results, embedded calls, etc.

Add history refs to printout output

Add a "use history" flag to text streams. When this is set, have PRINTOUT and friends check non-string values against the history, adding entries if needed. Also add $ref and $write printout primitives to force them to be generated.

Option tests

getopt/testopt/opt+ and lower-level creation. Test for side-effects, inheritance, overriding, and inherited side-effects.

Thread options

Look at standard thread opts and add getopt/setopt and init opts support.

libu8 contours fix

This is really a libu8 bug, but the best place to demonstrate it is by signalling a Scheme error. It looks like u8_throw_contour goes to the next contour, above the innermost one, while it should really longjmp to the current contour, which should then pop, potentially copying an error from the contour.

To explore, define (oops x) to generate an error. break on the u8_do_contour functions and evaluate (oops).

CONFIG tests

Check standard config vars, config-def! and varconfig! usage and ordering.

webtools documentation

Wiki page and man page.

ASYNC/buffered I/O.

Custom functions (main, etc). Config variables.

Provisionally document injection.

Apache configuration, mod_knoweb.

ZeroMQ base

  • just one zeromq_ctx
  • have read automatically handle more
  • check checking return values

Fix history references for non-symbol slot values

Right now, the value of the foo slot of #1 is only addressable as #1.foo, which can be tricky if the slotid is an OID or hard to type. There were problems using integer aliases for this, but that would be very handy. I think the problem may have come from objects being modified after display, especially when the keys are sorted.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.