lightsys / centrallix Goto Github PK

Some manual security testing has been done by interns, but it would be great to have some automated way to do "both generic fuzzing and probing using a compilation of typical web vulnerabilities." I know that tools along this line exist, but more research is needed to see what ones are available and whether they would make sense for Centrallix.

This includes drivers for:

• querytree objects
• query objects
• querypivot objects
• xml objects
• structure file objects
• UNIX local file/directory objects
• sybase
• mysql
• CSV
• "Web services (http/https driver), including layering JSON and XML on top of it for the fetched response data"
• probably some others too

Unit testing could be done in a contained way with mocking, but thorough integration testing for some of these drivers would require interaction with separate servers (the database drivers, for instance). One way to tackle this would be through automated provisioning of environments with different servers available - see #32.

When the user launches a pdf report, an error message should be displayed if ghostscript is not installed.

There are currently no tests for Centrallix's frontend, either end-to-end integration tests or smaller unit tests of individual JavaScript modules. Some ideas on frontend unit tests that could be useful (though some of these may be more on the integration side, depending on how much exactly is handled by individual JS modules):

• form states (no data, new, view, modify, etc.)
• objectsource replica contents
• variable contents
• form element values
• which control has focus
• etc...

One potential issue is how much Centrallix's system for handling JavaScript works with standard JS unit testing frameworks.

Currently, widgets are absolutely positioned and sized, which means they do not adapt to the size of the browser window or re-render when the window is resized. Implementing a truly responsive solution would be a lot of work given how much of the frontend currently leans on absolute positioning and sizing. So as a temporary stopgap, try using CSS calc() and percentages to at least resize widgets when the window is resized.

The calc function should look something like this:

calc(initial width + (100% - container's initial width) * flex ratio)

• "initial width" = width from first render (in px?)
• 100% = current width of window (as it's being resized)
• "container's initial width" = width of window from first render
• So "100% - containers initial width" = total new space available
• "flex ratio" = current widget's flexibility / total section flex (= % flex that is "mine")
• So (100% - container's initial width) * flex ratio is the amount of new space available that is "mine"

As far as I can tell, all positioning logic currently happens on the server side. apos.c uses the widget tree (wgtr.c) to calculate the x, y, width, height, flex width, and flex height of each "node" (widget?) (or maybe it creates "nodes"?), then builds and renders a grid that widgets place themselves in. The widgets decide how much of their (allotted?) width/height to use.

Supposedly every section and widget has "flex" area, and apos.c knows where inter-widget space needs to be static, vs what is flexible (possibly using section divisions?).

When compiling centrallix-lib on Ubuntu 18.04LTS (gcc v7.4.0, target:x86_64), the following error occurs:

src/qprintf.c: In function ‘qpf_internal_Translate’:
src/qprintf.c:650:17: error: ‘SIZE_MAX’ undeclared (first use in this function);

GCC info:

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.4.0-1ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)

Possible solution:

#ifndef SIZE_MAX
#ifdef __SIZE_MAX__
#define SIZE_MAX __SIZE_MAX__
#endif
#endif

Other possible solution:

#include <stdint.h>

See #14 (comment)

Currently, the only way to set up a Centrallix instance (whether for development, automated testing, or production) is to manually create a VM, set up the environment, and install and configure Centrallix. This takes time and introduces opportunities for mistakes whenever an intern or volunteer onboards, a client wants an update or a brand-new installation, or a developer wants to write a new kind of automated test. (Consider for instance that folks at LightSys will re-use VMs left over from other volunteers instead of creating new ones, even though that can lead to all kinds of fun surprises - clearly starting up a fresh Centrallix environment is a pain.) This is also an obstacle to doing any kind of continuous integration / automatically running tests asynchronously on push, and to having confidence that tests passing in a dev environment also hold true for a real prod environment.

It would be a boon to LightSys if there were some way anyone could quickly, easily create a new environment with a specific configuration (dev or prod, specific DB servers available, etc) and then work in it, run tests on it, or deploy it. Moreover, if the same system and environments could be used both for development + testing and production, it would add a significant measure of confidence that unexpected issues won't crop up in production. Ideally this system could also be easily dropped into a CI service for extra (automated!) confidence.

One potential issue is that LightSys currently only uses VMs on site, and has run into capacity limits in the past because of that. Most automated setup/provisioning systems like this make use of cloud services so they can easily scale, or they at least have more extensive onsite server capacity. But also consider how far it might go to just make it easy for people to spin up at least an instance or two on their own machines.

More research needs to be done on the tooling for a system like this, but some promising avenues of exploration are container systems like Docker and IT automation tools like Ansible.

Centrallix-Lib modules that still need test completion:

• mtsession module
• newmalloc module
• stparse module
• xarray module
• xhandle module
• xhash module
• xhashqueue module
• xringqueue module
• xstring module

On some systems, the JSON-C header files are stored in /usr/include/json-c instead of /usr/include/json.

Currently, Centrallix's JavaScript is structured rather like C code - lots of functions in the global namespace, with module prefixes on function names. This system should be updated to be more clean, standard, and modern.

One suggested structure is outlined here - essentially, moving Centrallix functionality into a structured global variable $CX.

Alternatively, could it be possible to use ES6 modules + imports instead (with standard JS objects, functions, classes, etc)? The main issue with this is that supporting older / less standard browsers would require setting up Babel/Webpack/some other transpilation and/or build toolchain (see #22).

There are currently no tests for Centrallix's frontend, either end-to-end integration tests or smaller unit tests of individual JavaScript modules. Some ideas on frontend integration tests that could be useful:

• Screendiff tests for visual appearance of widgets
• Tests for how UI interaction affects server data

I know there are tools out there for screendiffs, interaction with a UI via a headless browser, etc, but more research is needed on what specific toolchains would work well for Centrallix (especially for combining UI interaction with server-side data validation). It would also be worth thinking about how much of this testing should be centered in Centrallix vs. in Kardia.

Currently, Centrallix JavaScript does not use any kind of compilation process or transpiling, minification, etc. Drivers for different kinds of widgets, for instance, just have their raw JS files linked directly in the HTML.

Building Centrallix JS with a tool like Webpack instead could allow for various benefits:

• Babel transpilation, which lets you use modern ES6 JavaScript features while still retaining high browser compatibility. Having ES6 available means code can be much cleaner as well as more accessible to modern JS developers.
• Minification, which can drastically reduce JS file sizes and thus make requests for them much faster.
• Bundling, which can reduce the number of requests for JS files that need to be made to the server.
• More extensive compilation with a tool like Google Closure, which can compress JS files even further.
• It could be possible to start using TypeScript incrementally.
• These are all very standard, widely used modern JS practices that would be familiar to any reasonably experienced JS developer.

However, implementing a Webpack toolchain could also have some downsides:

• I'm not familiar with all the ins and outs of how Centrallix JS works, and wouldn't be surprised if some way it's set up would be difficult to integrate with Webpack and the kinds of JS architectures it's designed for.
• As such, it might take a fair bit of work to get a Webpack toolchain working reliably.
• Even though all of these tools are used and trusted widely, any kind of npm-based development involves complex dependency trees that some people see as security liabilities.
• Using a toolchain like this always adds some complexity - more potential points of failure, more things to learn, more onboarding necessary for new developers.

More research on the pros and cons is needed, as well as more consideration of how high-priority a project like this would be.

See #13 (comment)

Currently the MoneyType struct is comprised of an int WholePart (= floor(amount)) and an unsigned short FractionPart (= amount - floor(amount)). It should be converted to having a single long long Value that represents the amount in 1/10000ths of the currency unit.

This issue is just for the server-side handling. For the time being, the JSON representation send to the frontend should remain the same (with separate WholePart and FractionPart).

Currently, communication between the Centrallix client and server is done with iframes and anchor tags rather than standard AJAX requests. Here's an example:

This is a holdover from Netscape Navigator not supporting XMLHttpRequest and no longer necessary or ideal with modern browsers. So client/server communication should be converted to using standard AJAX with JSON instead. The Centrallix server already has JSON endpoints, so most of the work needs to be done on the client side so it actually uses those endpoints.

There is some more information about how the existing iframe/a tag system works here and here. And more information about how the Centrallix server handles JSON here and here.

On the client side, I have been told that some key files are htdrv_osrc.js, htdrv_page.js, and htdrv_component.js but I have yet to find anywhere that iframes or a tags are explicitly referred to on the client side :/

On the server side, a key file is net_http_osml.c, and there has been some work done in it to support json in the json_client_data branch.

Note that the existing API is stateful rather than RESTful (for instance, open query, then get data, then close query, etc instead of a single repeatable GET request), though there are some more RESTful pub/sub APIs. For the time being, just use the stateful APIs. Eventually Centrallix will be moving to a replication (? meaning pub/sub based?) model rather than a standard RESTful one.

Centrallix targets Chrome and Firefox. Safari and Edge support would be great too, but IE is not a priority. Centrallix also does not currently use Webpack, Babel, or any other kind of transpilation, minification, etc (see #22).

You may want to keep in mind the proposed JavaScript module reorganization in #21 as you work on this.

I'm guessing these are the modules here: https://github.com/LightSys/centrallix/tree/master/centrallix/utility This includes testing the Centrallix version of stparse.c (which is presumably different than the Centrallix-Lib version?).

There is some testing for memory leaks currently in Centrallix-Lib, but this could be expanded to be more thorough. The general idea is to perform an operation 3 times (or more?) and make sure the memory allocation situation is the same after each repeat. Some acceptable exceptions may need to be whitelisted.

It may be worth looking into whether there is some way to automatically add memory leak checking to existing unit tests, so the same operations don't have to be set up, torn down, etc in multiple places (to check for memory leaks and to do other unit testing). If that's not possible though, test helpers / utilities could help with that problem.

CXSQL tests that still need to be completed:

• tests for remaining SQL functions and operators (summer 2020 was focused on string functions)
• tests for remaining SQL clauses -- currently, we have coverage of collections, joins, and the "on duplicate" clause for inserts. we'd need:
  • INSERT ... SELECT statements (more complete, there is some coverage with the on duplicate tests)
  • DELETE statements
  • UPDATE statements (including updating through a join)
  • SELECT with GROUP BY (including two-level/nested aggregates)
  • SELECT with ORDER BY
  • SELECT without a FROM (i.e. selecting constants)
  • query scope collections
  • declared objects, including setting and using declared object properties, on both application scope and query scope
  • subqueries

Centrallix was originally written only supporting ASCII, and cannot be trivially converted to using UTF-8 since there are various places in Centrallix and Centrallix-Lib that make assumptions such as that 1 byte = 1 character. However, UTF-8 compatibility is important for supporting non-Roman scripts and languages.

There is more information about the current state of this project here: https://docs.google.com/document/d/1NvjWJlY7uDPGfuOSWR4nEFxmN4CD_QAznwm30zT0Y7A/edit Essentially, there is various work that has been done or still needs to be done along the lines of...

• Check functions operating on strings to make sure they're not making ASCII-related assumptions (such as 1 byte = 1 char).
• Test performance of any modified string functions.
• Validate strings entering Centrallix and make sure they are in correct UTF-8 format.
• Convert HTML pages to declare a UTF-8 charset.
• Convert database to using UTF-8.
• Fix misc bugs introduced by charset changes (like one with SVGs).
• There is also other work in the UTF-8 branch I can't immediately identify the purpose of, so there is probably some other stuff going on that I'm not aware of...