flhofer / real-time-containers Goto Github PK

Implement the resource planning algorithm as separate functions-> function variables, to allow quick exchange

This could also include thread capabilities ecc.
http://man7.org/linux/man-pages/man7/pthreads.7.html

Use real-time execution data to predict periodicity (if available) of FIFO or RR based tasks, to better fit resources.

The estimated period has to be approximated to find a better fit as it usually features numbers in nanoseconds.

Implement library instead of my internal parser lib

Negative PIDs in list may cause clash after all numbers are though (long running experiments)
Move negative PIDs to the end of the list and break at the first occurrence <0. However, I need a guard for the list to not get too big.

Test environment and try to build use case code

There is apparently no return status/error code for the commands involved.
To verify.

Implement return object to allow better diagnose of runtime
use pthread_exit for this purpose

Running tests with -fdk -S1 makes the process not react to SIGINT and it must be killed.
Running them without -k doesn't show the issue.

) + got
could cause data shift.

The resulting allocation does not match the test:

Implement and test with test 2 for resource scheduling, for use case testing

if startup is stopped, the system might remain in a state where many CPUs are disabled and configuration is err. Fix with startup option to force configuration reset.

For example, here..

... Trying to push CPU's interrupts
... CPU7 off-line
... CPU6 off-line
... CPU5 off-line
... CPU4 off-line
... CPU3 off-line
... CPU2 off-line
... CPU1 off-line
... CPU1 online
... CPU2 online

If this phase is halted, the software assumes 2 CPUs only at next startup

Change PID resource settings to allow CPU ranges as affinity value, significant for containers and images.

Gather information on I/O behavior of tasks to allow inferring on the rate of interruption and how instable a task may be.
These statistics should be updated in real-time and help to group tasks with similar behavior to different resources.

the download path of the patch is not correct for older patches, the reference config of the system should also be updated. Using 4.4.x

Fix test outcome for JSON tests. The new setup works differently and causes tests to fail

actually this defines NUMA

	char * numastr = "0"; // default numa string
	if (-1 != numa_available()) {
		int numanodes = numa_max_node();
		if (!(numastr = malloc (5)))
			err_exit("could not allocate memory!");

		sprintf(numastr, "0-%d", numanodes);
	}
	else
		warn("Numa not enabled, defaulting to memory node '0'");

Extend to find NUMA nodes and corresponding association

Review format as in dockerlink.c

#define PFX "[dockerlink] "
#define PFL "         "PFX
#define PIN PFX"    "
#define PIN2 PIN"    "
#define PIN3 PIN2"    "

Investigate there reason for this duplicate output. Either label or printing error

Review library function definitions.

• resource structures
• simple allocation tests
• mapping of hierarchical masks to resource plan
• test case resource plan allocation
• execution of resource plan
• additional variation tests for allocation
• error with overlapping resources

this is a prerequisite for #4

Improve monitoring resolution by changing to fTrace kernel monitoring.
fTrace functions are present but a parser is needed.

• Implement fTrace setup and call
• fTrace parsing of values
• implementation of statistics
• speed test and reliability comparison

The number of threads gets high, so I might need to change the configuration of threads.

First:

• cleanup sigaction and sigblock for all threads

• simplify thread commands to 1-2 signals and volatile, atomic variables
  then:

• consider pthread_cancel and pthread_cleanup for CPU monitoring threads

• possible review libtracer for invocation hooks. Relates to #3.

• review external/plugin relation for, e.g., dockerlink.

Add NUMA options to the slice configuation, including memory sizes and limits

• NUMA as Cpu assignments
• Memory allocation management in configuration
• Memory limit settings per slice
• JSON setting tests and validations

Consider multithreading for cores, dividing maximum load through threads, but still using the advantage of the faster pipes

Apply some performance fixes such as comment.char = "#", fore read.table to avoid further processing

Implement automatic build and deploy with actions

• integrate dependencies in repo/build script (downloads)
• create CI pipeline for build and test
• create AWS instance with use case and orchestrator
• test access to AWS instance/ ssh keys?
• create test run scripts
• deploy binaries automatically with actions

depends on #14

investigate issues, ex manage test, it goes in timeout for a sleep command when forking is enabled (maybe a function fature? )

Implement period based algorithm for adaptive/dynamic type resource allocation.

matching preferred, else lcm is preferred.

Important when many are disabled from the beginning, the max number may avid some parts of the Mask to be processed.

To ease graph readability, recheck the following points for data evaluation

• Data overlapping and comparison. Merged worker results should be summed not an overlapped print
• Add feature to script to rework all test result directories
• Add frame latency results to stats.csv output for both test batches
• Test and implement overlapping graphs, to better represent a group of results in one view
• Check for Tikz output from R data for vector scalability in PDF
• Review bug messages of script

Depends on #18

Write a parser to display the performance data of the use cases. Preferably in R, sh, or python

Printing formats have been revised, and use cases are now ready for review on bm

• create parser for UC1 (event - FIFO)
• create parser for UC2 (poll - DL)
• create parser for data generator
• create parser for data distributor
• implement newest test parsing only -> print to STDOUT
• implement combined metric print / out

recon all (void)sprintf and use return value (len) in functions for doublecheck

The scripts use folder variables starting from root. A glitch in the assignment caused 2 systems to be wiped out (running rm -f $variable/*)

Extend -D dry-run to enable demo mode for managing and rescheduling, ignoring the preparation of the environment which would require system access

Add support for Round robin and fifo tasks. At the moment only DL tasks are supported

Add description to .8 manual and readme

Reorganize test modules so to avoid duplicate namespaces.

• create header files and connect them into a suite
• create includes for common functions, such as config load
• update makescript to build all c functions to .o automatically

both perform a similar function. Investigate and fix

Implement functions to change CPU/MEM assignments of slice automatically so to be able to claim or release resources dynamically when needed, changing the burden to the system
No best effort influence this way

Migrate the functions to static/dynamic library to enable easy replacement once v1 is no longer supported

Can not create threads if no sudo is used. Thread creation fails with error EAGAIN

ERROR: could not start update thread: Resource temporarily unavailable
ERROR: could not start management thread: Resource temporarily unavailable

create a unique make file or, include make process of use cases, depends on #2

Combine as follows for a single memory structure, avoiding casts

	typedef struct c_parm {
		struct c_parm *next; 
		char *id;  // matching signatures -> Psing, Image or container IDs
		struct sched_attr *attr;// standard Linux PID attributes
		struct sched_rscs *rscs;// additional resource settings 
		union {
			struct c_parm  *cont;// pointer to the container settings, PID
			struct cs_parm  *pids;// linked list pointing to the pids, Container and Image	
		}
		union {
			struct c_parm   *img; // pointer to the image settings. PID and Container
			struct cs_parm *conts;// linked list pointing to the containers, Image
		}
	} cparm_t;

	typedef struct cs_parm {
		struct cs_parm *next; // list to next entry
		struct c_parm *parm; // matching entry
	} csparm_t;

When combining data generator and poll with event driven workers, the latter starve to death
Has to do with settings or misconfiguration

print formats seem scrambled. Even after the high amount of warnings has been removed (883d7e2), the print formats do not seem certain

change folds and filenames of schedstat to main/orchestrator to enable branch and file-compare throughout versions and history.

this is a prerequisite for #14!

Combine inline functions code for JSON_OBJECT_*

Try to setup scripts and configuration to be able to run most of the tests and simulations in user level. CAP - SYS_NICE fore a process, how to give that to a container without being root all the script? sudo?