Giter VIP home page Giter VIP logo

nanovms / nanos Goto Github PK

View Code? Open in Web Editor NEW
2.5K 33.0 132.0 9.84 MB

A kernel designed to run one and only one application in a virtualized environment

Home Page: https://nanos.org

License: Apache License 2.0

Makefile 1.75% Assembly 1.12% C 95.67% GDB 0.01% Shell 0.52% Go 0.43% JavaScript 0.01% HTML 0.01% PHP 0.01% Ruby 0.01% Rust 0.03% Python 0.42% Awk 0.01% C++ 0.02%
osdev security virtualization unikernel microservice edge operating-systems sandbox unikernels

nanos's Introduction

nanos

CircleCI

Nanos is a new kernel designed to run one and only one application in a virtualized environment. It has several constraints on it compared to a general purpose operating system such as Windows or Linux - namely it's a single process system with no support for running multiple programs nor does it have the concept of users or remote administration via ssh.

Read more about the Nanos Charter here.

  1. Getting Started
  2. Documentation
  3. Support

Getting Started

Please use the ops build tool to run your applications with Nanos unless you plan on hacking on Nanos itself. Many ready-to-use examples for running applications on Nanos using ops are available here.

Quick Start

Install ops && nanos:

curl https://ops.city/get.sh -sSfL | sh

Demo

Building/Running

It is highly encouraged to use ops to build and run your applications using Nanos unless you are planning on modifying Nanos. ops provides sensible defaults for most users and incorporates our understanding of how to appropriately best use Nanos. It is also currently highly coupled to Nanos.

If you are running in a vm already (which is a bad idea) you'll need to specify that you don't want hardware acceleration. For instance you can run Nanos in virtualbox on a mac but it will be slow and hard to configure.

You can build and run on mac and linux. Nanos supports KVM on linux and HVF on osx currently for acceleration. ops has facilities to deploy to public clouds (AWS, GCE, Azure, and many others).

Building From Source on a Mac

Note: This is only recommended for those that wish to make code changes to Nanos itself.

Install Homebrew:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Grab Dependencies:

This installs the correct toolchain and will install an up-to-date qemu. It is highly recommended to be running the latest qemu otherwise you might run into issues.

For Intel-based Macs:

brew update && brew install nasm go wget ent
brew tap nanovms/homebrew-toolchains
brew install x86_64-elf-binutils
brew tap nanovms/homebrew-qemu
brew install nanovms/homebrew-qemu/qemu

For ARM-based Macs (M1/M2):

brew update && brew install go wget ent qemu aarch64-elf-gcc aarch64-elf-binutils
# To build and link runtime tests or aarch64 linux user programs:
brew tap nanovms/homebrew-toolchains
brew install aarch64-linux-binutils

Create a Chroot: (this isn't absolutely necessary)

For Intel-based Macs:

mkdir target-root && cd target-root && wget
https://storage.googleapis.com/testmisc/target-root.tar.gz && tar xzf target-root.tar.gz

For ARM-based Macs (M1/M2):

mkdir target-root && cd target-root && wget
https://storage.googleapis.com/testmisc/arm64-target-root-new.tar.gz && tar xzf arm64-target-root-new.tar.gz

You should also set the environment variable NANOS_TARGET_ROOT to the path of target-root created above in order to create the example and test images.

Building From Source on Linux

Note: This is only recommended for those that wish to make code changes to Nanos itself.

Nanos doesn't need too many dependencies on Linux.

To build you need to install nasm && qemu:

sudo apt-get install qemu-system-x86 nasm

For tests you'll also need the following installed:

sudo apt-get install ent ruby

Rust:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

If you wish to use FUSE you'll also want to install

sudo apt-get install libfuse-dev fuse

To build:

make run-noaccel

Documentation

You can find more documentation on the ops docs site

Benchmarks

In an effort for transparency and to collect, categorize, and document benchmarks we will start listing them here. The aims should be to be as reproducible and contain as much information as possible:

Go on GCloud: 18k req/sec

Rust on GCloud: 22k req/sec

Node.JS on AWS: 2k req/sec

Node.JS on GCloud: 4k req/sec

Tests

To run tests:

make test-noaccel

Development Running Instructions

For Nanos try running the first example first:

make run

To try a different target currently found in test/runtime/ you can:

  1. cp the manifest file to target.manifest
  2. add your code and set a target in test/runtime/Makefile
make TARGET=mynewtarget run

You may also wish to use ops to develop locally. If that's the case a commonly used idiom is to simply copy the 3 required files to an appropriate release:

cp output/tools/bin/mkfs ~/.ops/0.1.47/.
cp output/platform/pc/boot/boot.img ~/.ops/0.1.47/.
cp output/platform/pc/bin/kernel.img ~/.ops/0.1.47/.

Contributing

Pull Requests

We accept pull requests as long as it conforms to our style and stated goals. We may reject prs if they violate either of these conditions.

If you are planning on spending more than a day to fix something it's probably wise to approach the topic in an issue with your planned fix before commiting to work.

Also, NanoVMs has paid kernel engineers with internal roadmaps so it's wise to check in with us first before grabbing a tkt. Tickets tagged 'low-priority' have a lower probability of collision.

Reporting Bugs

Please scan the issue list first to see if we are already tracking the bug.

Please do not create a new issue on a closed issue. If you think you are experiencing the same problem (is the error message the same?) you can open a new issue and link to it if you think it's the same.

Please try to provide the most basic reproducible example you can. If you have an error message or dump please include that.

Please try using the latest/nightly build first before reporting.

Please attach debugging output (--trace in ops). Please provide the config.json and anything else that allows us to reproduce the issue.

TFS

TFS is the current filesystem utilized by Nanos.

Optional Manifest Debugging Flags

thread tracing:

futex_trace: t

syscall tracing:

debugsyscalls: t

Read more about Security here.

Architecture

Debugging Help

Manual Networking Setup

Build Envs

Reference Materials

Feedback

How are you using Nanos? Take this very short survey and help us grow the project. It will take just a minute but help us out!

Feedback Form

Who is Using Nanos?

If you are using Nanos for a project at your company or organization feel free to open up a PR and list your project/company below.

Getting Help

If you need help using nanos or ops the the discussion forum is your best route for general questions. You may feel free to report a bug noting the bug reporting section ahead or open an issue for a feature request.

Support

Note: Nanos is open source, Apache2, however the builds that NanoVMs provides are not. Those using binary builds with over 50 employees must get a commercial subscription or you may freely build from source.

Similarily if you need something done now or want immediate attention to an issue NanoVMs offers paid support plans.

If you need email or drift support you will need to sign up for a support plan.

If you'd like more in-depth help reach out to the nanovms folks via drift or email engineering.

nanos's People

Contributors

billhuey avatar briankoco avatar cdosoftei avatar convolvatron avatar crazoes avatar crodnun avatar dpreilan avatar erectcrested avatar eyberg avatar fabiodmferreira avatar francescolavra avatar levex avatar mkhon avatar mkwal avatar nsharma1231 avatar praveenganapathy19 avatar quantumsheep avatar repu1sion avatar richiejp avatar rinor avatar roylee17 avatar rvoynov avatar sanderssj avatar sanketsudake avatar skupr avatar talshadow avatar tijoytom avatar uma-sethuraman avatar wjhun avatar x64k avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nanos's Issues

lwip may not be advancing receive window

there is a comment at the end of netsyscall.c:read_complete

// tcp_recved() to move the receive window

gdb input does call this function after consuming the payload, but there is a comment
there

// not necessarily

verify that this is the right thing to do and do it

fix backtrace

minor enhancement for debugging. one more step past this would get us a proper backtrace.

interrupts and tx completion flakey

there clearly remains a race, much more pronounced or possibly exclusively manifest in qemu 2.5

symptoms include silently ignored transmit request, and unwillingness to post receive interrupts.

most likely an issue with order of initialization, but could really be anything

support qSymbol in gdb

once #18 is done, wire that up into gdb.
this looks like it will let us do symbolic resolution without loading the objects into gdb

mmapped files aren't consistent - implement page fault handlers

intitially we treat an mmap on a file as a read...we should (for some value of should),

  • keep track of virtual regions (in the range tree)
  • allow registration of read and write handlers
  • wire up the read handler to demand-fill
  • wire up the write handler to stage a flush at some point (we want to
    window these to avoid doing a synchronized flush for every write)

loader assumes <= 128k elf file

the long term solution for this issue is to fold stage2 into stage1 so the load exactly match what is needed for the segments.

however, since the extended read call under qemu fills it what it can before returning an error, and the elf image defines the length of the 'drive', we can really just read contiguously until error. need to fix the reporting of the allocated base to stage2 and the application

human tuple syntax should be more readable for filesystems

finish the work in the tuple parser to allow for infix operators that make it easier to talk about
filesystem paths, and finish/fix quoted strings. also reinstitute the functionality of having
mkfs be able to use immediate file contents if it doesn't already.

parallel make causes simultaneous contgen builds and failure

Probably a good thing to fix along with header deps. Seems that including the runtime Makefile in various parts of the build leads to contention.

Solution should be to build contgen once for the whole tree.

$ make -j8
make -f image.mk image
cd test ; make
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
make[1]: Entering directory '/share/src/uniboot-memory2-test'
cd boot ; make
make[1]: Entering directory '/share/src/uniboot-memory2-test/test'
cc -std=c99 /share/src/uniboot-memory2-test/test/../runtime/contgen.c -g -o /share/src/uniboot-memory2-test/test/../runtime/contgen
make[2]: Entering directory '/share/src/uniboot-memory2-test/boot'
nasm -felf service32.s
cc -std=c99 /share/src/uniboot-memory2-test/boot/../runtime/contgen.c -g -o /share/src/uniboot-memory2-test/boot/../runtime/contgen
/share/src/uniboot-memory2-test/test/../runtime/contgen 10 10 > /share/src/uniboot-memory2-test/test/../runtime/closure_templates.h
/bin/sh: 1: /share/src/uniboot-memory2-test/test/../runtime/contgen: Text file busy
/share/src/uniboot-memory2-test/test/../runtime/Makefile:9: recipe for target '/share/src/uniboot-memory2-test/test/../runtime/closure_templates.h' failed
make[1]: *** [/share/src/uniboot-memory2-test/test/../runtime/closure_templates.h] Error 2
make[1]: Leaving directory '/share/src/uniboot-memory2-test/test'
Makefile:14: recipe for target 'test' failed
make: *** [test] Error 2
make: *** Waiting for unfinished jobs....

write a tool to augment a manifest from ldd output

no reason people should have to suffer tracking down dynamic dependencies

yuri@eclair:~/uniboot$ ldd examples/hw
linux-vdso.so.1 => (0x00007ffe0e320000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3a6c560000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3a6c92a000)

qemu disk setup requires us to use 2 files (image/image2)

one file for the bios boot device and one for the virtio driver. that seems pretty
ridiculous, but the qemo options/config are all over the map, and I've wasted
alot of time trying to get a magic configuration that works. this would make it
easier to throw around and run images, but this might get mitigated by the
wrapper framework

connect() is a stub

fill in interaction with epoll(), blocking completion, downcall to allocate lwip state and opcall to handle connect, timeout, and explicit error, feed unix errno up appropriately

argv off by one

mono interprets argv[0] as the first arg. is this a libc thing or a staring frame thing

tfs should compress log

if the number of removals gets to be a sufficiently large ratio of the log space, the
kernel should start a background process to write the in-memory version
(where the creates and removal have already been reconciled) to new segments.

the root of the log should be (atomically) replaced with a single sector containg
a next record pointing to the new log. this clearly depends on #20

if the disk is exhausted, then it may be impossible to regain this space because of the
lack of working area.

virtual memory not tracked

there are tens of little regions floating around, with no real guarantee that they are disjoint.

keep a global map of virtual allocations with some semantic annotations.

use this to generate a proper /proc/self/maps which pthreads is using to locate
the stack

can also add a debugging aid - what was this address mapped for

add byte file length to tls metadata

how embarrassing, the length of the file is being reported as the maximum covered extent...but
files actually have arbitrary byte length. add a metadata property for the actual written
length of the file

functions in filesystem

need to come up with a filesystem api that allows for a union fs root with functional handlers

memory size isn't reported to application

we look for the e820 region, but don't try to pass it to the user. it contains several segments, the most important being the ~4G before the 'pci gap' and what should be flat memory afterwards.

we could most likely move everything out of the pci gap, but we'd have to adjust the mtrrs and its really not worth it.

so add multi-segement reporting, and look at the 64 bit e820 extensions...or* maybe the virtio balloon interface lets just get the initial memory status - that would be a lot cleaner

gcc 6.3.0 internal compiler error when building mkfs.c

In a fresh filesystem branch:

wjhun@tiger:/share/src/uniboot$ cc --version
cc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

wjhun@tiger:/share/src/uniboot$ make
make -f image.mk image
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
make[1]: Entering directory '/share/src/uniboot'
cd boot ; make
make[2]: Entering directory '/share/src/uniboot/boot'
nasm -felf service32.s
cc -std=c99 /share/src/uniboot/boot/../runtime/contgen.c -g -o /share/src/uniboot/boot/../runtime/contgen
/share/src/uniboot/boot/../runtime/contgen 10 10 > /share/src/uniboot/boot/../runtime/closure_templates.h
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime stage2.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../x86_64/serial.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../x86_64/page.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../x86_64/elf.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/heap/zero_wrap.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../x86_64/kvm_platform.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../tfs/tlog.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../tfs/tfs.c -c
In file included from /share/src/uniboot/boot/../runtime/runtime.h:126:0,
from /share/src/uniboot/boot/../tfs/tfs_internal.h:1,
from /share/src/uniboot/boot/../tfs/tfs.c:1:
/share/src/uniboot/boot/../tfs/tfs.c: In function ‘fs_read_extent’:
/share/src/uniboot/boot/../tfs/tfs.c:28:85: warning: passing argument 5 of ‘fs->r’ from incompatible pointer type [-Wincompatible-pointer-types]
apply(fs->r, buffer_ref(target, i.start), range_span(i), u64_from_pointer(val), f);
^
/share/src/uniboot/boot/../runtime/closure.h:5:40: note: in definition of macro ‘apply’
#define apply(__c, ...) (
__c)(__c, ## VA_ARGS)
^~~~~~~~~~~
/share/src/uniboot/boot/../tfs/tfs.c:28:85: note: expected ‘status_length_handler {aka void ()(void *, long long unsigned int, struct table *)}’ but argument is of type ‘status_handler {aka void ()(void *, struct table )}’
apply(fs->r, buffer_ref(target, i.start), range_span(i), u64_from_pointer(val), f);
^
/share/src/uniboot/boot/../runtime/closure.h:5:40: note: in definition of macro ‘apply’
#define apply(__c, ...) (
__c)(__c, ## VA_ARGS)
^~~~~~~~~~~
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../tfs/merge.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/table.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/buffer.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/format.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/heap/id.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/symbol.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/heap/rolling.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/tuple.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/random.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/rtrie.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/runtime_init.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/extra_prints.c -c
cc -m32 -O -fno-stack-protector -include def32.h -I/share/src/uniboot/boot/../x86_64 -I/share/src/uniboot/boot/../runtime -I/share/src/uniboot/boot/../tfs -I. -I../runtime /share/src/uniboot/boot/../runtime/heap/debug_heap.c -c
ld -T linker_script -e _start service32.o stage2.o serial.o page.o elf.o zero_wrap.o kvm_platform.o tlog.o tfs.o merge.o table.o buffer.o format.o id.o symbol.o rolling.o tuple.o random.o rtrie.o runtime_init.o extra_prints.o debug_heap.o -o stage2.elf
strip stage2.elf -o stage2.strip
objcopy -S -O binary stage2.strip stage2
dd if=stage2 of=stage2.pad bs=512 conv=sync
81+1 records in
82+0 records out
41984 bytes (42 kB, 41 KiB) copied, 0.000657015 s, 63.9 MB/s
nasm -l stage1.lst -dSTAGE1SIZE=512 -dSTAGE2SIZE=41984 stage1.s -o stage1
cat stage1 stage2.pad > boot
make[2]: Leaving directory '/share/src/uniboot/boot'
cd mkfs ; make
make[2]: Entering directory '/share/src/uniboot/mkfs'
cc -g -c -include def64.h -I/share/src/uniboot/mkfs/../runtime -I/share/src/uniboot/mkfs/../x86_64 -I/share/src/uniboot/mkfs/../tfs mkfs.c
In file included from :32:0:
/share/src/uniboot/mkfs/../x86_64/def64.h: In function ‘format_pointer’:
/share/src/uniboot/mkfs/../x86_64/def64.h:48:5: internal compiler error: in build_va_arg, at c-family/c-common.c:5812
u64 x = varg(a, u64);
^~~
Please submit a full bug report,
with preprocessed source if appropriate.
See file:///usr/share/doc/gcc-6/README.Bugs for instructions.
Makefile:12: recipe for target 'mkfs.o' failed
make[2]: *** [mkfs.o] Error 1
make[2]: Leaving directory '/share/src/uniboot/mkfs'
image.mk:8: recipe for target 'mkfs/mkfs' failed
make[1]: *** [mkfs/mkfs] Error 2
make[1]: Leaving directory '/share/src/uniboot'
Makefile:8: recipe for target 'image' failed
make: *** [image] Error 2
wjhun@tiger:/share/src/uniboot$

musl tls not wired up

it looks* like this probably uses the gs convention, and it is calling a thread state related syscall on main startup. so look at it, and in particular add gs/fs save restore to the generic trap/context machinery

clock isn't normalized if not running on kvm

current clock support uses the kvm clock to calibrate rdtsc. if that fails, things proceed under the assumption that the clock scaling factor is reasonable when in fact it is not. a) throw an error b) implement an alternative normalization scheme (hpet?)

establish linker sets for initialization

  • define a section with a standard format for initializers
  • create a cpp namespace for initializers
  • initializers are a function with a standard signature and a dependency, which create a new name, and depend on an existing name
  • at run time do a toposort of a initializers and run them

initializations would then have explicit dependencies, and allow for fully link-time modularization of things like drivers, etc

application build is dirty

application builds shouldn't require a complicated makefile and parts of the lwip configuration. normalize so that the application build is presented with a single include tree and a small number of archive libraries to link to.

dont flush tlb on every map

this is only important if we change a mapping, which we should rarely if ever do. this becomes more important in the multiprocessor case, since telling everyone to flush is expensive

amd64 also has a per-page tlb flush instruction invlpg

network_test crashes after long run

Now with improved memory re-use, the kernel is outlasting network_test:

Could be we're hitting a limit with brk?

(gdb) r 192.168.42.76:8080
Starting program: /share/src/uniboot-memory-lwip/test/network_test 192.168.42.76:8080

Program received signal SIGSEGV, Segmentation fault.
0x0000555555557ff4 in zero (x=0xffffffffffffffff, length=32) at /share/src/uniboot-memory-lwip/test/../runtime/runtime.h:63
63              ((u8 *)x)[i] = 0;
1: x/i $pc
=> 0x555555557ff4 <zero+34>:    movb   $0x0,(%rax)
(gdb) bt
#0  0x0000555555557ff4 in zero (x=0xffffffffffffffff, length=32) at /share/src/uniboot-memory-lwip/test/../runtime/runtime.h:63
#1  0x0000555555558995 in table_set (z=0x2000003ffd40, c=0x100000000020, v=0x555556e290b0) at /share/src/uniboot-memory-lwip/test/../runtime/table.c:108
#2  0x0000555555556a99 in http_recv (p=0x555556e25340, b=0x555556e28db0) at http.c:115
#3  0x00005555555567aa in _apply_http_recv (z=0x555556e253f0, r0=0x555556e28db0) at http.c:72
#4  0x000055555555775d in connection_input (h=0x555555770010, f=6, e=3, p=0x555556e253f0) at /share/src/uniboot-memory-lwip/test/../unix_process/socket_user.c:82
#5  0x0000555555557659 in _apply_connection_input (z=0x555556e25410) at /share/src/uniboot-memory-lwip/test/../unix_process/socket_user.c:71
#6  0x0000555555557ec6 in epoll_spin (e=3) at /share/src/uniboot-memory-lwip/test/../unix_process/socket_user.c:168
#7  0x000055555555593c in main (argc=2, argv=0x7fffffffdd68) at network_test.c:67
(gdb) info reg
rax            0xffffffffffffffff       -1
rbx            0x100000000020   17592186044448
rcx            0x5      5
rdx            0x0      0
rsi            0x20     32
rdi            0xffffffffffffffff       -1
rbp            0x7fffffffda20   0x7fffffffda20
rsp            0x7fffffffda20   0x7fffffffda20
r8             0x555556e29040   93825018269760
r9             0x0      0
r10            0xffffffff       4294967295
r11            0x246    582
r12            0x555555554be0   93824992234464
r13            0x7fffffffdd60   140737488346464
r14            0x0      0
r15            0x0      0
rip            0x555555557ff4   0x555555557ff4 <zero+34>
eflags         0x10286  [ PF SF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
(gdb) disas /s

[...]

   0x0000555555558945 <+312>:   test   %rax,%rax
   0x0000555555558948 <+315>:   jne    0x555555558873 <table_set+102>
   0x000055555555894e <+321>:   cmpq   $0x0,-0x58(%rbp)
   0x0000555555558953 <+326>:   je     0x555555558a17 <table_set+522>
   0x0000555555558959 <+332>:   movq   $0x20,-0x30(%rbp)
   0x0000555555558961 <+340>:   mov    -0x10(%rbp),%rax
   0x0000555555558965 <+344>:   mov    (%rax),%rax
   0x0000555555558968 <+347>:   mov    (%rax),%rax
   0x000055555555896b <+350>:   mov    -0x10(%rbp),%rdx
   0x000055555555896f <+354>:   mov    (%rdx),%rdx
   0x0000555555558972 <+357>:   mov    -0x30(%rbp),%rcx
   0x0000555555558976 <+361>:   mov    %rcx,%rsi
   0x0000555555558979 <+364>:   mov    %rdx,%rdi
   0x000055555555897c <+367>:   callq  *%rax
   0x000055555555897e <+369>:   mov    %rax,-0x38(%rbp)
   0x0000555555558982 <+373>:   mov    -0x30(%rbp),%rdx
   0x0000555555558986 <+377>:   mov    -0x38(%rbp),%rax
   0x000055555555898a <+381>:   mov    %rdx,%rsi
   0x000055555555898d <+384>:   mov    %rax,%rdi
   0x0000555555558990 <+387>:   callq  0x555555557fd2 <zero>
=> 0x0000555555558995 <+392>:   mov    -0x38(%rbp),%rax
   0x0000555555558999 <+396>:   mov    %rax,-0x40(%rbp)
   0x000055555555899d <+400>:   cmpq   $0xffffffffffffffff,-0x40(%rbp)
   0x00005555555589a2 <+405>:   jne    0x5555555589b5 <table_set+424>

make no-kvm failing with "FATAL ERROR:system clock is inaccessible"

On a fresh master tree:

$ qemu-system-x86_64 -version
QEMU emulator version 2.8.1(Debian 1:2.8+dfsg-6+deb9u4)
Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
$ make run-nokvm
make -f image.mk image
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
make[1]: Entering directory '/share/src/uniboot-test2'
cd boot ; make
make[2]: Entering directory '/share/src/uniboot-test2/boot'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/share/src/uniboot-test2/boot'
cd mkfs ; make
make[2]: Entering directory '/share/src/uniboot-test2/mkfs'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/share/src/uniboot-test2/mkfs'
cd stage3 ; make
make[2]: Entering directory '/share/src/uniboot-test2/stage3'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/share/src/uniboot-test2/stage3'
cd examples ; make
make[2]: Entering directory '/share/src/uniboot-test2/examples'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/share/src/uniboot-test2/examples'
make[1]: Leaving directory '/share/src/uniboot-test2'
cp image image2
qemu-system-x86_64 -boot c -drive file=image,format=raw,if=ide -nographic -m 2G -device isa-debug-exit -drive file=image2,format=raw,if=virtio
create fs
kernel complete
pages heap: 0000000000225000, length 00000000002f8000
physical memory:
base 0000000000800000, length 000000007f600000
FATAL ERROR:system clock is inaccessible
Makefile:46: recipe for target 'run-nokvm' failed
make: [run-nokvm] Error 1 (ignored)

rolling heap doesn't deal with multipage allocations

trying to register 256 file descriptors with epoll:

zero (length=4096, x=0x1001a0000) at ../runtime/runtime.h:61
61 ((u8 *)x)[i] = 0;
(gdb) bt
#0 zero (length=4096, x=0x1001a0000) at ../runtime/runtime.h:61
#1 resize_table (buckets=512, z=0x1000580e3) at ../runtime/table.c:66
#2 table_set (z=0x1000580e3, c=c@entry=0x105, v=0x10019cc90) at ../runtime/table.c:120
#3 0x000000000f01b9cb in epoll_ctl (epfd=, op=, fd=261, event=0x500159c40) at ../unix/poll.c:122
#4 0x000000000f01906a in syscall_debug () at ../unix/unix.c:120
#5 0x000000000f013f34 in syscall_enter ()

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.