cloudius-systems / osv-apps Goto Github PK

iperf-zcopy fails to build because both zcopy-rx.diff and zcopy-tx.diff patches fail to apply.
The reason they fail to apply is because the iperf source files contain trailing whitespaces while the patches don't. As I see there are 2 solutions:

1. Add missing trailing whitespaces to the patches (though I wouldn't call something that adds useless characters at end of lines a clean solution)
2. A more elegant solution is, IMHO, to remove trailing whitespaces from the files to be patched before applying the patches. I'll submit a pull request with this change shortly

Right now, "make image=rogue" creates an image with rogue, but also Java, tests/*, and a whole lot of other irrelevant stuff.

We should make "tests" and "java" modules of their own, and for example have the "default" module depend on "tests", the "mgmt" module depend on "java", and similarly the "tomcat" module depends on "java" - but "rogue" does NOT depend on java.

OSv v0.09
eth0: 192.168.122.15
page fault outside application, addr: 0x0000000000000000
[registers]
RIP: 0x000000000041cf65 <strlen+37>
RFL: 0x0000000000010246 CS: 0x0000000000000008 SS: 0x0000000000000010
RAX: 0x0000000000000000 RBX: 0x0000100000713820 RCX: 0x00002000001ffb5c RDX: 0x00002000001ffb60
RSI: 0x00002000001ffb68 RDI: 0x0000000000000000 RBP: 0x00002000001ffb80 R8: 0x00002000001ffa68
R9: 0x00002000001ffa64 R10: 0x0000000000000019 R11: 0x0000000000000024 R12: 0xffffc0003eaff8c0
R13: 0x00002000001ffa64 R14: 0x0000000000000019 R15: 0x0000000000000024 RSP: 0x00002000001ffb80
Aborted

[backtrace]
0x0000000000320f0c <mmu::vm_sigsegv(unsigned long, exception_frame*)+76>
0x0000000000320fa2 <mmu::vm_fault(unsigned long, exception_frame*)+146>
0x0000000000372ea9 <page_fault+105>
0x0000000000371de6 <ex_pf+34>
0x0000100000226263 <set_var_init()+132>
0x000010000021acfd <init_global_datetime_format(enum_mysql_timestamp_type, st_date_time_format**)+1366>
0x000010000021bc31 <main+123>
0x00000000003d599f <osv::run(std::string, int, char**, int*)+255>
0x00000000003d5daf <osv::run(std::string, std::vector<std::string, std::allocatorstd::string >, int_)+575>
0x000000000020ed75 <run_main(std::vector<std::string, std::allocatorstd::string > const&)+245>
0x0000000000210a2f <do_main_thread(void_)+415>
0x0000000000420475 <sync+69>
0x00000000003bfd9a <thread_main_c+26>
0x0000000000372d65 <thread_main+7>

A drop in replacement for Cassandra - written in C++ for speed and efficiency and scale. http://www.scylladb.com

Is anyone working on osv-hdfs? I am interested in building a unikernel image for hdfs, is there a reference available to jump start?

Server seeds are a subset of Casandra nodes which every new nodes go to to get cluster info.
Seeds are contacted when joining the ring, no other communication with seeds is necessary afterwards
All nodes should have the same seed list.

As such the IP of the seeds server need to be configure in all servers cassandra.yaml: (from [2])

seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
    parameters:
         - seeds:  "110.82.155.0,110.82.155.3"

Datastax provides a reflector service.
Each node register to the service with a with a unique cluster id, and after some time, get the details of the other nodes in the cluster.
With the number of nodes in the cluster (from cloud-init), this is enough to set seeds.

Here is where datastax AMI interact with the reflector:
https://github.com/riptano/ComboAMI/blob/2.5/ds2_configure.py#L445
It loops till the number of server in the response match the number of cluster in the service.
The last is a parameter in the cloud-init

[2] http://www.datastax.com/documentation/cassandra/1.2/cassandra/initialize/initializeSingleDS.html

On fedora 25 with gcc 6.3:

scripts/build mode=debug image=iperf nfs=false
...
make[3]: Entering directory '/home/justin_cinkelj/devel/mikelangelo/osv-t1/osv/apps/iperf/iperf-2.0.5/compat'
if g++ -DHAVE_CONFIG_H -I. -I. -I..  -I../include -I../include  -Wall -fPIC  -MT delay.o -MD -MP -MF ".deps/delay.Tpo" -c -o delay.o delay.cpp; \
then mv -f ".deps/delay.Tpo" ".deps/delay.Po"; else rm -f ".deps/delay.Tpo"; exit 1; fi
In file included from /usr/include/c++/6.3.1/cmath:42:0,
                 from /usr/include/c++/6.3.1/math.h:36,
                 from ../include/headers.h:85,
                 from ../include/Timestamp.hpp:63,
                 from delay.cpp:53:
/usr/include/c++/6.3.1/bits/cpp_type_traits.h:205:12: error: redefinition of ‘struct std::__is_integer<int>’
     struct __is_integer<int> /*---------------------------------------------------------------*/
            ^~~~~~~~~~~~~~~~~
/usr/include/c++/6.3.1/bits/cpp_type_traits.h:138:12: error: previous definition of ‘struct std::__is_integer<int>’
     struct __is_integer<bool> /*------------------------------------------------------------------------------*/
            ^~~~~~~~~~~~~~~~~~
Makefile:273: recipe for target 'delay.o' failed

Problem comes from

justin_cinkelj@jcpc:~/devel/mikelangelo/osv-t1/osv/apps/iperf$ less iperf-2.0.5/config.h
/* The size of `bool', as computed by sizeof. */
#define SIZEOF_BOOL 0
...
#define bool int

justin_cinkelj@jcpc:~/devel/mikelangelo/osv-t1/osv/apps/iperf$ less iperf-2.0.5/config.log
configure:6939: checking size of bool
configure:6944: gcc -o conftest  -fPIC    conftest.c  >&5
conftest.c: In function 'longval':
conftest.c:98:57: error: 'bool' undeclared (first use in this function)
 static long int longval () { return (long int) (sizeof (bool)); }

I noticed that orig iperf 2.0.5 does not exit VM once done. Some threads are left over, and some extra work is required to get them terminated.

OSv - master at

commit 4a716433ceca860b9b94b24384b96cd45ea32b01
Author: Nadav Har'El <[email protected]>
Date:   Mon Aug 21 14:48:56 2017 +0300

Host:

justin_cinkelj@jcpc:~/devel/mikelangelo/osv-t1/osv$ iperf -s

OSv:

justin_cinkelj@jcpc:~/devel/mikelangelo/osv-t1/osv$ sudo ./scripts/run.py -nv -d -e '/tools/iperf -c 192.168.122.1 -t0.1 '
OSv v0.24-432-g4a71643
eth0: 192.168.122.90
------------------------------------------------------------
Client connecting to 192.168.122.1, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 192.168.122.90 port 19756 connected with 192.168.122.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 0.1 sec   151 MBytes  12.4 Gbits/sec

gdb:

 244 (0xffff8000037dd040) dhcp            cpu1 status::waiting dhcp::dhcp_worker::dhcp_worker_fn() at core/dhcp.cc:830 vruntime  8.70543e-22
 247 (0xffff800003a3c040) >/tools/iperf   cpu0 status::waiting waiter::wait(sched::timer*) const at include/osv/wait_record.hh:47 vruntime  6.83171e-21
Number of threads: 139

(gdb) osv thr 247
(gdb) bt
#0  sched::thread::switch_to (this=0xffff8000037e3040) at arch/x64/arch-switch.hh:75
#1  0x00000000005b6f10 in sched::cpu::reschedule_from_interrupt (this=0xffff8000017de040, called_from_yield=false, preempt_after=...) at core/sched.cc:339
#2  0x00000000005b67cc in sched::cpu::schedule () at core/sched.cc:228
#3  0x00000000005ba547 in sched::thread::wait (this=0xffff800003a3c040) at core/sched.cc:1214
#4  0x00000000005649d4 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=..., pred=...) at include/osv/sched.hh:1063
#5  0x000000000056475a in sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1074
#6  0x0000000000564722 in waiter::wait (this=0x2000003ffe90, tmr=0x0) at include/osv/wait_record.hh:47
#7  0x0000000000563751 in condvar::wait (this=0xffffa000038d9700, user_mutex=0xffffa00002ee0440, tmr=0x0) at core/condvar.cc:43
#8  0x000000000069fc75 in pthread_cond_wait (cond=0x100000e19840 <ReportCond>, mutex=0x100000e19870 <ReportCond+48>) at libc/pthread.cc:593
#9  0x0000100000c0d465 in reporter_spawn ()
#10 0x0000100000c132c9 in thread_run_wrapper ()
#11 0x000000000069ebf7 in pthread_private::pthread::<lambda()>::operator()(void) const (__closure=0xffffa00002d8b800) at libc/pthread.cc:114
#12 0x00000000006a1704 in std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/6.3.1/functional:1731
#13 0x0000000000443fbc in std::function<void ()>::operator()() const (this=0xffff800003a3c070) at /usr/include/c++/6.3.1/functional:2127
#14 0x00000000005ba528 in sched::thread::main (this=0xffff800003a3c040) at core/sched.cc:1208
#15 0x00000000005b63fc in sched::thread_main_c (t=0xffff800003a3c040) at arch/x64/arch-switch.hh:170
#16 0x0000000000481553 in thread_main () at arch/x64/entry.S:113

So after main() returns, one app thread is left, and VM doesn't terminate - app is still considered to be up. Correct?

I think there is an issue in how OSv combines commandlines - this method is not immune to special characters. When creating an image made of multiple parts, such as image=python3x,httpserver,mysql, --env=TERM=unknown is treated as a flag for the previous program in the image. Creating an image made of these parts in a different order, image=httpserver,python3x,mysql, works just fine:

The fact that python3's output is shown before mysql'd output doesn't affect functionality. Python3x uses --env=TERM=unknown as part of it's cmdline.

The nginx app runs failed with the error: 'fork() stubbed, nginx:[emerg] fork() failed'

There is a patch to replace fork in src/os/unix/ngx_process.c while another fork is still in src/os/unix/ngx_daemon.c. So, maybe the fork in ngx_daemon.c need to be replaced too.

Hi folks,

The ruby app specifies 2.2.0, however, after building, there's a runtime symbol lookup failure for "__gmp_version"

Trying Ruby 2.1.5 yields "failed looking up symbol ruby_sysinit"

Cheers,

Rowland

For OSv virtual appliances, it will be useful to provide application level information in addition to the OSv level.
Redis for example, can provide information which will allow building a Redis tab in the GUI
cloudius-systems/osv-gui#37

Now that we have perl, I would like to add cado, which is a general purpose code-generation language based on perl:

• cado - will provide cado-base (from perl-base)
• cado-test - test suite for cado (from cado-base)

The cado-test suite provides an good aplication level test suite for perl-base.

The cado project is hosted on github and sourceforge:

• https://github.com/russt/cado
• http://sourceforge.net/projects/cado

iperf output is not always in Mbps.

It's also possible to be kbit, and Gbit/second.

https://github.com/cloudius-systems/osv-apps/blob/master/iperf/jenkins/iperf-xml.py#L44

Creating issue to track progress on Perl port to OSv.

specjvm does not provides module.py and default cmdline to run. Please consider add a default cmd to run. So

$ make image=specjvm
$ scripts/run.py

will work.

When we can support running java.so multiple times. We can even run all the benchmark by default one by one.

I run:
sudo capstan run
Created instance: node-express-example
OSv v0.23
eth0: 10.0.2.15
Waiting for connection from host...

I noted that the eth0 interface does not use any of my IP addresses and localhost:3000 does not give any response.
How do I change so that vbox uses localhost?

When I build my image with
./scripts/build image=openjdk8-zulu-full,tomcat,httpserver-html5-cli

I get the following error message at launch:

Failed to load object: /libhttpserver.so. Powering off.

This particular .so is nowhere to be found in in the directory tree.

Cassandra fails to load JNA:

INFO 08:26:31,098 JNA link failure, one or more native method will be
unavailable.

which causes startup to fail as follows when starting Cassandra on an
image where it has already been run (and has data files):

ERROR 08:26:34,525 Unable to create hard link with exec.  Suggest
installing JNA to avoid the need to exec entirely.  Exception follows:
java.io.IOException: Cannot run program "ln": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
at org.apache.cassandra.utils.FBUtilities.exec(FBUtilities.java:556)
at org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:202)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:165)
at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:76)
at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1002)
at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1566)
at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1611)
at org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.java:1798)
at org.apache.cassandra.db.DefsTable.fixSchemaNanoTimestamp(DefsTable.java:215)
at org.apache.cassandra.db.DefsTable.fixSchemaNanoTimestamps(DefsTable.java:163)
at org.apache.cassandra.db.SystemTable.finishStartup(SystemTable.java:97)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:336)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:441)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:484)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at io.osv.RunJava.runMain(RunJava.java:153)
at io.osv.RunJava$1.run(RunJava.java:124)
at io.osv.OsvSystemClassLoader$1.run(OsvSystemClassLoader.java:70)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
... 21 more
ERROR 08:26:34,683 Exception encountered during startup

If you look at images/{cassandra,rogue,tomcat}.y, they only say they contain themselves. This file should only be necessary if it needs to specify something non-obvious.

But more importantly, why is images/*.py in the main OSv repository? It should be in the apps repository.

Config.json should also be in the apps repository.
We can read both osv/config.json and osv/apps/config.json, to allow both Takuya's original method and the new apps/ method to work.

hi,

https://github.com/cloudius-systems/osv-apps/blob/master/iperf/jenkins/iperf-xml.py#L44
"mesure" --> "measure"

Java heap size should be extracted from the VM size

When using the python3x image, one of the big no-gos is import sqlite3. Because it is a standard library module, it is present in every python3x image. However, importing it directly or indirectly causes OSv to crash:

I needed sqlite to work after finding out django would insist on importing sqlite even when using a different database as the main data storage. I struck gold when I realized that OSv-apps contained a working sqlite3 application, already compiled as a shared library sqlite.so. So I decided to see if they would be compatible... and they were! Here are the steps I took:

1. Update sqlite ot the newest version by changing parameters in sqlite's GET to
   VERSION=3270200 and YEAR=2019,
2. Build an sqlite image
3. Build a python3x,httpserver image
4. Upload /apps/sqlite/sqlite.so to OSv with the httpserver, placing it as /usr/lib/libsqlite3.so.0
5. Import sqlite on python3 successfully.

I was wondering what the best way to upstream this python improvement? For personal use, I added
rsync -a $BASEDIR/../sqlite/sqlite.so $ROOTFS/usr/lib/libsqlite3.so.0 at the end of main() in python3x's GET, which works just fine. For upstream use, we probably shouldn't assume sqlite has been build before python was. I tried using api.require in module.py but it only seems to accept a single argument...

I would be happy to write a PR fixing sqlite in python3x, but I need some help figuring out the proper way to do this. In the meantime, I'd be happy to upstream the version update to sqlite since that seems to work just fine, but I've only tried a few functions from the sqlite shell outside using it in a python project, so I'm not confident it is a non-breaking update.