• check ltp testcase

• printf 还是正常工作

• getpid 有时候还是正常返回的

• syscall 没有被调用

• 多次调用的结果不相同

• 如果在进入到 dune 之前进行函数调用，那么就是正确运行

  • dune 没有实现要求的地址保持一致: 如果调用过一次，那个跳转就应该被修改掉了

• 为什么 gettimeofday 不是在 vdso 中间的 ?

• 测试 tlbrefill 的方法，分配 1 GB 的空间，然后全部访问一次

I don't know why the guest can't continue in the exception context, it has to be redirected to the next instruction of syscall.

calling a function which demands a huge stack will cause gup fail

void get_writesize(){
  printf("get_writesize\n");
	char read[4096];
  printf("read : %p\n", read);

  char buf[0x3d000];
  printf("buf location %p", buf);
}

➜  test git:(main) ✗ sudo ./stack
  Info: open /dev/kvm
  Info: KVM_GET_API_VERSION
  Info: KVM_CREATE_VM
  Info: KVM_SET_USER_MEMORY_REGION
  Info: ebase address : fff4310000
--- Process Map Dump ---
heee
after open
after fread
after setvbuf
0x000000012001c000-0xffffffffffffffff ---S ffffffffffffffff
0x0000000120028000-0x000000012002c000 RWXP 00018000 /home/loongson/dune/test/stack
0x00000001207c4000-0x00000001207c8000 ---P 00000000 [heap]
0x00000001207c8000-0x00000001207cc000 RWXP 00000000 [heap]
0x000000fff430c000-0x000000fff4314000 RWXP 00000000
0x000000fff4314000-0x000000fff431c000 RWXS 00000000 anon_inode:kvm-vcpu:0
0x000000fff431c000-0x000000fff4320000 RWXP 00000000
0x000000fffbac4000-0x000000fffbae8000 RWXP 00000000 [stack]
0x000000fffbff4000-0x000000fffbff8000 R-XP 00000000
0x000000fffe178000-0x000000fffe180000 R--P 00000000 [vvar]
0x000000fffe180000-0x000000fffe184000 R-XP 00000000 [vdso]

[256480.819242] kvm [3289]: huxueshi trace enter kvm_vz_vcpu_run:3584 cause=40008020
[256480.819250] kvm [3289]: kvm_mips_handle_exit: cause: 0x1080000c, PC: 120007804, kvm_run: 000000001af98063, kvm_vcpu: 0000000042c2abdb cause=40008020
[256480.819253] kvm [3289]: TLB ST fault:  cause 0x1080000c, status 0x740000a0, PC: 00000000f229a1c2, BadVaddr: 0xfffbaa5650
[256480.819257] kvm [3289]: Failed to find VMA for hva 0xfffbaa4000
[256480.819260] kvm [3289]: update_pc(): New PC: 0x120007808
[256480.819263] kvm [3289]: [0x120007808] OP_SD: eaddr: 0xfffbaa5650, gpr: 0x120032610, data: 0x120032610
[256480.819267] kvm [3289]: kvm_mips_handle_exit : cause register = 40008020 exit_reason=6
[256480.819269] kvm [3289]: huxueshi trace leave kvm_vz_vcpu_run:3586 cause=40008020

Yes, it breaks out of the stack boundary, the stack size is 0x24000

因为 child 在创建出来虚拟机之前就会收到信号，在模拟的 fork 返回的位置，信号处理函数正确执行，之后模拟并没有这个信号的干扰。

• I guess not, because it may failed the gup

• how we set up GPR and csr of new process?

• kvm_vz_vcpu_load_tlb
• kvm_vz_vcpu_load_wired
  • why load tlb in kvm_vz_vcpu_run instead of everytime vmexit

TLB need special attention

1. should we copy TLB used in host to guest mode ?

Copy code from tlb.c is error-prone

• sharing global variable with mips kvm maybe disturb kvm

We shouldn't remove request related code

• vz:requests and remote tlb flush

We shouldn't remove mode related code

• vz_vcpu::mode , check it's corresponding code in the kvm and find all the code

• there is no need to destroy the parent vcpu

• fork 之后，parent 和 child 之间的 pid 距离为 3，正常应该是 1 的

Something related to emualted:

• mapped and unmapped
  • kvm_mips_gpa_pte_to_gva_unmapped, kvm_mips_gpa_pte_to_gva_mapped
  • kvm_mips_handle_kseg0_tlb_fault <= kvm_trap_emul_gva_fault <= kvm_get_inst <= used by emulate.c
• kseg0
  • kvm_mips_handle_kseg0_tlb_fault
• emulated
  • x: kvm_trap_emul_gva_fault
    • x: kvm_get_inst
  • kvm_mips_handle_mapped_seg_tlb_fault : caller is emulate

Boring stuff

• move mmu notifier to mmu.c
• kvm_pgd_alloc
• examine transparent hugepage
• dune_arch_vcpu_put : some effort to complete the code
• kvm_mips_handle_commpage_tlb_fault : now that commpage is used for tlb, should dune remove it ?
• remove kvm_host.c
• remove dirty bit tracking in mmu.c

vcpu, ept and guest page table

• A sthread fork will create two guest page table
• A Linux fork with VM clone will create two
• A dune_enter doesn't create user space percpu
  • dune_enter create a new vcpu and ept

kvm_memory_slot

• maybe we should take advantage of it
• move memory_slot related code into one place

review mode

• how kvm_main works

ideas

• host should work in another thread, then anything did in host_loop just like another thread, so it works just like two threads run in parallel.

• page flags
• PDX
• level
• pgroot
  • cr3 for mips ?
• vm.c : dune_vm_default_pgflt_handler : we will not handle it until I finished hello world

• find the book : Dynamic Binary Modification
• how x86 dune take advantage of minimal loader, why not create a similar project for this?
• proxychain can redict all kinds of network related syscall, can I use similar way?

We only care about the syscall, that's all!

• a special syscall to escape to host
• a setjmp and long jump to just to where issue the syscall.

• init
  • seems no hardware enable is needed, kind of unbelieveable
  • hardware_enable_all
  • kvm_vz_hardware_enable
  • kvm_vz_vcpu_put
• entry setup
  • remove kvm_host.h : kvm_vcpu and arch_vcpu
  • register the entry
• hypercall / skip one instruction
• PF_VCPU

	if (current->flags & PF_VCPU)
		kvm_vz_vcpu_save_wired(vcpu);

what should be inited and how?

currently, I just want to do a following how dune works:

• dune_enter => vz_launch => create_vcpu && create_vm (Yes, VM and VCPU is one-to-one pair)
• dune_init(this is module related init)

use Loongson VZ instead of VZ

• we will diff the code later

pipe is special

in /home/maritns3/core/loongson-dune/cross/arch/mips/kernel/scall64-64.S

EXPORT(sys_call_table)
	PTR	sys_read			/* 5000 */
	PTR	sys_write
	PTR	sys_open
	PTR	sys_close
	PTR	sys_newstat
	PTR	sys_newfstat			/* 5005 */
	PTR	sys_newlstat
	PTR	sys_poll
	PTR	sys_lseek
	PTR	sys_mips_mmap
	PTR	sys_mprotect			/* 5010 */
	PTR	sys_munmap
	PTR	sys_brk
	PTR	sys_rt_sigaction
	PTR	sys_rt_sigprocmask
	PTR	sys_ioctl			/* 5015 */
	PTR	sys_pread64
	PTR	sys_pwrite64
	PTR	sys_readv
	PTR	sys_writev
	PTR	sys_access			/* 5020 */
	PTR	sysm_pipe
	PTR	sys_select

Everybody started with sys_ but pipe started with sysm_

/home/maritns3/core/musl/src/unistd/mips64/pipe.s

.set	noreorder
.global	pipe
.type	pipe,@function
pipe:
	lui	$3, %hi(%neg(%gp_rel(pipe)))
	daddiu	$3, $3, %lo(%neg(%gp_rel(pipe)))
	daddu	$3, $3, $25
	li	$2, 5021
	syscall
	beq	$7, $0, 1f
	nop
	ld	$25, %got_disp(__syscall_ret)($3)
	jr	$25
	dsubu	$4, $0, $2
1:	sw	$2, 0($4)
	sw	$3, 4($4)
	move	$2, $0
	jr	$ra
	nop

• work on la host
• or fix ccls cross compile?
  • similar problem happens on arm64 too.

• create a standard output
  • How many TFAIL in native testcases

• https://stackoverflow.com/questions/18863199/gcc-inline-asm-jump-to-a-label-with-crossing-throwing-an-exception
• in savereg.c, we should use m instead of r, so we will not pollute the regsiter

Sometimes, it will crash.

• understand how stack works. That's right, we can't understand what we are writing.

• .no at
• mips doesn't allow access pc directly
  • https://stackoverflow.com/questions/15331033/how-to-get-current-pc-register-value-on-mips-arch
• https://stackoverflow.com/questions/744055/gcc-inline-assembly-jump-to-label-outside-block

#include <stdio.h>

int
main(void)
{
  asm("jmp label");
  puts("You should not see this.");
  asm("label:");

  return 0;
}

• what's hi and lo in MIPS

• ⭐ A short reference for MIPS

• read register to variable with inline assembly code

 unsigned int x;
 asm volatile ("move %0, $ra" : "=r" (x));
 asm volatile ("sw $ra, %0" : "=m" (x));

• we can find the testcase in ltp, why the ltp doesn't report the issue to me?

The most tricky part is about CSR initialization, which we can refer to Stanford dune.

void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack)
{
	unsigned long random_factor = 0UL;

	if (current->flags & PF_RANDOMIZE)
		random_factor = arch_mmap_rnd();

	if (mmap_is_legacy(rlim_stack)) {
		mm->mmap_base = TASK_UNMAPPED_BASE + random_factor;
		mm->get_unmapped_area = arch_get_unmapped_area;
	} else {
		mm->mmap_base = mmap_base(random_factor, rlim_stack);
		mm->get_unmapped_area = arch_get_unmapped_area_topdown;
	}
}

Legacy or not makes a difference?

Although dune can't make the program which is a parameter of syscall exec run in a virtualization environment, making the program run correctly without too many limitations is always better.

• check it with original tests

• 还有，按道理来说，即使是将 dune_enter 放到 parent process 上，也不应该出现 pr_err("KVM_CREATE_VM"); 的错误。

• it's very impolite to crash in the dune, it should return to dune_enter caller?
  • what if it crashed in when we make a fork?

Maybe we insert too many log for debugging.

Maybe we should set up a standard interface for developers to port to another system.

• musl / gcc
  • how clone works
• ltp setup
• kernel
  • .config
  • regs get and set method
• manual
  • SIMD and float save and restore
  • general register save and restore
  • TLB refill entry
  • TLB refill
  • CSR register setup
  • Physical Address Range
  • syscall entry
  • xphysicall offset
  • vmcall
• ABI
  • callee saved and caller saved register
  • gcc asm
    • .set noat
    • .set noreorder
  • syscall
  • why $2 is used for TLS register

从 vcpu_load 的位置切换到 kvm_vz_vcpu_run 大大的减少了发生错误的情况，但是还是存在一些这种情况。

• 至少说明，现在这些情况的确是由于 timerid 的刷新导致的
• 既然 vcpu_load 的实现总是正确的，那么这种同步为什么出错?

➜  mmap git:(master) sudo  ./dune-mmap16
  Info: top = fffb800000, limit = 800000
  Info: open /dev/kvm
  Info: KVM_GET_API_VERSION
  Info: KVM_CREATE_VM
  Info: KVM_SET_USER_MEMORY_REGION
  Info: ebase address : fff6620000
mmap16      0  TINFO  :  Found free device 0 '/dev/loop0'
mmap16      0  TINFO  :  Formatting /dev/loop0 with ext4 opts='-b 1024' extra opts='10240'
mmap16      1  TBROK  :  mmap16.c:168: mkfs.ext4 failed with exit code 128
mmap16      2  TBROK  :  mmap16.c:168: Remaining cases broken

I expected it failed with "unsupported syscall", but didn't

• runtest/syscalls-ipc 's relation with runtest/syscalls ?

syscalls-ipc is just subset of syscalls

No More vcpu

• fcntl36
• init_module01

Need more

• dune-fork09

• inotify09 is much more slowly than works

• I think HYPCALL instruction works, just GCC doesn't support it
  • kernel module containing the 0x42000028 will crash the kernel, but random instruction cause readable error
• how to cause the gup?
  • make KVM recompile easy and debug in the kernel
  • DEBUG option
  • so, check the vmexit ?

trifle

• change boot time from 5s to 0s

Context Switch && Interrupt, here I came.

• dune_config
• dune_trap_regs : general purpose regs + rip + rflags, we can ignore it savely.

It's used for debugging the fork syscall, there is no need to worry about it, but cpu_id is more critical.

• create a program that we will come across KVM_EXIT_INTR 100%
• If I define a signal handler, will it handled in guest space or host space? (host)

--------------------------page break-----------------------

MIPS Virtualization Manual

• what's dune's limitations?
• what's do we need?

There are many fail cases about mmap.