Metadata about mosa-project HOT 43 CLOSED

@charsleysa The bitmap is already compact and extraordinarily fast to search - O(1) or constant. So its hard to see the benefit of replacing that data structure that requires a slow string search.

Take a look at these research papers on Method Dispatch:

Efficient Implementation of Java Interfaces: Invokeinterface Considered Harmless

A Fast and Efficient Method for Dispatching

from mosa-project.

@tgiphil @kiootic are the bitmaps post-compilation extensible?

From what I can see they do not allow extensibility.

Also where do we use the bitmaps in runtime currently? Aren't the methods' address placed in the binary along with the call instruction instead of fetched from Metadata?

So I found the answers looking through the code, took a while but found out that it's used in CILTransformationStage.Callvirt.

Now bitmaps are nice and they are good for performance but MOSA needs to be extensible post-compilation especially once we introduce a full driver system, MOSA would become too big and complicated if we tried to compile everything at once.

We can get away with that at the moment since MOSA is relatively small but once we start adding features such as full driver system, CIL VM, etc I don't think we will be able to manage it as easily so we need to devise a way to keep performance while also having extensibility.

How do we make bitmaps extensible? Keeping in mind that the Runtime does not know about interfaces in other system files until it loads the file.

from mosa-project.

@charsleysa You are correct; the types' interface bitmaps are not extensible. This is almost by designed since the compiler development is focused AOT instead of JIT. JIT will require different data structures.

However, in this particular case, the data structure can be fixed. Just before the bitmap, store the starting and ending Interface ID that the bitmap represents. This will both truncate the bitmap and if even additional interfaces are added, this data structure would remain fixed and unchanged. The runtime would have to additional check the range before access the bitmap - but that's acceptable.

There are a few other optimizations that can be made. If a type implements only a few interfaces, a list of Interface ID can be stored and searched instead.

Note: There is a interface method dispatch table that is extremely large. I'm open to ideas on how to reduce the system and yet be extensible.

from mosa-project.

@tgiphil I'm starting to think there will be no way of implementing extensibility without sacrificing performance somewhere.

All day the only idea that keeps coming back as the most flexible is using some sort of metadata token and having a few arrays in memory each time MOSA boots which basically have the addresses for each token and since the arrays are sliced up by token type they are faster to search through. I know it's a disliked idea but MOSA won't be able to stay a one file OS forever so eventually we'll need to sacrifice something.

But before that I think we should take a step back and decide at how we are going to load in extra files and how we are going to patch them and what we will patch inside them.

On a side note, I started implementing the new Metadata system we decided upon, its going to be a big change as it affects everything from the Metadata stage through to the Runtime lookups,

from mosa-project.

@tgiphil I have completed the Metadata Stage for the new Metadata System.

MOSA compiles but doesn't run, if you compile it and take a look at the map file you'll get an idea of what the new Metadata System looks like, we'll be using that for rewriting the lookup routines and whatnot.

Custom Attributes currently only support simple value types (e.g. not enums).

from mosa-project.

@charsleysa There is nothing wrong with tokens; the InterfaceSlotOffsets (aka InterfaceID in other discussions) are examples of tokens.

With MOSA, I hope we attempt to build a hybrid AOT/JIT compiler where most methods are loaded with highly optimized re-compiled versions, and fall back to the JIT for when construction new generic or when some method optimizations becomes invalid when a new assembly is loaded.

With a hybrid JIT, the underlying data structure do not need to be fixed. They can be constructed, re-sized and updated dynamically when the runtime loads in an additional assembly. The JIT (or dynamic linker when loading pre-compiled binaries) can patch the actual binary code in memory. For example, maybe the JIT re-assigned tokens values within the actual executable code (similar to how Windows & Linux load shared libraries), or the JIT uses indirect jump tables (similar to Global Offset Table [GOT] and trampolines).

At the moment, those are wishful thoughts. I'd like to stay somewhat focused development-wise on completing the compiler. As that road needs to be completed before a JIT can even exists.

from mosa-project.

@tgiphil The new Metadata System is initially comlpete.

The test suite only shows 8 failures, which I think are existing failures.
I have merged the code into master as everything compiles and runs and I haven't noticed any issues but I only did a few quick tests using the test suite.

from mosa-project.

@tgiphil @kiootic
I've started work on implementing the Runtime side of Metadata.
There are some pretty big changes but so far all changes are working.
Take a look at the commit on my metadata branch for more details. charsleysa@b63758e

from mosa-project.

@tgiphil @kiootic
Progress report! Making good headway, have currently partially/fully implemented roughly 10 classes for the Korlib and counting. Doing some big changes to how the Runtime does Types and Reflection to match .NET and keep Korlib multi-purpose.

Hopefully I should have a running Reflection by the end of the week with the ability to find Types by name, get Custom Attributes, create instances using Activator, and Invoke methods.

The biggest roadblock which I'm going hit fairly soon is invoking methods during runtime.
I need to devise a way of being able to make the 32bit Calling Convention from the compiler available in the Runtime, or at least available as a Native call. Any ideas?

from mosa-project.

@charsleysa Thanks for the update. It's exciting to hear how far along runtime reflection is coming along.

The VES (via the Mosa.Platform.Internal.x86.Runtime and some internal compiler code) should provide the scaffolding necessary to involve a method. As for invoking methods, take a look at how delegates are invoked in Mosa.Compiler.Framework.DelegatePatcher.cs. This might help with the implementation.

from mosa-project.

@tgiphil thanks for the advice but the delegate patcher seems to have compile time information about the method signature from the delegate definition which the invoke method does not have.

The invoke method does not know anything until runtime and only accepts a this object parameter and a object[] parameter.

All information we need about the method and such that we usually use during compile time is available at runtime for invoking methods, just not during compile time.

Also on a side note, the MOSA Runtime Tables diagram got too big so I added some color to make it easier to follow. https://raw.githubusercontent.com/charsleysa/MOSA-Project/metadata/Docs/MOSA-Runtime-Tables.png

from mosa-project.

MosaUnit contains CustomAttributes.
https://github.com/charsleysa/MOSA-Project/blob/fe018150487e80d3edf255d1eb05f3f9aa6b75bf/Source/Mosa.Compiler.MosaTypeSystem/Units/MosaUnit.cs#L33

In the method MosaParameter.Equals they are compared with Equals.
https://github.com/charsleysa/MOSA-Project/blob/fe018150487e80d3edf255d1eb05f3f9aa6b75bf/Source/Mosa.Compiler.MosaTypeSystem/Units/MosaParameter.cs#L35

One need to reimplement CustomAttributes collection to provide correct (nondefault) Equals implementation. Otherwise the comparision of parametres gives mismatch even when CustomAttributes are empty in both parameter instances...

from mosa-project.

@ArsenShnurkov thanks!
I'll update that to properly compare MosaParameter CustomAttributes.

from mosa-project.

@tgiphil thanks to an err in git command usage a whole truckload of code was irreversibly wiped.
Call this a miracle! Apparently git was able to recover the commits using the SHA hash, don't know how.

Hopefully within the next couple of days I should have fixed everything and got it to the point of being able to compile and run MOSA with the refactored Runtime, but only a handful of Reflection features will be working.

from mosa-project.

@charsleysa I'm glad you figured it out and didn't loss anything. GIT, if used correctly, is remarkable robust.

from mosa-project.

@tgiphil so this has become an absolutely massive task which isn't helped by all the problems with structs and whatnot.

I can't seem to figure out a bug to do with structs using base.GetHashCode();.
For some reason the compound load keeps getting an operand that is not a memory address as a destination for the compound load.

I'm hesitant to merge in my code as I have no idea whether or not it will work as I haven't been able to get past the compiler issues which I don't know how they appeared as I haven't really touched the compiler for anything more than adding metadata, properties, etc.

from mosa-project.

@charsleysa Check in the code into your repo and I'll look at it tonight. Which struct do you see the compound load issue?

from mosa-project.

@tgiphil code is all checked in on the Metadata branch in my repo.

I first got the issue with the CustomAttributeNamedArgument struct but I thought it was a one off so I changed the GetHashCode() method, but after further investigation it seems to be happening to other structs as well.

Currently the first struct to fail during compilation is CustomAttributeNamedArgument.

from mosa-project.

@tgiphil correction.

Currently the first struct to fail during compilation is CustomAttributeTypedArgument.

from mosa-project.

I found that Mosa.Platform.Internal.x86.Runtime.GetTypeImpl method is missing. This will need to be resolved to move forward in the debugging.

from mosa-project.

@tgiphil where is it needed? A lot of that kind of functionality was changed and you can find Type internals in Mosa.Platform.Internal.x86.TypeImpl

from mosa-project.

The System.Type::GetType(System.String method invokes that method.

Here's how you get there: In Architecture.cs, add the new StopStage() just prior to the new IRTransformationStage():

    new StopStage(),
    new IRTransformationStage(),

This forces the compiler to skip all the subsequent stages. And if it could run to completion, we could explore all the transformations that lead up to the IR.CompoundLoad and better see the problem.

The immediate case of the assertion is raised because the compiler expects the load into a memory location rather than a virtual register. Once the prior transformation can be seen, we can work out a plan to address the compound load.

from mosa-project.

@tgiphil I've found some bugs and fixed them.

Now the compiler doesn't even reach the the IRTransformationStage, it fails at the StaticAllocationResolutionStage with the .cctor of System.Type contain the Newarr CIL instruction and for some reason there is no MosaMethod.

.method private hidebysig specialname rtspecialname static 
        void  .cctor() cil managed
{
  // Code size       25 (0x19)
  .maxstack  8
  IL_0000:  ldc.i4.s   46
  IL_0002:  stsfld     char System.Type::Delimiter
  IL_0007:  ldc.i4.0
  IL_0008:  newarr     System.Type
  IL_000d:  stsfld     class System.Type[] System.Type::EmptyTypes
  IL_0012:  ldnull
  IL_0013:  stsfld     object System.Type::Missing
  IL_0018:  ret
} // end of method Type::.cctor

from mosa-project.

@tgiphil good news! and some bad news.

I solved the problem with the StaticAllocationResolutionStage, it can't handle Newarr instructions for some reason so I've disabled it from processing Newarr instructions.

Now we are back to the CompoundLoad issue.
Here's the IR that's outputted during the StopStage from CustomAttributeTypedArgument.GetHashCode():

Block #1 - Label L_0000
  Prev: L_FFFFFFFF
L_0000 IR.BlockStart
L_0000 IR.Nop
L_0001 IR.Move V_1 [System.Reflection.CustomAttributeTypedArgument&] <= this [EBP+4h] [System.Reflection.CustomAttributeTypedArgument&]
L_0002 IR.CompoundLoad [System.Reflection.CustomAttributeTypedArgument] V_2 [System.Reflection.CustomAttributeTypedArgument] <= V_1 [System.Reflection.CustomAttributeTypedArgument&], const {0} [System.Int32]
L_0007 IR.Call V_3 [System.Object] <= System.Void* Mosa.Platform.Internal.x86.Runtime::Box64(System.RuntimeTypeHandle*, System.UInt64) [System.Void*], System.Reflection.CustomAttributeTypedArgument$TypeDef [System.Void*], V_2 [System.Reflection.CustomAttributeTypedArgument] {System.Void* Mosa.Platform.Internal.x86.Runtime::Box64(System.RuntimeTypeHandle*, System.UInt64)}
L_000C IR.Call V_4 [System.Int32] <= System.Int32 System.Object::GetHashCode() [System.Void*], V_3 [System.Object] {System.Int32 System.Object::GetHashCode()}
L_0011 IR.Nop
L_0012 IR.Jmp L_0014
L_0012 IR.BlockEnd
  Next: L_0014

I'll do some further debugging and see if I can come up with anything.

from mosa-project.

Please check it in. I would like to review the state before and after some of the SSA and Promotion stages. Thanks.

from mosa-project.

@tgiphil I have already checked in the code to my metadata branch, do you want me to check in the single commit to the main repo?

from mosa-project.

No; don't check into the main repo. I've checked the code out from your repo; however, I still get the same error about not finding GetTypeHandleImpl.

Note: I'm may not be able to look into this under Sunday.

from mosa-project.

@tgiphil that is so strange. Where/which stage is that error coming from?
I don't have the error at all, are you sure you have all my commits?
I have rebuilt the entire solution multiple times and never received that error.

from mosa-project.

I checked in your code into my repo. Same branch name. Only two minor differences - one to add the stop stage and an new assert statement. But still get an error when compiling TestWorld.

from mosa-project.

@tgiphil yes there should be an error when compiling due the the CompoundLoad issue but there shouldn't be anything to do with the Type class.

from mosa-project.

@tgiphil I found the issue.

We are implementing ldobj incorrectly. ldobj copies the value on to the stack for compound types.

As far as my understanding goes we are using the stack only for input and output variables so we need to:

1. if it is a compound type then we need to allocate memory for the copy
2. we need to copy the value from using the pointer to the newly allocated memory
3. we need to set the result as a memory location

EDIT: I can see that there is a stage called ConvertCompoundMoveStage that seems to fix this only if there is a Move instruction afterwards.

from mosa-project.

@charsleysa I have not had much time to dive into this. Have you made any progress?

from mosa-project.

@tgiphil I have indeed.

As I am posting this I am currently fixing another bug. I'll be committing the changes to my metadata branch once I have everything at least compiling.

Through much study of the code I figured out that the ConvertCompoundMoveStage handles more than just moves, so I renamed it ConvertCompoundStage.
Then I modified the ldobj instruction in CILTransformationStage so it would no longer handle compound types as this is now handled by the ConvertCompoundStage.
I learnt that the stack is also used for locals (duh!) and so the ConvertCompoundStage uses the stack to store the copied object from the ldobj instruction.

from mosa-project.

@tgiphil done!
The test OS doesn't run properly but at least everything compiles.
Also I've tried an alternative method for solving #55

from mosa-project.

@tgiphil fixed a bug in TinySimulator and fixed the newarr bug in StaticAllocationResolutionStage.

from mosa-project.

@charsleysa Thanks; keep up the good work.

from mosa-project.

@tgiphil good news!

I have got the OS's running! Everything is coming along nicely, I'm almost ready to merge, I just have to fix a couple more bugs then the first iteration of the new Metadata and Reflection system will be complete and ready to use.

This will be a massive merge and will change a lot of things, mostly on the Runtime and Metadata side of things. I've also added in optimizations and re-written stages for better standardization and portability. After I finish and merge everything I might be off the radar for a while as I am also currently doing my Industry Project (using Java :-/ yea...)

from mosa-project.

@charsleysa That's great news. Well done. I've been very busy with my
current project but will also like to test your commit.

from mosa-project.

@charsleysa Good news indeed!

Since it's a large merge, please commit to your repo and let me know. I'll spend some time testing it before merging it into the main repo.

from mosa-project.

@tgiphil so I created a PR so you can take a look at and test the merge.

I have been bested by pointers and I can no longer think straight so I've given up on trying to fix the remaining bugs for now. Everything seems to run fine so if you keep away from reflection you shouldn't have any problems. Only 12 unit tests are failing so it's not so bad.

from mosa-project.

@charsleysa - Fixed a bug: d3a1b14

from mosa-project.

@tgiphil reflection bugs fixed!
I've also implemented your bug fix. Merge is good to go, just need your sign-off!

from mosa-project.

Looks great!

from mosa-project.