ironlanguages / dlr Goto Github PK

I'm working on a compiler for a little language and keep running into hangs when calling into the compiled delegates with DynamicInvoke. According to dotTrace 42.9% of the time below that is CreateDelegate and the rest seems to be invisible to the profiler. (I assume it's unmanaged.) I've tried pausing in the debugger at random points and ended up deep in the IL generation code somewhere.

Are there any additional debug options I can enable or tips for narrowing this down? I'm linking against debug builds of Microsoft.Dynamic.dll and Microsoft.Scripting.dll, but I think the issue might be farther down.

I'm happy to share access to the repo if anybody is interested in taking a look, but so far I haven't been able to narrow this down to a reproduction case that doesn't involve my compiler and runtime code, which is ~6k lines of C# so far.

I've attached a sample DebugView of the compiled expression tree, but it has dynamic calls back into the runtime. This tree hangs 100% of the time.
failing-expression-tree.txt

ParamDictionaryAttribute is declared in namespace Microsoft.Scripting. However the file in which this class is defined, is Src/Microsoft.Scripting/Runtime/ParamDictionaryAttribute.cs. All other attribute classes are defined in namespaces that match their location in the directory structure (although it is not true in a general case: Src/Microsoft.Scripting/Runtime contains a few other classes that are defined in Microsoft.Scripting).

Looking at other attributes, all of them but two are defined in Microsoft.Scripting.Runtime. It seems to me that ParamDictionaryAttribute conceptually belongs to the same group. And unlike some other "misplaced" types, it is used in many places, so for instance code that uses [ParamDictionary, NotNull] has to open two namespaces.

Should ParamDictionaryAttribute be moved to namespace Microsoft.Scripting.Runtime? Can it be done without causing a breaking change?

Or maybe the .cs file should be moved one directory up? Do we care?

And in general, what are the rules of matching namespaces to directories, if any?

I've written a Visual Studio custom debugging visualizer for expression trees (and other types in System.Linq.Expressions.Expression).

I would like to expose links to the documentation at https://github.com/IronLanguages/dlr/tree/master/Docs from within the visualizer's context menu, much like I already link to the standard Microsoft documentation. Specifically, to expr-tree-spec and dlr-overview. Currently, the files are stored as PDFs and Word documents, which means it's impossible to link within these files, only to the files themselves.

Are there any plans to move this documentation to Microsoft Docs? Or, perhaps to the wiki section on this repo? Or some other globally available host? Presumably it would be easier to work with these files in Markdown; I would be willing to carry out this conversion.

If the documentation is moved to the wiki, what guarantees would I have that the headings -- required for deep linking -- would remain the same?

Are there any plans to support .Net Standard in the future?
.Net Core and .Net Framework are already supported, so I'm guessing this shouldn't be impossible.

Thanks for all your great work on IronPython - it's really appreciated, and we use it heavily for our plugin system. Unfortunately that's currently holding us back from porting our application to .Net Standard...

Might be interesting to split the COM stuff out into its own assembly...

Hi all,
I'm an app developer and I've tried to add IronPython reference to Xamarin project.
The NuGet prompts the following incompatible messages.
Will DLR support Xamarin project in the future? Guess it will be a quite great feature.
Thanks for your great work!

================================================================
Installing DynamicLanguageRuntime 1.2.2.
NU1202: Package DynamicLanguageRuntime 1.2.2 is not compatible with monoandroid81 (MonoAndroid,Version=v8.1). Package DynamicLanguageRuntime 1.2.2 supports:

• net45 (.NETFramework,Version=v4.5)
• netcoreapp2.0 (.NETCoreApp,Version=v2.0)
• netcoreapp2.1 (.NETCoreApp,Version=v2.1)

Hello,

a really Q&D example for showing the issue.

First the simple callsitebinder:

private class BinaryCallSiteBinder : BinaryOperationBinder
{
    public BinaryCallSiteBinder()
        : base(ExpressionType.Add)
    {
    }

    public override DynamicMetaObject FallbackBinaryOperation(DynamicMetaObject target, DynamicMetaObject arg, DynamicMetaObject errorSuggestion)
    {

        return new DynamicMetaObject(
            Expression.Convert(
            Expression.Add(
                Expression.Convert(target.Expression, typeof(int)),
                Expression.Convert(arg.Expression, typeof(int))
            ), typeof(object)),

            BindingRestrictions.GetTypeRestriction(target.Expression, typeof(int)).Merge(
                BindingRestrictions.GetTypeRestriction(arg.Expression, typeof(int))
            ));
    }
}

And the calling code:

var expr = DynamicExpression.Dynamic(new BinaryCallSiteBinder(), typeof(object), Expression.Constant(40, typeof(object)), Expression.Constant(2, typeof(object)));
var f = Expression.Lambda<Func<object>>(expr);
var f2 = f.Compile();
textbox1.Text = f2().ToString();

If I run the code under Windows 8.1 Phone or the desktop .net framework, it works fine.

But under Windows 10 UWP I get the following exception:

System.ArgumentException wurde nicht von Benutzercode behandelt.
  HResult=-2147024809
  Message=Expression of type 'System.Object' cannot be used for parameter of type 'System.Runtime.CompilerServices.CallSite' of method 'Boolean SetNotMatched(System.Runtime.CompilerServices.CallSite)'
  Source=System.Linq.Expressions
  StackTrace:
       at System.Dynamic.Utils.ExpressionUtils.ValidateOneArgument(MethodBase method, ExpressionType nodeKind, Expression arg, ParameterInfo pi)
       at System.Linq.Expressions.Expression.Call(MethodInfo method, Expression arg0)
       at System.Runtime.CompilerServices.CallSite`1.CreateCustomNoMatchDelegate(MethodInfo invoke)
       at System.Runtime.CompilerServices.CallSite`1.MakeUpdateDelegate()
       at System.Runtime.CompilerServices.CallSite`1.GetUpdateDelegate(T& addr)
       at System.Runtime.CompilerServices.CallSite`1.GetUpdateDelegate()
       at System.Runtime.CompilerServices.CallSite`1.Create(CallSiteBinder binder)
       at System.Runtime.CompilerServices.CallSite.Create(Type delegateType, CallSiteBinder binder)
       at System.Linq.Expressions.DynamicExpression.Reduce()
       at System.Linq.Expressions.Interpreter.LightCompiler.CompileNoLabelPush(Expression expr)
       at System.Linq.Expressions.Interpreter.LightCompiler.Compile(Expression expr)
       at System.Linq.Expressions.Interpreter.LightCompiler.CompileTop(LambdaExpression node)
       at System.Linq.Expressions.Expression`1.Compile()
       at TestLua.MainPage.btnClick_Click(Object sender, RoutedEventArgs e)

What is wrong?

Same issue IronLanguages/main#1231

MutableTuple allows creation of tuple of size greater than 128 by nesting tuples (e.g. a Tuple`2 which contains a Tuple`128 and a Tuple`8 if we had a size of 136). In fact, the accessor methods (e.g. MutableTuple.GetAccessPath) expect nested tuples to be laid out exactly this way. However the MutableTuple.GetSize method traverses the whole tuple and also counts the number of elements of any generic argument of type MutableTuple.

Because the size is calculated incorrectly MutableTuple.GetAccessPath can have unexpected behavior. For example, the following should throw ArgumentException but ends up throwing InvalidOperationException:

int size = 8;
Type[] args = new Type[size];
for (int i = 0; i < size; i++) {
    if (i == 5) {
        args[i] = MutableTuple.MakeTupleType(typeof(int), typeof(int));
    } else {
        args[i] = typeof(int);
    }
}

var tupleType = MutableTuple.MakeTupleType(args);
MutableTuple.GetAccessPath(tupleType, 8).ToArray(); // this should throw ArgumentException since the index 8 >= size

This also causes the failure in IronLanguages/ironpython3#926

1. In .NET 8, we finally get a way to save the Assembly. See AssemblyBuilderImpl
2. As for the missing of CompileToMethod, we may use this library as a workaround?

I believe BigInteger and Complex64 (in Microsoft.Scripting.Math) were only there for pre .NET 4.0 frameworks. Since we are no longer targeting those we can get rid of these redundant types.

I added FEATURE_ARGITERATOR since System.ArgIterator was not available in .NET Core's contracts, however it turns out that it is available in the assemblies even if it is not in the contracts so we still need to check for it.

In any case FEATURE_ARGITERATOR is only used in Microsoft.Scripting.Generation.CompilerHelpers.GetConstructors in order to avoid a BadImageFormatException. There may be other types which fail similarly (e.g. ReadOnlySpan in .NET Core) so instead of specifically blacklisting ArgIterator I propose adding a try/catch.

All methods below and other similar just are copies of what they cause. So deprecate it, because will better, if methods will called directly.

We should get rid of the IsCompactFramework since we're not supporting the compact framework (which is dead). I would suggest setting the field to false and adding an obsolete tag to it.

Hi,

I would like to know under which license the sample code is distributed.

Best regards, Paulo

Since .NET Core 3.0 supports COM interop, a netcoreapp3.0 target should be added that defines FEATURE_COM (so Microsoft.Scripting.ComInterop stuff will be included in the assembly).

As in IronLanguages/ironpython2#14 is the plan to drop Silverlight here as well?
If yes, I could try to add a PR.

I can't find any good (up-to-date, anyways) documentation for creating and embedding a new language using the DLR. Since the GitHub project recommended just opening issue, I'm being shameless and submitting it here. I am trying to specify a domain-specific language for NPC and game logic for a hobby game in Unity 3d. So I was particularly trying to define a syntax that worked well using green threads and coroutines. But right now I can't even figure out how to run a calculator.

I could just post some code if desired in multiple layers of disclosure since it's just a trial project, but I'm away from it right now. Eventually I think I'd just pastebin the whole mess since I think the important parts are just in a single file. I defined a grammar using ANTLR4 and I have a visitor that is able to crunch a one-liner calculator-style program statement. My goal was to then make that play in the DLR. Here's what I have so far:

1. I create a ScriptRuntime for my language
2. I get ScriptEngine for my language from that ScripRuntime
3. I create a ScriptSource using that engine using a one-line string for my test program
   • ...at this point, it's going to my LanguageContext.CompileSourceCode implementation, which is where I use my ANTLR4 stuff to make a System.Linq.Expression root node. I shove this into a ScriptCode subclass I created
   • ...something inside the DLR engine internals knows to call the Run() method on my custom ScriptCode, which is where I run my expression as a lambda

The problem is that it's just an incestuous loop right now. I mean, it does actually work, but it's just a bunch of handling to do what I already did. I assume instead of System.Linq.Expression that I should be turning this into some kind of byte code and feeding it to some interpreter. I see an interpreter:

https://github.com/IronLanguages/dlr/blob/master/Src/Microsoft.Dynamic/Interpreter/Interpreter.cs

And the bulk of its syntax:
https://github.com/IronLanguages/dlr/tree/master/Src/Microsoft.Dynamic/Interpreter/Instructions

Is this what the cool kids are using? What's the general route where I get that thing going and I feed it my stuff?

All the information on this is too old and everything seems to have changed. The iron language projects are sitting under many, many layers and I lose track of what is going on. I can't figure out what I should be doing from that--especially for something basic for starters.

I also wonder then how I am supposed to get debug capability, as I believe that is one perk of the DLR. Do I need to associate PDB data with my script or something? Where do I pump that in?

The other issue is after all this, I don't even know if that interpreter is going to help me with green threads. I would want to trip across a custom instruction that would tell the interpreter to stop. At that point, I want to be able to take the entire context the interpreter was using, store it into a frame, and possibly just move on to a totally different script. Preferably I could even serialize that frame so I can save and load it from game data. I'd rather have that than create chains of yields and things like that. Preferably, my parser/pre-compiler would tactfully insert these before major calls and such too.

Microsoft.Scripting.Generation.CompilerHelpers.Compile<T>(this Expression<T> lambda, bool emitDebugSymbols) throws on Mono when emitDebugSymbol is true. This causes IronPython to crash on Mono when debug is set in the runtime options.

This probably started occurring in Mono 5.0 when it switched to using the .NET Core implementation (which throws PNSE).

Here's some (IronPython) sample code to replicate the failure:

var _scriptEngine = Python.CreateEngine(new Dictionary<string, object>() { { "Debug", true } });
File.OpenWrite("aaaa.py").Close();
Python.ImportModule(_scriptEngine, "aaaa");

Probably the best we can do is add a guard to prevent a PNSE...

We should remove or update Microsoft.Scripting.AspNet (or potentially move to a separate repo?).

I got reported an issue:
neolithos/neolua#73

What is going wrong?

There is a certain inconsistency in handling parameters marked with ParamArrayAttribute or ParamDictionaryAttribute. It is reflected in both comments in the DLR code, as well as code itself. Since this is a fundamental concept in the overload resolution, I want to discuss it explicitly and agree on a principle, before changing the DLR behaviour in one way or another. It would be very useful to know which behaviour is desirable and which should be considered a bug. Once this principle is agreed on and implemented, it may be difficult to reverse on it in the future, since many parts of the overload resolution code depend on it.

Let's look at the current situation first. For instance, ParameterWrapper.cs reads:

// params arrays & dictionaries don't allow assignment by keyword

On the other hand, a comment on ParamDictionaryAttribute reads:

/// For languages which don't have language level support the user will be required to
/// create and populate the dictionary by hand.
/// 
/// This attribute is the dictionary equivalent of the System.ParamArrayAttribute.

This suggests that it should be possible to pass such param dictionary and/or param array somehow. Since passing by keyword should be forbidden, this leaves that passing by position should be supported. But this leads to logical inconsistencies, which I describe a little bit later below.

On the actual code side, some parts of the code assume that passing param dictionary/array by keyword is allowed, and some assume that it is forbidden. This leads to erroneous behaviour, ranging from unexpected results, through confusing error messages, to exceptions thrown. Example (from IronPython):

import clr
clr.AddReference("rowantest.methodargs")
from Merlin.Testing.Call import VariousParameters
obj = VariousParameters()
obj.M560(1, 2, 3, z=4, b=5, c=6)

Method M560 has signature public void M560(int x, [ParamDictionary] IDictionary<string, int> y, params int[] z) and the call should succeed. Instead, and IndexOutOfRangeException is thrown:

Traceback (most recent call last):   
  File "<stdin>", line 1, in <module>
IndexError: Index was outside the bounds of the array.

Interestingly, the exception is not thrown during overload resolution, but during the actual call itself, because the call site has been constructed incorrectly: it contains three keywords for keyword arguments (z, b, c) but only two values (5, 6). The value for parameter z is taken over by an actual param array to accommodate additional positional arguments (2 and 3).

Regarding passing by position, the current DLR code does accept param array passed by position:

import System
ar_int = System.Array[System.Int32](3)
# public void M202(params int[] arg)
obj.M202(ar_int) # OK
ar_obj = System.Array[System.Object](3)
obj.M202(ar_obj) # ar_obj passed directly to arg
obj.M202(ar_obj, ar_obj) # arg is an array of 2 references to ar_obj

The last two lines show inconsistency in handling matching type values. I personally think it is unfortunate for a dynamic language, but it matches C#.

Param dictionary, on the other hand, is not accepted when passed as a positional argument:

dict_obj = System.Collections.Generic.Dictionary[System.Object, System.Object]()
# public void M203([ParamDictionary] IDictionary<object, object> arg)
obj.M203(dict_obj)

TypeError: M203() takes no arguments (1 given)

Inconsistencies by allowing keyword binding

Regardless the error, method M560 raises a question what the desired behaviour should have been. For instance, one may say that z=1 should not be interpreted as providing an argument for params int[] z because 1 is not int[], and instead it should be put in dictionary y. But what if the argument (which could be just a variable) were of type int[]? Then we would have a strange behaviour that depending on the type of the argument, a named argument is being processed quite differently. This may be OK in statically-typed languages, but in dynamic languages the type of a variable may not be obvious or may be undetermined at the call site. Finally, such resolution would significantly complicate the current DLR algorithm, which does arity based resolution first, then type based (and determining which positional and keyword arguments go where is part of method arity determination).

Another option could be to always assume that z=some_value binds to the param array and report an error in the case above. Such error would be confusing at best (something like expected int[] got something), but also eliminate a possibility for passing arbitrary keyword arguments to be collected in the param dictionary. Traditionally (IronPython 2), this was solved by using Unicode character U+00F8 trick in the parameter name, which made the name a valid identifier in C# but not in Python. This trick is no longer working with IronPython3 because Python 3 identifiers can contain Unicode letters too. A better solution would be to use a dedicated parameter attribute (say PositionalOnlyAttribute) to explicitly prevent binding by name. This could work but it would mean that some methods with param arrays behave differently than other, and it may be confusing to the user especially if there are several methods in the method group to choose from.

Similar reasoning can be done for keyword binding of param dictionary. In a such case, the option to always assume that y=some_value (where y is a param dictionary) always binds by name to the whole param dictionary is even more untenable, because an often assumption is that a param dictionary as a sole parameter should be able to catch any keyword arguments.

So my conclusion is that the comment in the DLR that params arrays & dictionaries don't allow assignment by keyword is the best way to resolve the ambiguities and the code should be updated accordingly.

Inconsistencies by allowing positional binding

Let's assume the following signature:

public static void M([ParamDictionary] IDictionary<object, object> kwargs, params object[] args)

Arguably, such method should be able to take an arbitrary number of positional and keyword arguments thus never throw a type error.

Further let's assume that dynamic variables kw and ar contain values typed IDictionary<object, object> and object[] respectively, and a a value of any other type. What should be the result of the following calls:

M(a)

Expected: args[0] == a, kwargs empty, nothing strange yet.

M(kw)

kwargs == kw, args empty? (since kw in sin position 1 of kwargs). Or args[0] == kw?

M(kw, a)

kwargs == kw, args[0] == a? (since kw is in position 1 of kwargs). Or args[0] == kw, args[1] == a?

M(a, kw)

kwargs == kw, args[0] == a? By what rule? Or args[0] == a, args[1] == kw?

M(kw, ar, a)

kwargs == kw, args[0] == ar, args[1] == a? (since kw is in position 1 of kwargs, but ar is in position of 2 and not bound to args). Or args[0] == kw, args[1] == ar, args[2] == a?

M(ar)

args == ar? By what rule? ar is in position 1 so there is a type conflict between ar and kwargs. Or maybe args[0] == ar, like in the first example. At least that would be consistent. And what if ar had such a value that ar[0] == kw? Should kwargs == kw (positional binding) or args == ar?

M(ar, a)

I suppose this must be args[0] == ar and args[1] == a, but it is quite different handling of ar than in the previous example.

M(a, ar)

I suppose this must be args[0] == a and args[1] == ar. But a is in position of 1, so it does not bind positionally. Nor does ar, although the type matches.

M(kw, ar)

kwargs == kw, and args == ar, obviously, if positional binding is allowed. Otherwise args[0] == kw, args[1] == ar.

M(ar, kw)

args[0] == ar, args[1] == kw? This means that positional binding is suspended. Quite different outcome than above.

Things get way more complicated with more normal or optional parameters and mixing position and keyword arguments.

I see the only simple and consistent way of handling it by prohibiting positional binding to param arrays and param dictionaries. In the examples above, the types of kw and ar were known, but in a dynamic case, a call site may get its arguments passed on from some unknown source, of an unknown type. It would be really confusing that it may change how the overload resolution is performed.

Prohibiting positional binding means that param dictionaries and param arrays are always constructed implicitly from unassigned/superfluous keyword or positional arguments. If a language has a means (i.e. a special notation) for splatting (like Python), then there is an explicit way for the caller to provide args or kwargs in their array or dictionary form. If a language does not have such syntax, such calls are not possible. I see no reason why DLR should go out of its way to support such calls if the language itself does not.

I do realize that it is a radical call, not the least because the DLR for years already supported positional argument for param arrays (though not for param dictionaries). So a more compromising solution could be to keep it supported, but explicitly prohibit positional argument for param dictionaries. After all, while param arrays are being used by .NET, param dictionaries are specific to the DLR, so a dynamic language implementation is in full control when and if to use them. The price for this option is a more complicated overload resolution ruleset.

One more thing. Generalizing this principle of prohibiting positional binding to be available to other parameters would be very useful. Specifically I was thinking of providing a dedicated attribute (e.g. KeywordOnlyAttribute) to be applied in such cases. This would be a counterpart to PositionalOnlyAttribute.

For instance, a Python builtin function sorted(iterable, /, *, key=None, reverse=False) could be implemented by a method with the signature:

object sorted([PositionalOnly]object iterable, [KeywordOnly]object key = null, [KeywordOnly]bool reverse = false)

I am not sure about the names of the attributes, though. Technically, [PositionalOnly] would actually mean [ProhibitKeyword]. Also, [NonKeyword] would be shorter. Similarly, [KeywordOnly] is informative, but actually means [ProhibitPositional] which is a mouthful; another option [NonPositional] is not the shortest and less informative. [ProhibitPositional] could also well fit a case when applied on a param array implemented by a dynamic language, if the DLR principle turns out to stay that param arrays by default are allowed to be bound by position.

Putting both [PositionalOnly, KeywordOnly] on the same parameter sounds contradictory and should (?) be prohibited (IronPythonAnalyzer job), because it would made the parameter inaccessible to the language. In IronPython, there is already a way of making things inaccessible to the language: [PythonHidden] that is applicable to a wider range of entities but not parameters, so I doubt whether hiding parameters in this fashion would be useful.

This language will be reestablished?

Better yet will be to have a IronTypescript.

From #261

Thread.Abort is not supported and throws PlatformNotSupportedException. Related to #259 and IronLanguages/ironpython3#1024.

The NuGet package contains Microsoft.VisualStudio.CodeCoverage.Shim.dll in the lib\net45 folder.

While reviewing the WinForms implementation of VARIANT interop I was comparing with various implementations found on the web and I noticed you are treating VT_INT/VT_UINT as IntPtr/UIntPtr.

The resolution in the WinForms PR was that VT_INT/VT_UINT need to be 4 byte, even though the C/C++ headers are a bit ambigous the spec is clear and its also what native code is doing.

Just leaving you this as a note in case you want to fix your implementation.

dotnet/winforms#3197 (comment)

In the headers (wtypes.h) it is written up as such:

/*
...
 *  VT_INT              [V][T][P][S]  signed machine int
 *  VT_UINT             [V][T]   [S]  unsigned machine int
*/

But... things like VariantCopyInd() always treat them as a 4 byte value. This aligns with the MS-OAUT specification which states:

VT_INT:
Context Description
V, S, T Either the specified type, or the type of the element or contained field MUST be a 4-byte signed integer.

VT_UINT:
Context Description
V, S, T Either the specified type, or the type of the element or contained field MUST be a 4-byte unsigned integer.

Good time of day.

I am not sure, it is right place to ask, by I'll try.

I develop plugin in C# for soft develop another company.
And sometimes my plugin crush with error

System.NullReferenceException: Object reference not set to an instance of an object.
at System.Dynamic.ExcepInfo.GetException()
at System.Dynamic.ComRuntimeHelpers.CheckThrowException(Int32 hresult, ExcepInfo& excepInfo, UInt32 argErr, String message).

I suggest that can be result of execution 92 and 93 lines in https://github.com/IronLanguages/dlr/blob/master/Src/Microsoft.Dynamic/ComInterop/ExcepInfo.cs
in case exception is null.
Please, could you recommend way to intercept "internal Exception GetException()" in runtime to see "ExcepInfo" fields and "message" if it exists before plugin crush?
Could you suggest way to prevent this error raise or method execution change in runtime?

P. S. Sorry for my bad spelling.
With best regars,
Gorbunov Ivan.

Unhandled exception. System.PlatformNotSupportedException: Thread abort is not supported on this platform. at System.Threading.Thread.Abort(Object stateInfo) at Microsoft.Scripting.Hosting.Shell.BasicConsole.<.ctor>b__24_0(Object sender, ConsoleCancelEventArgs e) at Microsoft.Scripting.Hosting.Shell.BasicConsole.<.ctor>b__24_1(Object sender, ConsoleCancelEventArgs e) at System.Console.HandlePosixSignal(PosixSignalContext ctx) at System.Runtime.InteropServices.PosixSignalRegistration.<OnPosixSignal>g__HandleSignal|10_0(Object state) at System.Threading.Thread.StartCallback() Aborted (core dumped)

At first I assumed Thread.Abort is unsupported on Linux, but then I realized the API is now obsolete on all platforms. How hard would it be to add cancellation tokens and what would be the performance impact?

There are a few forks of the interpreter out there. It would be nice to figure out the difference between these and incorporate any useful changes into our interpreter.

• PowerShell
• .NET Core

What APIs of the Interpreter are used by different project? Could a common assembly be produced?