From lines ~180 of gc.lua

local a = {}; setmetatable(a, { __mode = 'vk' });
local x, y, z = {}, {}, {}
a[1], a[2], a[3] = x, y, z
-- fill a with some `collectable' values
for i = 1, 3 do a[{}] = i end
collectgarbage()
print(next(a, 2)) -- Returns 2, table: 593634ad

• Port LuaJ 3 Debug changes
• Split into "debug handler" and "debug lib"

Cobalt's handling of tail calls isn't entirely correct.

• Java functions are tail called. While this was considered a feature of LuaJ, it makes implementing #31 (or rather, the function name aspects of it) rather annoying. I don't think we gain anything from it, so I'd be in favour of removing this.

• Tail calls should be tracked in stack traces. Lua 5.3 prints (...tail calls...) (previous versions do (tail call): ?). This should be possible to do just by setting a tail call flag - I believe that's relatively similar to what PUC Lua does.

Hi,
Is it possible to use Cobalt with cli?

One of Cobalt's weaker points is that it does not impose any limits on the amount of memory the VM can use. Ideally CC: Tweaked would switch over to a more native-style VM which does support this (see cc-tweaked/CC-Tweaked#769), but I think that's a long way away.

Unfortunately, it is impractical to track every single allocation - this would make the implementation significantly more complex, and incur a massive overhead.

One alternative idea, inspired by this OCaml package (though perhaps obvious in retrospect) is to monitor a small sample of our allocations, and estimate actual memory usage based on those. To further simplify things, I propose we only track array allocations: memory usage will be higher than our estimate, but it should still be bounded by some constant factor (2-3x?).

Implementation

• Add some Allocator class, which provides a newTypeArray(LuaState, int size) method for the various core types (byte, LuaValue, a generic T), as well as a corresponding resizeArray.
• The LuaState is augmented with three fields:
  • int allocationCounter: Tracks how many allocations are left before we take another sample.
  • final long maxMemory: The maximum memory we can allocate.
  • AtomicLong currentMemory: The current memory. Note this needs to be atomic as we'll decrement it from another thread.
• When allocating an array, we compute the size of this array in bytes. If the size of the array is larger than a constant (16KiB?) or if decrementing the allocation counter would take it to < 0, then:
  • If this allocation would take us above the maxMemory, then error.
  • Otherwise, increment currentMemory and add this object to a queue of WeakReferences.
  • Update allocationCounter to be a random number between 0 and 2 * our sampling rate (probably 1k). This provides a very basic form of abuse mitigation, by making which allocations are sampled non-deterministic.
• This reference queue is polled on a separate thread (it can be shared across all Lua VMs). Each WeakReference stores its original size and a reference to the owner's currentMemory. When the weak reference is poled, we decrement its owner's memory.

Concerns

The main concern here is this is heavily tied to Java's GC. It's possible the Lua VM could no longer hold a reference to a large object, but the GC hasn't got to it yet, so the currentMemory is still large.

It might be safer to set the max memory to something arbitrarily high (1GiB?) and expose the memory usage via a metric. This way we can get a better idea of the current behaviour before doing anything more drastic.

This Gist, when executed with Cobalt, produces an incorrect error message:

The reported error should mention the upvalue upvalue, not the something variable.

It seems like the test case can be modified to force the output of any register name with the proper setup, I've seen attempt to call local '(for generator)' (a table value) in the wild.

Tested on CCEmuX/CCEmuX@64e3a4a and CCEmuX/CCEmuX@a122391.

Problem

Cobalt needs very long to end his process if the class org.squiddev.cobalt.cmd.lua is used.
Even with simple hello world scripts:

print("hello world")

Measurements

Windows:

PS > Measure-Command {java -cp Cobalt-0.5.1-SNAPSHOT.jar org.squiddev.cobalt.cmd.lua ./helloworld.lua}


Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 0
Milliseconds      : 200
Ticks             : 602003290
TotalDays         : 0,00069676306712963
TotalHours        : 0,0167223136111111
TotalMinutes      : 1,00333881666667
TotalSeconds      : 60,200329
TotalMilliseconds : 60200,329

WSL

$> time java -cp Cobalt-0.5.1-SNAPSHOT.jar org.squiddev.cobalt.cmd.lua ./helloworld.lua
hello world
java -cp Cobalt-0.5.1-SNAPSHOT.jar org.squiddev.cobalt.cmd.lua   0.12s user 0.11s system 0% cpu 1:00.20 total

System Information

Cobalt commit 0803688
Java version:

• Windows: 1.8.0_181, 64bit, oracle [ used to build Cobolt ]
• WSL debian: 11.0.2, 64bit, oracle

I dare say we've got a whole load more of these - might be nice to test for this in the various table methods too?

In vanilla Lua, it attempts to convert strings to numbers, falling back to metamethods if not. Cobalt does not fall back.

Simple test cases:

getmetatable("").__add = function(a, b) return tostring(a) .. tostring(b) end
print("a" + "b") -- "ab"
print("a" + "2") -- "a2"
print("a" + 2) -- "a2"
print("2" + "2") -- "4"
print("2" + 2) -- "4"

Lua defines a "metamethod cache" for tables, which determines which metamethods are absent. See the code in ltm.c

const TValue *tm = luaH_getshortstr(events, ename);
  lua_assert(event <= TM_EQ);
  if (ttisnil(tm)) {  /* no tag method? */
    events->flags |= cast_byte(1u<<event);  /* cache this fact */
    return NULL;
  }
  else return tm;
}

#define gfasttm(g,et,e) ((et) == NULL ? NULL : \
  ((et)->flags & (1u<<(e))) ? NULL : luaT_gettm(et, e, (g)->tmname[e]))

This flag cache needs to be invalidated every time we update the table (search for usages of invalidateTMcache within Lua). Doing it when updating the node's value within rawset should be sufficient.

Discovered on discord with the CC FAQ bot.

It would be nice to save the state of the Lua VM. This should be possible when executing Lua Closures, but we will need to support xpcall and pcall functions.

It would be nice to support LuaJC classes with something like these libraries. It might be possible to do something with JavaFlow

Hello, i'm using metalua for generating ast from lua source and the pattern for MASK_PUNCT inside StringMatch.java is missing punctuations values.
I've check inside luaj current source code and the test i've found is this one

if ( ( c >= '!' && c <= '/' ) || ( c >= ':' && c <= '@' ) || ( c >= '[' && c <= '`' ) || ( c >= '{' && c <= '~' ) ) {
CHAR_TABLE[i] |= MASK_PUNCT;
}
https://github.com/luaj/luaj/blob/master/src/core/org/luaj/vm2/lib/StringLib.java at line 834

Meaning it's an correction made by luaj your missing inside
https://github.com/SquidDev/Cobalt/blob/master/src/main/java/org/squiddev/cobalt/lib/StringMatch.java at line 39

Thanks in advance for the correction, otherwise tell me if you prefer a merge request from me...

For more explanations of how i'm trying to make metalua woking, i'm using checks.lua from https://github.com/denglf/FormatLua and last metalua from luarocks, to prevent using dll ...

The following code should print out abc: 0xffffffff

print(tostring(setmetatable({}, {__name="abc"})))

However, it instead prints out table: 0xffffffff. This is going to be a really nasty change to make here, as we now need to thread LuaState to all the toLuaString methods (though I guess we need that for #66 too).

While working on CraftOS-PC I noticed that Cobalt can give inconsistent results with the length operator when there are multiple nil areas in a table. This appears to be due to the binary search algorithm landing on specific entries and assuming they're on a boundary (which is true). Here's some examples of different results from very similar tables:

According to PUC Lua, these should be returning 9, 8, and 9 (as the contents are statically allocated*). A user might expect all three to resolve to 3, as the first boundary is at index 3.

I know the specification for the length operator is a bit ambiguous, and that these results are technically correct. However, they can be wildly variant, and since users often think it returns the boundary of the first nil they might be caught off-guard with statically allocated* tables of this nature. I'm not sure how this will affect speed, but it might be better for user experience to either a) return length using an array search first, then fall back on hash search (as PUC Lua does), or b) do a linear search (which is slower but more reliable to the user).

^{*When I use "statically allocated" I refer to creating a table with pre-filled slots, including nils in between. Not sure exactly how to refer to this, but whatever.}

I am trying to test Cobalt out and currently have the following that works with LUAJ where I call an Init Function and then call another function to get the result. I cannot see how this is implemented in Cobalt.

public static void main(String[] args) throws Exception 
{
	String script = "add2nums.lua";
	
	// create an environment to run in
	Globals globals = JsePlatform.standardGlobals();
	globals.get("dofile").call( LuaValue.valueOf(script));
			
  //call the function InitAdd2Nums with two parameters 5, and 5
	LuaValue InitAdd2Nums = globals.get("InitAdd2Nums");
	LuaValue retvalsInit = InitAdd2Nums.call(LuaValue.valueOf(5), LuaValue.valueOf(4));	
	
	//print out the result from the lua functio
	System.out.println(retvalsInit.tojstring(1));
	
  //call the function Add2Nums with no parameters and get result back
	LuaValue Add2Nums = globals.get("Add2Nums");
	LuaValue retvalsAdd = Add2Nums.call();			

	//print out the result from the lua functio
	System.out.println(retvalsAdd.tojstring(1));
}

local num1
local num2
local num3

function InitAdd2Nums( pnum1, pnum2 )
num1 = pnum1
num2 = pnum2
print ("INIT num1 " .. num1)
print ("INIT num2 " .. num2)
return 0
end

function Add2Nums()
print ("ADD num1 " .. num1)
num1 = num1 * 10
print ("ADD num2 " .. num2)
num3 = num1 + num2
num4 = num3 / num2
return num3
end

string.rep("foobar", -1) should be valid. It currently throws a negative array exception.

HOW DO I USE THIS PLEASE I BUILT IT IDK HOW TO USE IT I REALLY NEED IT PLEASE

Found this issue when i was messing with table.sort.

print("first")
local test = {"d","a","b","c"}
table.sort(test, function(a, b) return a < b end)
for k, v in ipairs(test) do
  print(k, v)
end
print("second")
local test = {[1]="d",[2]="a",[3]="b",[4]="c"}
table.sort(test, function(a, b) return a < b end)
for k, v in ipairs(test) do
  print(k, v)
end

This code generates this output in Cobalt. As one can see second table is not getting sorted...

While in other lua instances it creates:

first
1    a
2    b
3    c
4    d
second
1    a
2    b
3    c
4    d

This seems to imply there is something wrong with table related logic in Cobalt.

Does have a plane to support "goto" keyword ? I want to use lua5.2 , lua5.1 is too old.

LuaJ 3.0 supports multiple Lua instances at once, this is a major priority so it would be nice to backport this feature of LuaJ to 2.0.

Some tests will fail randomly but work the next time (mostly nextvar). This means CI isn't as effective as it should be.

I need to research whether this was Lua 5.2 or Lua 5.3:

• debug.getinfo includes nparams and isvararg
• debug.getlocal can be called on functions to return function arguments.

• string.find("", "[") causes an array out of bounds, should be malformed pattern (missing ']')

http://lua-users.org/wiki/GotoStatement

Would be useful.

There's a lot of changes Cobalt has made which would be worth upstreaming to the new luaj/luaj repository.

Some of these may not be possible, or at least worth discussing in more detail first (our interpreter work, new error system, etc...) but some of the minor changes are definitely worth it.

Consider the following:

> return 1a
For input string: "1a"

Clearly this is undesirable, as it contains no source positions, etc... The problem here is that number parsing just calls Double.parseDouble, without catching the resulting exception. Doing so, and then rethrowing a more informative error is most likely the way to go.

LuaJC should ideally have full support for string.dump and debug.getinfo. debug.sethook for call, return and line would be nice as would debug.setupvalue (and debug.getupvalue). I do not think that debug.get/setlocal is possible so I can ignore that.

Debug frames are currently only created for a small number of functions (namely Lua closures and those which extend ResumableVarArgFunction). This is because pushing the debug frame is created in the function definition (i.e. LuaFunction.call/LuaFunction.invoke) rather than in the invoking code (OperationHelper.call/OperationHelper.invoke).

We should clearly change this, as it would remove the need for a whole bunch of nasty hacks (for example), and makes things much more consistent with PUC Lua.

("%s"):format(setmetatable({}, {__tostring=function()return "X"end}))

This returns X in PUC Lua and table: xxxxxxxx in Cobalt.

so i need to learn how to use this api can you tell me how to load a lua file and read and write stuff to it be awesome just need to use this for a project i want to like call functions in java using lua from this api but i cant figure out how to use this i appreciate if you helped me i really need this.

• Weak keys do not modify the existing table. We'd need to proxy it.
• next errors on a recently deleted key. This doesn't occur in normal Lua.
• Errors do not display the local name (attempt to call foobar (a nil value))
• debug.getinfo uses the wrong stack position for the name

Java's String.format doesn't produce the same output as ISO C's printf for %g format specifier. As a result, we cannot directly use String.format for this case.

Currently, we roll our own formatting functionality. However, this uses internal APIs (sun.misc.FormattedFloatingDecimal), which is a problem should people want to use Cobalt on other JVMs, or versions later than Java 8 (where it now lives in jdk.internal).

It'd be nice to have something a little more compatible, however I can only see a couple of solutions, and none of them nice:

• License Cobalt under GPL + Classpath exception, and copy in the Sun APIs.
• License Cobalt under Apache 2.0, and copy from Apache Harmony. Eh, this then depends on an IBM thing I think?
• Yoink from Graal
• Roll our own Dragon4 implementation. The paper is available, so it'd definitely be possible. Though again, not entirely desirable and very error prone.
• Post-process String.format. This was our previous solution, and it did end up being a little hit-and-miss. However, it's worth noting that this is what Nashorn does, and it seems to work for them. Looks like Nashorn switched to a Java implementation of double-conversion.

While long-term I do want to remove support for bytecode (as part of the Lua 5.2+ transition), it would be good to support it as well as possible right now.

We've seen some issues on CC:T where we load invalid bytecode due to binary/plain-text incompatibilities (cc-tweaked/CC-Tweaked#1604), which causes errors when internal VM errors when running the program. We should probably do some very basic verification of the bytecode (just check values are in range) so we catch these issues at parse time rather than compile time.

There's a couple of features we don't currently support. Quoting from 3.1 of the Lua manual

The escape sequence \z skips the following span of white-space characters, including line breaks; it is particularly useful to break and indent a long literal string into multiple lines without adding the newlines and spaces into the string contents

We can specify any byte in a short literal string by its numeric value (including embedded zeros). This can be done with the escape sequence \xXX, where XX is a sequence of exactly two hexadecimal digits [...]

The UTF-8 encoding of a Unicode character can be inserted in a literal string with the escape sequence \u{XXX} (note the mandatory enclosing brackets), where XXX is a sequence of one or more hexadecimal digits representing the character code point.

Doing this would allow us to remove most modifications to the utf8.lua test suite.

Remove Fix TailcallVarargs

This isn't really needed, so can be removed. Most of the time .eval() is called on it straight away.

local function a() 
  return error("Um") 
end
print(select(2, xpcall(a, debug.traceback))) -- Error occurs here, when it should occur at line 2.

--[[
input:2: Um
stack traceback:
    [C]: in function 'error'
    input:2: in function <input:1>
    [C]: in function 'xpcall'
    input:4: in main chunk
]]

Fix string bug

By default encodeTo/decodeFromUtf8 should not be used and a straight new String(bytes) should be used.

• Include line numbers in the output
• Ensure edianness is consistent with Lua 5.1

• Rename type methods to camelCase
• Extract concat to operation helper
• Move type to final method and field instead of virtual dispatch.

For instance, 2f parses fine, as does 2e3d. This is because Java's double parsing implementation is perfectly happy with these modifiers. We should either check for these before parsing, or see if there's a stricter option for parsing.

A compatibility edge case was discovered during discussion here, so adding this issue for reference.

SquidDev commented:
For reference:

$ lua5.1
Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio
> =("%d"):format(0/0)
-9223372036854775808

$ lua5.3
Lua 5.3.6  Copyright (C) 1994-2020 Lua.org, PUC-Rio
> =("%d"):format(0/0)
stdin:1: bad argument #1 to 'format' (number has no integer representation)

Meanwhile Cobalt returns 0.

I think the best way to handle this is to modify StringFormat, as I think LuaValue.checkLong should return the value it currently returns. So a check if the value is double, and if so, either call Double.doubleToLongBits (lua 5.1), or signal an error (lua 5.3).

https://github.com/SquidDev/Cobalt/blob/2cec621e10982ed98024952daa138ce13ad718d9/src/main/java/org/squiddev/cobalt/lib/StringFormat.java#L79-L86

The following code:

local function f(...)
  f(1, ...) -- should be _print
end
f()

Results in an error like so:

We probably need to flatten these args once they reach a certain depth (say, 10). Otherwise we've just invented a very inefficient linked list.

String formatting on specific numbers appears to produce incorrect values.

> ("%.f"):format(32.3*1000)
32300
> ("%.14g"):format(32.3*1000)
323

Using a value with a large number of decimals makes this bug more apparent:

Credit to @Lustyn for finding this bug ^{_{and entirely failing to report it >_>}}.

• pairs should now query the __pairs metamethod (see 5.3's implementation).

• LuaTable.inext should use .get instead of rawget. We'll probably need to inline it into the main BaseLib implementation.

  We should possibly also support the __ipairs metatable. While this is deprecated in 5.3, it's still there under the LUA_COMPAT_IPAIRS/LUA_COMPAT_5_2 flags

Compare Cobalt:

> string.sub(false)
stdin:1: bad argument: number expected, got boolean

with PUC Lua:

> string.sub(false)
stdin:1: bad argument #1 to 'sub' (string expected, got boolean)

The latter is obviously much more descriptive and useful. There's two things we should be doing here:

• Keeping argument indexes in the error message.
• Include error messages. See luaL_argerror for how it's done by Lua, but it's not entirely clear this is correct in the presence of tailcalls - we may need some additional sanity checks.

Also worth thinking about how, if at all, we can expose this to CC.

Currently, LuaUserdata overrides LuaValue.toString to return the object's value:
https://github.com/SquidDev/Cobalt/blob/99ecbf020a07a4f1fc0202937c32d91e26817629/src/main/java/org/squiddev/cobalt/LuaUserdata.java#L43-L46

However, doing this causes the string value of the userdata to be formatted differently from PUC Lua:

I expect the result of tostring(userdata) to be in the format userdata: 1234fedc, but instead the result is just a number in this case - it could be different for other userdata values, but it will never start with userdata.

The fix would be to remove the override, but I'm not sure if it holds any significance so I'm not making a PR for it.

From the docs:

debug.upvalueid(f, n)

Returns an unique identifier (as a light userdata) for the upvalue numbered n from the given function.

These unique identifiers allow a program to check whether different closures share upvalues. Lua closures that share an upvalue (that is, that access a same external local variable) will return identical ids for those upvalue indices.

debug.upvaluejoin (f1, n1, f2, n2)

Make the n1-th upvalue of the Lua closure f1 refer to the n2-th upvalue of the Lua closure f2.

local function get_name()
    print(debug.getinfo(1).name)
end

local function no_tail() get_name() end
local function tail() return get_name() end

no_tail()
tail()

This prints "get_name, tail" on Cobalt, and (the correct) "get_name, nil" on Lua 5.1. debug.getinfo should ensure the function on the parent stack is the same as the function currently being requested.