Function Overloading Overhaul (@function annotation)

[carsakiller]

The Problem

I think function overloading needs some changes in order for it to really function in the way that most people would find useful. This is especially problematic with event systems, as I and others have encountered.

Example

Currently, let's say I have the following function that I want to provide an overload for:

function Shape:listen(event, callback) end

Using @overload

The first logical option is to use @overload:

---@class Shape
local Shape = {}

---Subscribe to an event on this shape
---@param event "Destroy"
---@param callback fun(self: Shape)
---@overload fun(event: "Repair", callback: fun(self: Shape, amount: number))
function Shape:listen(event, callback) end

But there is a problem, when using methods (:), the first parameter only gets completions for the first @param and ignores the @overload entirely.

Ok, so for testing, let's replace the method (:) with a static function (.):

---@class Shape
local Shape = {}

---Subscribe to an event on this shape
---@param event "Destroy"
---@param callback fun(self: Shape)
---@overload fun(event: "Repair", callback: fun(self: Shape, amount: number))
function Shape.listen(event, callback) end

This still isn't great, we are still offered both callbacks even though the info we have entered only matches the @overload. At least the first parameter was completed this time.

Multiple Definitions

So then maybe we try defining multiple functions where each event param has the type set to the event name we are looking for:

---@class Shape
local Shape = {}

---Subscribe to an event on this shape
---@param event "Destroy"
---@param callback fun(self: Shape)
function Shape:listen(event, callback) end

---Subscribe to an event on this shape
---@param event "Repair"
---@param callback fun(self: Shape, amount: number)
function Shape:listen(event, callback) end

Now, even as methods (:) we are receiving correct completions for the first parameter... nice! However, we are still receiving two completions for the callback - there is no narrowing happening. The completion also shows the event as "Destroy", which is incorrect for our second definition as we have only allowed "Repair".

At least when defining the function twice, we are able to write a separate description for each of them as well as their @params and @returns. However, we receive a warning saying that we have a duplicate field.

Proposed Solution

See @flrgh's idea to add a @function annotation to add more in-depth support for defining functions overall.

[Derpius]

This would also fix another issue I'm having where classes that have both a static and method variant of the same function aren't narrowing, eg:

---@meta

---@class Foo
---@overload fun(): Foo
Foo = {}

---@param eventName string
---@param callback function
function Foo.Subscribe(eventName, callback) end

---@param eventName string
---@param callback function
function Foo:Subscribe(eventName, callback) end

...

local inst = Foo()
inst:Subscribe(" --> Provides autocompletion for the static method, treating the string as the callback

[carsakiller]

Yeah, I don't know of a way where you can change the callback param given a certain first parameter, which is a very common use-case for an event system or asynchronous code. Hopefully changing how overloads work and improving the narrowing makes it possible 🙂

[serg3295]

This problem has been partially solved. However, the inconvenient thing is that it is impossible to display comments for each event separately.

---Registers a callback function for an event.\
---`function(client[, topic[, message]])`. The first parameter is always the client object itself.\
---Any remaining parameters passed differ by event:
--- @class Emit
--- @field on fun(self, event: string, cb: function)
--- @field on fun(self, event: '"connect"',  cb: fun(client))
--- @field on fun(self, event: '"connfail"', cb: fun(client, reason: integer))  #If the event is `"connfail"`, the 2nd parameter will be the connection failure code.
--- @field on fun(self, event: '"message"',  cb: fun(client, topic: string, message: string)) @If event is `"message"`, the 2nd and 3rd parameters are received topic and message, respectively, as Lua strings.
local emit = {}

emit:on("message", function (client, topic, message) end)

or

---comment**********
---@class EmitL
---@field listen fun(self, eventName: string, cb: function)
--- comment for e1
---@field listen fun(self, eventName: '"e1"', cb: fun(i:integer)):table
---@field listen fun( eventName: '"e2"', cb: fun(s:string))  @comment for e2
---@field listen fun(self, eventName: '"e3"', cb: fun(i:integer, s:string))  @comment for e3
local emitL = {}


emitL:listen("e1", function (i) end)
emitL.listen("e2", function (s) end)

[flrgh]

Poking my head in to add ideas here (read: personal wish list features 😉). I think a good solution/step would be to support functions as first-class objects alongside classes.

The issue with function aliases (fun(...))

The fun(...):... syntax is so terse/dense that it becomes cumbersome to use for all but the most simple cases. Also, if you're properly namespacing your custom types in order to not pollute the global ns, you wind up with reeeeeally long lines:

--- This is my function. If you pass it a callback, it does something different.
---
---@param foo string
---@return integer
---@overload fun(foo: string, callback: libraryname.functionname.callback):libraryname.functionname.result
local function do_the_thing(foo) end

---@alias libraryname.functionname.callback fun(result: string|nil, error: string|nil):boolean

An alternative: standalone function definitions

This would make things a lot more flexible.

The @function label

Example:

--- This is an optional callback.
--- Because it has it's own block (instead of `fun(...)`), I can write doc strings for everything (yay!).
---
--- The `@function` label comes first here for consistency with `@class`
---
---@function libraryname.functionname.callback
---@param result? string  # result text (`nil` if there was an error)
---@param error?  string  # an error message (always `nil` when `result` is not `nil`)
---@return boolean success

--- This is my function.
---
---@param foo string
---@return integer
local function do_the_thing(foo) end

--- This is also my function, but when passed a callback.
---
---@function libraryname.functionname-with-callback
---@param foo string
---@param callback libraryname.functionname.callback
---@return libraryname.functionname.result

Reusing @Class

Alternatively, the @class label/decorator/keyword could be overloaded to support function annotations:

--- This is an optional callback.
--- Because it has it's own block (instead of `fun(...)`), I can write doc strings for everything (yay!).
---
---@class libraryname.functionname.callback : function
---@param result? string  # result text (`nil` if there was an error)
---@param error?  string  # an error message (always `nil` when `result` is not `nil`)
---@return boolean success

Applying the overload

Now, to apply libraryname.functionname-with-callback as an overload of do_the_thing, we have options...

Overload at the target by passing a type name to @overload

--- This is my function.
---
---@param foo string
---@return integer
---@overload libraryname.functionname-with-callback
local function do_the_thing(foo) en

Annotate at the source with a new @overloads label

--- This is also my function, but when passed a callback.
---
---@function libraryname.functionname-with-callback
---@param foo string
---@param callback libraryname.functionname.callback
---@return libraryname.functionname.result
---@overloads `do_the_thing` -- resolve "do_the_thing" as a lua identifier in local scope

If we allow @function to be applied to actual code, then we can take it a step further and allow overloading from different sources/files, which would be nice for anyone working with framework-y code

• libraryname.lua

--- This is my function.
---
---@function libraryname.functionname
---@param foo string
---@return integer
local function do_the_thing(foo) en

• other.lua

--- This is also my function, but when passed a callback.
---
---@function libraryname.functionname-with-callback
---@param foo string
---@param callback libraryname.functionname.callback
---@return libraryname.functionname.result
---
---@overloads libraryname.functionname -- resolve "libraryname.functionname" as a fully-qualified type name

---

The benefit of @function and @overloads is that now it's possible to create function annotations without having to render actual code, which would be a big win IMO.

[carsakiller]

I think adding a @function annotation would be a nice replacement for the fun(...):... syntax. I too dislike the current way of defining functions because of all the information that is missing and because they have to be one line.

@overload could also be "replaced" with the functionality you explain with@overloads which refers to a @function or normal Lua function. For this to really work though, we would probably need to namespace annotations, like @flrgh said. It would be nice if this could be handled automatically so that an annotation on a function myFunction in myLibrary.lua could be referred to by using myLibrary.myFunction.

[carsakiller]

Here is another issue I am experiencing with @overload, the returns are not narrowing and it results in a union type... no matter how I try to define it:

---@param event "A"
---@param callback fun()
---@return boolean
---@overload fun(event: "B", callback: fun()): string
local function listen(event, callback) end

local result = listen("A", function() end)
local result = listen("B", function() end)



---@overload fun(event: "A", callback: fun()): boolean
---@overload fun(event: "B", callback: fun()): string
local function listen2(event) end

local result = listen2("A", function() end)
local result = listen2("B", function() end)

result is boolean|string in all of these cases

[TurtleP]

One other thing with this is that if you're trying to overload something that already exists, it will give you both definitions, which is not ideal (observed by @carsakiller and myself).

On top of that, if you try to nil out an existing function (which perhaps isn't really supported because why would you do it?) doesn't do anything. I'd like to be able to remove an existing documented function that's in the LÖVE documents, but doesn't exist for LÖVE Potion.

See #1834 for the discussion about this.

[hinell]

1. It's enough to introduce @function tag which would define a new instance function type that can be referenced by @type
2. @overload per doc isn't intended to be used for function declaration, it's OOP concept for already declared funcs
3. TypeScript is bad in most of the cases; don't bring anything from that language; you may learn how some things are done, but let's not borrow anything blindly; the TS has too many pitfalls and syntax allows you to write very unoptimized, completely dumb code.

[bavalpey]

I'm wondering if there is any status on this or plans for the @function tag?

[vaderkos]

Looks like same issue

[tomlau10]

reply to: #1456 (comment)

It's unrelated to @overload, it's because in the fun() syntax, there is an ambiguity between function has multi return and function has multiple arguments.

• When you write fun(valu, key, cnt): any
  => this is a function that returns any
• However when you write fun(valu, key, cnt): any, tbl: table
  => this is also a valid syntax that a function return any and table
• so it get parsed as a multi return of the fun() that you defined for consume
  rather than a 2nd tbl argument for Table.Each itself

You have to add () to the fun() in your first @overload to eliminate this ambiguity:

local Table = {}

---@overload fun(consume: (fun(value, key, cnt): any), tbl: table): nil, integer
---@overload fun(consume: (fun(value, key, cnt): any)): fun(tbl: table): nil, integer
Table.ForEach = {}  --[[@as function]]

local mock = { 'a', 'b', 'c', 'd', 'e', 'f' }

local each = Table.ForEach(function () end)

local res1, cnt1 = Table.ForEach(function () end, mock) -- res1: nil, cnt1: integer
local res2, cnt2 = each(mock)   -- res2: nil, cnt2: integer

This should work for your case 😄 @vaderkos

[tomlau10]

I want to add that, I tested some of the incorrect type narrowing issues mentioned in the thread and they actually work in latest version of v3.10.5 🎉 (maybe get fixed in the middle, or by the recent #2765 )

I tested the following and they work, so they maybe closed 😄

• the code snippet that @carsakiller provided above: Function Overloading Overhaul (@function annotation) #1456 (comment)
• Infer overload based on argument values #1898
• Unioned Function Types don't properly work as expected #2102

[carsakiller]

Wow, the original example does seem to work!

I still have trouble with CC:Tweaked's event system, though, as an example.

I don't want to define the function 30+ times with all the same description, so I can't just do what is in the original example (forgot descriptions are inherited from any overload with one). But overloads also don't work for me here. There is no way to handle the string... variadic returns for computer_command events, and because os.pullEvent() can be used with custom events, it means I have to also accept any string and return potentially any... value.

Any?

---@async
---@param event? ccTweaked.os.event
---@return any ...
---@overload fun(event: "alarm"): "alarm", integer
---@overload fun(event: "char"): "char", string
---@overload fun(event: "computer_command"): "computer_command", string...
function os.pullEvent(event) end

This results in all of my returns being any even when they are explicitly defined as having a set value for that case.

local event, alarmID = os.pullEvent("alarm")
  --> ccTweaked.os.event, any
local event, character = os.pullEvent("char")
  --> ccTweaked.os.event, any
local event, arg1, arg2, arg3 = os.pullEvent("computer_command")
  --> ccTweaked.os.event, any, any, any

Unknown?

---@async
---@param event? ccTweaked.os.event
---@return unknown ...
---@overload fun(event: "alarm"): "alarm", integer
---@overload fun(event: "char"): "char", string
---@overload fun(event: "computer_command"): "computer_command", string...
function os.pullEvent(event) end

This results in everything being unknown and I don't even get a warning for :sub-ing an unknown.

local event, alarmID = os.pullEvent("alarm")
  --> ccTweaked.os.event, integer|unknown
local event, character = os.pullEvent("char")
  --> ccTweaked.os.event, string|unknown
local event, arg1, arg2, arg3 = os.pullEvent("computer_command")
  --> ccTweaked.os.event, string|unknown, unknown, unknown


local s = character:sub(0, 3)
-- no unknown warning on :sub-ing a possibly unknown value??


---@type unknown
local a -- cannot infer unknown type

Try redefining?

---@async
---@param event? string The event to listen for
---@return string event The name of the event that fired
---@return unknown ... Any values returned by the event
function os.pullEvent(event) end
---@param event "alarm"
---@return "alarm"
---@return integer alarmID
function os.pullEvent(event) end
---@param event "char"
---@return "char"
---@return string character
function os.pullEvent(event) end
---@param event "computer_command"
---@return "computer_command"
---@return string ...
function os.pullEvent(event) end

This is slightly better and slightly worse than the above attempt to use @overload with unknown. The other problem here is this wouldn't be an option at all if I weren't writing @meta definitions and were instead writing runtime code.

local event, alarmID = os.pullEvent("alarm")
  --> string|"alarm", integer|unknown
local event, character = os.pullEvent("char")
  --> string|"char", string|unknown
local event, arg1, arg2, arg3 = os.pullEvent("computer_command")
  --> string|"computer_command", string|unknown, string|unknown, string|unknown

I have tried a bunch of combinations of different tricks but am still unable to get os.pullEvent working nicely with a proper description, completion, and type checking for all 30+ events plus custom events.

... pull requests accepted 😆

[tomlau10]

There is no way to handle the string...

Yeah, seems <type>... is not supported in fun() syntax when using it in @overload. This only works when you write it by redefining functions under a @meta definition file (where you can use @return string ... there)
LuaLS's official wiki also suggested to use @meta with function definition instead of just using @overload 😂
https://luals.github.io/wiki/annotations/#overload

---

I think the issue you encountered here is that you have to support custom events, in which you have to add string to the @alias ccTweaked.os.event. This just messes up the luals type narrowing feature, because all your built-in event (alarm / char / etc) can be casted to string. So the base function signature that returns any ... always matches and cannot be filtered out. 😕

If you remove the base function signature, then everything works fine. (ofc I know this cannot be the solution, because you have to support custom events).

Some immature idea

When I was trying to fix #2758 (comment), I leant that the logic of finding matched call resides in here:

lua-language-server/script/vm/function.lua

Line 359 in ddc96bd

function vm.getExactMatchedFunctions(func, args)

Currently it just uses vm.canCastType when checking if each param can match that in the overload. I have some immature idea that, maybe we can try to further narrow the funcs[]:

• The first pass (current logic) will check param type can be casted to function definition param type.
  As "alarm" can be casted to string, it matches
  => so the basic function signature is included there (which causes the problem)
• We may actually try to further narrow it, say in the second pass, we keep only the longest exact match in param list (without the use of vm.canCastType, but I don't know how currently)
• Take your code as an example, when you write os.pullEvent("alarm"), in the 1st pass it matches 2 function signatures when using the vm.canCastType(): the overload one "alarm" and the basic one string
  • However if we require exact match, only the overload one can match the 1st param "alarm"
  • So the overload one has an exact match of length 1, while the basic one has length 0
  • => finally we only keep function signature that are equal to longest exact match length, which is only the overload one 🤔

---

As this is unrelated to @function which this issue suggested, I should probably open a new one to track it when my idea become more mature 😄 Stay tuned @carsakiller

[tomlau10]

reply to: #1456 (comment)
(this maybe a bit late 🙈 )

I think I know why your example code doesn't work as expected. It's because how LuaLS filter out unmatched function signatures. It first check the params count:

lua-language-server/script/vm/function.lua

Lines 392 to 408 in ddc96bd

	function vm.getMatchedFunctions(func, args, mark)
	local funcs = {}
	local node = vm.compileNode(func)
	for n in node:eachObject() do
	if n.type == 'function'
	or n.type == 'doc.type.function' then
	funcs[#funcs+1] = n
	end
	end
	local amin, amax = vm.countList(args, mark)
	local matched = {}
	for _, n in ipairs(funcs) do
	local min, max = vm.countParamsOfFunction(n)
	if amin >= min and amax <= max then
	matched[#matched+1] = n

In your code, you have Foo.Subscribe(eventName, callback) and Foo:Subscribe(eventName, callback)

• the first one has 2 params (min = 2, max = 2)
• the second one can be viewed as Foo.Subscribe(self, eventName, callback)
  => that's 3 params (min = 3, max = 3)

So when you are typing the code in the middle inst:Subscribe("
it can be viewed as inst.Subscribe(self, "

• only 2 params are defined (amin = 2, amax = 2)
• thus the Foo:Subscribe(eventName, callback) overload get filtered out
  it requires 3 params, which is more than you supplied in this intermediate code
  (the if amin >= min and amax <= max check failed)
• so only the Foo.Subscribe(eventName, callback) left ...
  => then LuaLS infers that your "" is the callback param 🤦‍♂️

---

If the callback param is declared optional then LuaLS can type narrow it correctly, because in that case min = 2, max = 3

---@meta

---@class Foo
---@overload fun(): Foo
Foo = {}

---@param eventName string
---@param callback? function
function Foo.Subscribe(eventName, callback) end

---@param eventName string
---@param callback? function # add the `?` to callback
function Foo:Subscribe(eventName, callback) end


local inst = Foo()
inst:Subscribe(" --> this will be eventName now

I don't know if this type narrow logic can be optimized, but I don't think an @function will help this case either. What I can be certain is that it should be another issue to track it. 🤔 @Derpius

[carsakiller]

LuaLS's official wiki also suggested to use @meta with function definition instead of just using @overload 😂
https://luals.github.io/wiki/annotations/#overload

Someone really smart and good-looking must've written that, @tomlau10 😉

[Bilal2453]

After 2 days (probably more) of trying to find a solution, I think I got it! Here is what we have, the following is what people have already been doing to get type narrowing correct with the callback completion, as described by @serg3295:

---@class Emitter
---@field on fun(self: self, event: string, cb: function)
---@field on fun(self: self, event: "success",  cb: fun(client))
---@field on fun(self: self, event: "fail", cb: fun(client, reason: integer))  # Cool Message 1
---@field on fun(self: self, event: "message",  cb: fun(client, topic: string, message: string))
local emitter = {}

emit:on("message", function (client, topic, message) end)

Problem with this as described is you won't be able to have descriptions which is what the proposed @function is for in the context of this problem (it is useful for other things overall and should be added), a workaround for this is using aliases with enumerations ---|, so like this:

---@alias events.success
---|"success" # Emitted on successful connections - important to have only a single entry in this alias

---@alias events.fail
---|"fail" # Emitted on failures

---@alias events.message
---|"message" # Emitted when a message is received

--- @class Emitter
--- @field on fun(self: self, event: string, cb: function)
--- @field on fun(self: self, event: events.success,  cb: fun(client))
--- @field on fun(self: self, event: events.fail, cb: fun(client, reason: integer))  # Cool Message 1
--- @field on fun(self: self, event: events.message,  cb: fun(client, topic: string, message: string))
local emitter = {}

emit:on("message", function (client, topic, message) end) -- now you see a description that otherwise would have no been possible to see

It is tedious when you have a lot of events like this, but not like I will be the on responsible for the large annotation packages in the market place 😝

Edit: never-mind perhaps! seems like I was too excited and didn't notice callback not being narrowed correctly anymore, how sad!

A different approach to this is to define a general enumeration alias that contains all events with their descriptions, but this will result in a duplicated suggest, like below:

Screencast.From.2024-10-27.03-04-43.mp4

[tomlau10]

reply to: #1456 (comment)

I tested the given code snippet, although the auto suggestion is not narrowed correctly at the beginning, it works after typing f and retrigger auto completion 🤔

emitter:on("success", f--< retrigger completion here after typing `f`

I guess this is due to different logic of type narrow between completion and match call

• The type narrow in a match call is more complicated and thus perform better, but it will first check the current call arguments count match
  • i.e. the signature on fun(self: self, event: events.success, cb: fun(client)) requires 3 arguments: self, event, cb
  • but when we are still writing the call emitter:on("success",, there are only 2 arguments: self, event
  • thus luals didn't perform the more complicated type narrow process
  • matchCall > vm.getExactMatchedFunctions > vm.getMatchedFunctions:

    lua-language-server/script/vm/compiler.lua

    Lines 600 to 611 in 3894968

    	local function matchCall(source)
    	local call = source.parent
    	if not call
    	or call.type ~= 'call'
    	or call.node ~= source then
    	return
    	end
    	local myNode = vm.getNode(source)
    	if not myNode then
    	return
    	end
    	local funcs = vm.getExactMatchedFunctions(source, call.args)

  • it will check the amin, amax (min/max argument) count:

    lua-language-server/script/vm/function.lua

    Lines 430 to 446 in 3894968

    	function vm.getMatchedFunctions(func, args, mark)
    	local funcs = {}
    	local node = vm.compileNode(func)
    	for n in node:eachObject() do
    	if n.type == 'function'
    	or n.type == 'doc.type.function' then
    	funcs[#funcs+1] = n
    	end
    	end
    	local amin, amax = vm.countList(args, mark)
    	local matched = {}
    	for _, n in ipairs(funcs) do
    	local min, max = vm.countParamsOfFunction(n)
    	if amin >= min and amax <= max then
    	matched[#matched+1] = n

• On the other hand, when trying to type narrow during a completion, luals simply checks if there is a string type event name argument and a literal match.
  Therefore when you changed to use alias inside the fun() annotation, the type narrow in completion fails to work 😕
  

  lua-language-server/script/vm/compiler.lua

  Lines 962 to 977 in 3894968

  	local function dealDocFunc(n)
  	local myEvent
  	if n.args[eventIndex] then
  	local argNode = vm.compileNode(n.args[eventIndex])
  	myEvent = argNode:get(1)
  	end
  	if not myEvent
  	or not eventMap
  	or myIndex <= eventIndex
  	or myEvent.type ~= 'doc.type.string'
  	or eventMap[myEvent[1]] then
  	local farg = getFuncArg(n, myIndex)
  	if farg then
  	for fn in vm.compileNode(farg):eachObject() do
  	if isValidCallArgNode(arg, fn) then
  	vm.setNode(arg, fn)

[Bilal2453]

I see. What would be the suggested "fixes"?

I would imagine fixing the completion step is the preferred route here, by allowing alias types / alias strings along side string values. Maybe also make it so that the match all case account for the 1 argument difference?
Not really experienced around this code which appears complex to me😆

Reminder that the alias solution is a hack around the last proposed one, which ends up duplicated event suggestions. This one feels easier to fix, by merging equal values as one suggestion instead of letting it duplicate, this merge should account for things like comments.

[tomlau10]

I would imagine fixing the completion step is the preferred route here, by allowing alias types / alias strings along side string values.

Yes, I agree that this is the preferred route.
I am trying to come up with a fix for this use case, and after some trial and error I think I nailed it 👀 .
I will open a PR for it later after more testing. 😄 @Bilal2453

Here is my work in progress (subject to change): tomlau10@4de84f3