This is not a recommendation, but just a couple of days ago someone linked to this project, claiming similar goals to lua, great performance, and gradual typing:

https://cyberscript.dev/

I can’t tell you what it’s actually like though.

A more established, proven option is Haxe. Haxe has a lot of libraries but I think it’s specifically designed to be batteries-optional. This Haxe VM in particular looks pretty impressive:

https://hashlink.haxe.org/

Haxe has the distinction of having been used to ship loads of successful games made by small teams with custom engines.

Another option designed for simplicity, low-complexity and easy embedding is wren:

https://wren.io/

Implementation is apparently only 4000 lines.

Check out scheme. The syntax is incredibly simple. Watch like the first 20 minutes of a first sicp lecture, and you’ll know enough lisp to read basic scheme.

Macros are magic, and I’m guessing they would be really useful for you to abstract interactions with the engine. Check out the for macro and how it’s compared to the standard loop macro. Both implement a DSL to make iteration both simpler and easier to read (if you understand the specific language of the macro).

• open source

• Licence will depend on the implementation, but I think most are open source. Guile was the GNU champion in the embedded language war. Also, you can go full GNU with GNU make, GCC, and GNU guile.

• small footprint even if it at the cost of some complexity (I need it as a scripting engine, not as a replacement for compiled application languages)

• Scheme’s whole thing is to be minimal, while letting you make anything you need. For example the r5rs specification is 50 pages long, and I think has recursion as the only iteration method. Specific implementations add additional features like while loops.

• integer support (I don’t care if I could just round it on the backend)

• I’m not sure what you mean by int support, but there should be exact/inexact and “convert between them” functions for numbers. Check out the scheme specs and implementation docs for specifics.

• C or C++ ABI

• interop also depends on implementation, but guile should be pretty good.

• can be embedded into a software (yes, there are people that suggest you to use janky solutions like passing data in files and command line)

• there are a few schemes made specifically for embedding

Documentation is really varied. Also, schemes usually don’t have all that much online info available, but the bigger ones have good communities.

Just to warn you, you’re probably going to end up using Emacs if you decide to work with scheme. Doom Emacs is great. For magic check out lispy and paredit.

I know not what I talk about, but maybe daScript?

I’ve used Squirrel a bit ages ago and it worked quite well. Similar to Lua, but OOP and more C-style syntax.

I’m surprised Lua isn’t working out for you, though. It seems like such a simple language. Although I just admit I’ve never used it before.

I think Wren checks your boxes.

I have used DukTape in the past, it is an ECMAScript (javascript) implementation. The API is amazing, very consistent parameter and naming schemes, uses a stack based approach (feels like a better version of how lua goes about things). It is designed to be embedded. Documentation for the API is some of the best you will see, lots of good examples. It even has a debugger that you can hook into.