Lua / C++ DLL linking - Multi core use

[Franck83]

It will be nice if we ressurected this project (can't do it, i've not c++ knowledge):

https://github.com/FAForever/supcom-se/ ... /scse/scse

I seems that we can use DLL to dispatch some intensive CPU task between all CPU cores. Of course we cannot interrupt sim execution to wait DLL results.

But it may be cool to unlock all this cores power to mod dev. I'm thinking about all in game units dataprocessing....

[nine2]

I reckon that will never work.

The engine needs a rewrite, not some magic dll.

The benefits of multi core has always been overstated.

[Franck83]

I apologise if I was misinterpreted anihilnine, but i was not talking about unlocking multicore as a main feature or refining supcom engine with dll.

Just say that i would have use of some multicore power from dll for modding.

[nine2]

Sorry for being so quick to be negative.

I could be wrong but my understanding is
- multicoring is achieved through threading
- threading will desync immediately

[Franck83]

You can do some task in background with the DLL (ie during game execution send all the unit data and in background process them (tree...))

But you right about desync with other players since no everybody got the same CPU power. We are always linked to the slowest CPU.

[Uveso]

Maybe i am the only one with this opinion, but CPU power ist not the mainproblem.

Why we have absurd fast CPU's for Supcom and still lag!?!
While having some PC here i did some testing.

Surprise surprise, it's RAM speed.

For example; i changed from DDR3 to DDR4 RAM (I have a board with DDR3 and DDR4 slots).
The simspeed was increasing from +5 to +6
I got also good results when overclocking the HT-link (AMD -> connection between Northbridge and RAM) or overclocking the RAM itself (low CAS Latency, 4xBank Interleaving, reduced refresh cycle(yeah, i know, i know... ))

Maybe we should try to not waste too much LUA memory in huge arrays
(Memory use from big unit models, LOD settings etc are not influencing the game speed)

Some benchmarks to compare:
18328 QuadCore Q8400 @ 2,5GHz
18543 FX-8150 @ 4,2GHz
18545 PhenomII-X4-955 @ 3,8GHz / 200MHz
18547 PhenomII-X4-955 @ 3,8GHz / 211MHz
18559 PhenomII-X4-955 @ 4,0GHz / 200MHz
18576 PhenomII-X4-955 @ 4,2GHz / 205MHz
18585 PhenomII-X4-955 @ 4,1GHz / 200MHz NB 2200
18589 PhenomII-X4-955 @ 4,0GHz / 200MHz NB 2400
18595 PhenomII-X4-955 @ 4,1GHz / 200MHz NB 2400
18645 i7 6700K @ 4.4GHz
18708 i7 4790K @ 4.4GHz

[Franck83]

Surprise surprise, it's RAM speed.

You totally right, i switched my Ram speed from DDR3-1600 to DDR3-2133, played with the timings and i saw + 10-15% fps increase.

[Katharsas]

Sorry for being so quick to be negative.

I could be wrong but my understanding is
- multicoring is achieved through threading
- threading will desync immediately

Your understanding is wrong. For the game to not desync, there is a simple condition: Every player, provided with the game state of the previous tick and all actions of all players to be processed, needs to be able to calculate the game state at the next tick deterministically (-> have same result on any hardware/software). This is perfectly achievable with various forms of multi-threading, for example:

Spoiler: show

Say we have a given game state and player input, and x0, y0, u, v, x1, y1 are all just numbers / variables:

game_state_at_tick_0 = { unit_A_position : x0, unit_B_position : y0 }
player_input_at_tick_0 = move unit_A for u, move unit_B for v

Then we use single-threaded simulation code to cacluclate next game state:
Main thread:
unit_A_pos_new = x0 + u = x1
unit_B_pos_new = y0 + v = y1
game_state_at_tick_1 = { unit_A_position : x1, unit_B_position : y1 }

or we could use muti-threading:
Thread 1:
unit_A_pos_new = x0 + u = x1
Thread 2 (at the same time):
unit_B_pos_new = y0 + v = y1
Main thread (waiting until Thread 1 & 2 have finished and then collects results):
game_state_at_tick_1 = { unit_A_position : x1, unit_B_position : y1 }

Both methods will have calculated an identical, deterministic result, and there is no desync in the simulation. You could even run clients with and without multi-threading in same game. As long as they calculate the same things, everything is fine. WHO FAST this happenes, does not matter, because faster clients simply wait for the slowest until they proceed with next sim tick.


Now, of course things get more complicated if we get into Lua coroutines and how they could be multithreaded, and i don't think that would work very well. My understanding is that Lua supports coroutines in the same way that for example Javascript supports async function execution. BUT this is very different from real multithreading, because it doesn't require REAL synchronization, locks, semaphores or whatever you need when working with state that is shared accross multiple threads.
In javascript async function (unless using webworker), you never need to worry about what a will happen to a variable when two threads try to write to it ast the SAME time, because even if functions are async, they can never be executed at the same time. because the Javascript executor runs single-threaded, like a Lua host does as well i guess.
But maybe you can to very clever things in the Lua Host that the game using. Wich leads me to the question:

Which Lua Host does the game use ot execute Lua? There are massive differences in performance between some of them, There is for example this:
http://luajit.org/

Which shows massive performance improvements in regard to control structures (control structures are things like "if", "for", "while", etc..), and probably also for pure calculations, but probably does not help too much if the Lua code is bottlenecked by memory access / bad cache layout.

Edit:
I think i misunderstood Franck. He doesn't want to parallelize the Lua execution, he simply wants to call native code FROM Lua (and this native code can, of course, be multi-threaded). That should work fine.

[nine2]

This is perfectly achievable with various forms of multi-threading

I think it's ambitious to do that
1. with an external dll call
2. without a desync
3. through all of the engine hoops
4. with it actually causing a performance increase

[Katharsas]

This is perfectly achievable with various forms of multi-threading

I think it's ambitious to do that
1. with an external dll call
2. without a desync

The guy in the other thread that Franck linked had the external dlls working i think, being called by Lua code, and while desync may be a problem, it has nothing to do directly with multithreading, as explained.

3. through all of the engine hoops
4. with it actually causing a performance increase

These are the real problems. Making some threads and joing them back in every tick will probably be slower than single-threaded. If it's somehow possible to have a thread-pool with some workers, that live over the course of multiple ticks, it could work though. Still, i see the best shot at improving Lua performance in switching out the Lua interpreter in favor of LuaJit, and, like Usevo said, optimizing memory layout for more cache hits, if that is possible in Lua besides just using less memeory.