Stat collection - need some lua-fu

Post here if you want to help developing something for FAF.

Stat collection - need some lua-fu

Postby uberge3k » 13 Dec 2014, 06:30

I would like to get some proper stat data collected from games, but am not having much luck finding where I can access the unit information that I need. To start with, I would like to log information each time a unit is created or destroyed. Any lua experts willing to lend a hand? :)

- OnCreate
- A table of unit IDs that assisted with its construction (eg, t1 land factory, 5 t1 engineers, ACU). Later on it would be cool if we could break down the percentage that each unit type contributed to its completion, but just getting the list of units that were assisting with construction as it completed should work for 95% of common usage scenarios.

- OnKilled
- A table of unit IDs that assisted with its destruction. As before, ideally each entry would also contain the percentage of health that it contributed to the kill. Alternatively, the unit that landed the killing blow should be enough for most cases.

I have approximately ten thousand ideas for other stats that I want to track, but these two are the most important to get some minimum working prototype in order and be able to start analyzing data in some nontrivial way. I know how to override these functions in Unit.lua, but I'm not sure whether the bits of information that I need are already there or not.

For exporting the data, eventually I would like to simply store each record in a table, and send that table to a network service at the end of the game. For now, I'm content to just use the Log command and parse the log file with another tool while I'm testing.

[edit]: Answered part of my own question myself; GetGameTime() looks like it will be sufficient for recording the time that the event occurred.
[edit2]: Evidently that's only a part of the UI layer, not the sim layer. My attempts at hacking in a timer by forking a thread, calling WaitSeconds and incrementing a value have so far been met with failure.
Ze_PilOt wrote:If you want something to happen, do it yourself.
User avatar
uberge3k
Supreme Commander
 
Posts: 1034
Joined: 04 Sep 2011, 13:46
Has liked: 2 times
Been liked: 48 times
FAF User Name: TAG_UBER

Re: Stat collection - need some lua-fu

Postby nine2 » 13 Dec 2014, 10:08

uberge3k wrote:I would like to get some proper stat data collected from games, but am not having much luck finding where I can access the unit information that I need. To start with, I would like to log information each time a unit is created or destroyed. Any lua experts willing to lend a hand? :)

- OnCreate
- A table of unit IDs that assisted with its construction (eg, t1 land factory, 5 t1 engineers, ACU). Later on it would be cool if we could break down the percentage that each unit type contributed to its completion, but just getting the list of units that were assisting with construction as it completed should work for 95% of common usage scenarios.

- OnKilled
- A table of unit IDs that assisted with its destruction. As before, ideally each entry would also contain the percentage of health that it contributed to the kill. Alternatively, the unit that landed the killing blow should be enough for most cases.

I have approximately ten thousand ideas for other stats that I want to track, but these two are the most important to get some minimum working prototype in order and be able to start analyzing data in some nontrivial way. I know how to override these functions in Unit.lua, but I'm not sure whether the bits of information that I need are already there or not.

For exporting the data, eventually I would like to simply store each record in a table, and send that table to a network service at the end of the game. For now, I'm content to just use the Log command and parse the log file with another tool while I'm testing.

[edit]: Answered part of my own question myself; GetGameTime() looks like it will be sufficient for recording the time that the event occurred.
[edit2]: Evidently that's only a part of the UI layer, not the sim layer. My attempts at hacking in a timer by forking a thread, calling WaitSeconds and incrementing a value have so far been met with failure.


I have all of this done already. As well as a tool that can parse the log file and watch a virtual replay in my own tool. I have unit movement, reclaim amounts, mass income, etc. You can rewind in my replay viewer, and do other nice things... like your 10000 ideas you mentioned. Will share soon.

Anyway just make a new sim thread that every tick dumps the stats of every unit to the log file. I keep track of the last value of each stat so I only dump changes not everything every frame. Prepare for huge log files. What I don't do is override functions of unit like you say because you have to understand the game a lot that way ... if you just dump everything every second then you don't need to know how to tap into the event say when something is captured - instead the armyid of the unit will just change and you will pick that up. In fact if you do it my way you will pick up things you don't even know about.
nine2
Councillor - Promotion
 
Posts: 2416
Joined: 16 Apr 2013, 10:10
Has liked: 285 times
Been liked: 515 times
FAF User Name: Anihilnine

Re: Stat collection - need some lua-fu

Postby nine2 » 13 Dec 2014, 10:12

run GoThread() in a new sim thread:

Spoiler: show
Code: Select all
local displayedTick = false
local lastUnitIds = {}
local brains = {}
local stps = 10

local lastReclaimableIds = {}

function GoThread()
   local wholeMap = Rect(0,0,10000,10000)
   while true do
      import('/lua/sinstr.lua')
      displayedTick = false
      showTick()

      local thisUnitIds = {}
      local thisReclaimableIds = {}

      #### units

      for ak, av in ListArmies() do

         local brain = brains[ak]
         if (brain == nil) then
            brain = GetArmyBrain(ak)
            brains[ak] = brain
            brain.dcx_props = {}
            brain.armyId = ak
         end

         local units = brain:GetListOfUnits(categories.ALLUNITS, false)
         for k,v in units do

            thisUnitIds[v:GetEntityId()] = true

            if lastUnitIds[v:GetEntityId()] == nil then
               dcxEntry("D", v:GetEntityId(), "Created2")
            end

            for ek, ev in v.dcx_events do
               dcxEntry("UE", v:GetEntityId(), ev)
               v.dcx_events[ek] = nil
            end

            local p =  v:GetPosition()
            compareUnit("UP", v, v:GetBlueprint().BlueprintId, "BlueprintId")
            compareUnit("UP", v, v:GetEntityId(), "Id")
            compareUnit("UP", v, v:GetArmy(), "Army")
            compareUnit("UP", v, math.floor(p.x), "X")
            compareUnit("UP", v, math.floor(p.y), "Y")
            compareUnit("UP", v, math.floor(p.z), "Z")
            compareUnit("UP", v, math.floor(v:GetMaxHealth()), "MaxHealth")
            compareUnit("UP", v, math.floor(v:GetHealth()), "Health")
            compareUnit("UP", v, v:GetBlueprint().StrategicIconName, "StrategicIconName")
         end

         

         compareArmy(brain, brain:GetEconomyIncome("MASS") * stps, "MassIncomePerSecond")
         compareArmy(brain, brain:GetEconomyIncome("ENERGY") * stps, "EnergyIncomePerSecond")

         compareArmy(brain, brain:GetArmyStat("Economy_Reclaimed_Mass", 0.0).Value, "TotalReclaim")
         compareArmy(brain, brain:GetArmyStat("Economy_income_reclaimed_Mass", 0.0).Value, "Economy_income_reclaimed_Mass")
         compareArmy(brain, brain:GetArmyStat("Economy_AccumExcess_Mass", 0.0).Value, "Economy_AccumExcess_Mass")
         compareArmy(brain, brain:GetArmyStat("Economy_Output_Mass", 0.0).Value, "Economy_Output_Mass")
         compareArmy(brain, brain:GetArmyStat("Economy_TotalConsumed_Mass", 0.0).Value, "Economy_TotalConsumed_Mass")
         compareArmy(brain, brain:GetArmyStat("Economy_Income_Mass", 0.0).Value, "Economy_Income_Mass")
         compareArmy(brain, brain:GetArmyStat("Economy_TotalProduced_Mass", 0.0).Value, "Economy_TotalProduced_Mass")

      end

      for k,v in lastUnitIds do
         if thisUnitIds[k] == nil then
            dcxEntry("UE", k, "Killed")
         end
      end
      lastUnitIds = thisUnitIds

      #### reclaim

      local reclaimables = GetReclaimablesInRect(wholeMap)
      for k, v in reclaimables do
         if v.MassReclaim ~= nil or v.EnergyReclaim ~= nil then

            thisReclaimableIds[v:GetEntityId()] = true
         
            if lastReclaimableIds[v:GetEntityId()] == nil then
               dcxEntry("RE", v:GetEntityId(), "Created2")
            end

            local p =  v.CachePosition
            compareUnit("RP", v, v:GetBlueprint().BlueprintId, "BlueprintId")
            compareUnit("RP", v, v:GetEntityId(), "Id")
            compareUnit("RP", v, math.floor(p.x), "X")
            compareUnit("RP", v, math.floor(p.y), "Y")
            compareUnit("RP", v, math.floor(p.z), "Z")
            compareUnit("RP", v, math.floor(v.EnergyReclaim), "EnergyReclaim")
            compareUnit("RP", v, math.floor(v.MassReclaim), "MassReclaim")
            compareUnit("RP", v, math.floor(v.MaxEnergyReclaim), "MaxEnergyReclaim")
            compareUnit("RP", v, math.floor(v.MaxMassReclaim), "MaxMassReclaim")
         end
      end
      
      for k,v in lastReclaimableIds do
         if thisReclaimableIds[k] == nil then
            dcxEntry("RE", k, "Killed")
         end
      end
      lastReclaimableIds = thisReclaimableIds

      #### end

      WaitSeconds(1)
   end

end

function compareUnit(prefix, unit, val, name)
   local key = "dcx_last"..name
   if val ~= unit[key] then
      unit[key] = val
      dcxEntry(prefix, unit:GetEntityId(), name, val)
   end
end

function compareArmy(armyBrain, val, name)
   local key = "dcx_last"..name
   if val ~= armyBrain[key] then
      armyBrain[key] = val
      dcxEntry("AP", armyBrain.armyId, name, val)
   end
end

function showTick()
   local tick = GetGameTick()
   displayedTick = true
   dcxEntry("T", tick)
end

function dcxEntry(type, a, b, c)
   LOG(type .. "," .. (a or "") .. "," .. (b or "") .. "," .. (c or ""))   
end


random snippet from log file example:
Spoiler: show
info: T,1,,
info: D,0,Created2,
info: UE,0,Created,
info: UP,0,BlueprintId,uel0001
info: UP,0,Id,0
info: UP,0,Army,1
info: UP,0,X,14
info: UP,0,Y,6
info: UP,0,Z,241
info: UP,0,MaxHealth,12000
info: UP,0,Health,12000
info: UP,0,StrategicIconName,icon_commander_generic
info: AP,1,MassIncomePerSecond,0
info: AP,1,EnergyIncomePerSecond,0
info: AP,1,TotalReclaim,0
info: AP,1,Economy_income_reclaimed_Mass,0
info: AP,1,Economy_AccumExcess_Mass,0
info: AP,1,Economy_Output_Mass,0
info: AP,1,Economy_TotalConsumed_Mass,0
info: AP,1,Economy_Income_Mass,0
info: AP,1,Economy_TotalProduced_Mass,0
info: D,1048576,Created2,
info: UE,1048576,Created,
info: UP,1048576,BlueprintId,ual0001
info: UP,1048576,Id,1048576
info: UP,1048576,Army,2
info: UP,1048576,X,241
info: UP,1048576,Y,6
info: UP,1048576,Z,14
info: UP,1048576,MaxHealth,11000
info: UP,1048576,Health,11000
info: UP,1048576,StrategicIconName,icon_commander_generic
info: AP,2,MassIncomePerSecond,0
info: AP,2,EnergyIncomePerSecond,0
info: AP,2,TotalReclaim,0
info: AP,2,Economy_income_reclaimed_Mass,0
info: AP,2,Economy_AccumExcess_Mass,0
info: AP,2,Economy_Output_Mass,0
info: AP,2,Economy_TotalConsumed_Mass,0
info: AP,2,Economy_Income_Mass,0
info: AP,2,Economy_TotalProduced_Mass,0
info: AP,3,MassIncomePerSecond,0
info: AP,3,EnergyIncomePerSecond,0
info: AP,3,TotalReclaim,0
nine2
Councillor - Promotion
 
Posts: 2416
Joined: 16 Apr 2013, 10:10
Has liked: 285 times
Been liked: 515 times
FAF User Name: Anihilnine

Re: Stat collection - need some lua-fu

Postby nine2 » 13 Dec 2014, 10:12

and p.s. when something dies, that id gets reused
nine2
Councillor - Promotion
 
Posts: 2416
Joined: 16 Apr 2013, 10:10
Has liked: 285 times
Been liked: 515 times
FAF User Name: Anihilnine

Re: Stat collection - need some lua-fu

Postby Anaryl » 13 Dec 2014, 17:15

Just take a look at the code inside PlayerTrack mod - that will show you how to pull all the stats you need from live game states and send them wherever you want.
Image
User avatar
Anaryl
Banned
 
Posts: 453
Joined: 20 Dec 2011, 18:54
Location: Rohangrad
Has liked: 13 times
Been liked: 6 times
FAF User Name: Anaryl


Return to Contributors

Who is online

Users browsing this forum: No registered users and 1 guest