Forged Alliance Forever

Re: export hotstats to html

2013-10-24T11:00:06+02:00

I don't get it, I put the modded hotstats.zip and replace my original one.

Though I have no log generated. I select the replay from FAF interface in the replay tab, start the replay, wait a bit and then get to the scores screen.

The file replay.log is in the logs folder, but is only filled with the standard information from FAF client. No record or whatsoever about hotstats.

Statistics: Posted by -_V_- — 24 Oct 2013, 11:00

Re: export hotstats to html

2013-01-19T04:01:16+02:00

It appears the game can make logs with 0.1 second incriment, by changing hotstats the way you mentioned before.
The new version of FAF-STATS can't be changed the way you mentioned before and log files with 0.1 second incriments are not read.
Would be nice to have them read.

Looks like when the log file gets bigger that about 10 mb, it can't be handled by the program. Is it possible to fix this issue?

1 second intervals are already large and if you make the interval any bigger to cut down on file size it only makes things worse.
(if you integrate mass rate in with 1 second step, you underestimate the actual mass income by a noticeable amount, enough to make data analysis problematic)

Statistics: Posted by SeraphimLeftNut — 19 Jan 2013, 04:01

Re: export hotstats to html

2013-01-18T23:17:38+02:00

Analogous power data for the same game, same player.

Looks like someone was power stalling. (that removes the oscillations)

Statistics: Posted by SeraphimLeftNut — 18 Jan 2013, 23:17

Re: export hotstats to html

2013-01-18T23:15:48+02:00

Mass use vs time data has a lot of oscillations in it (see first image attached), that correspond to factories going on and off as units come out of them. The frequency of this oscillation is dependent on the unit being produced, build capacity associated with that factory and available resources.
Second image is FFT of this data(sliding a 3 minute window, one second at a time). Red color is zero, blue color is intense. Vertical axis is the detected frequencies going from low to high(bottom to top).
As the game progresses you see more and more high frequencies, which are associated with air production and increased build capacity on factories being used.

The oscillations in power are also present.

Unfortunately I only have access to data from a single random game right now. I expect these plots to be very telling in setons games, especially for the air players. (allies that leech off allies for power will have the oscillations of those they leech from imprinted on their eco)

Also the plots would be much better if the sampling rate wasn't 1 second. This hides a lot of things, by forcing the use of a wide 3 minute window, which smears out dynamic events.

Statistics: Posted by SeraphimLeftNut — 18 Jan 2013, 23:15

Re: export hotstats to html

2013-01-18T18:10:48+02:00

fffmmm wrote:
JSON objects are easily parsed using python, thus I think it's a good choice (Even easier to parse than CSV).

String[] strArr=line.split(",");
if you only have numbers in the array which is the case. otherwise there are full blown csv parsers for any programming language.

fffmmm wrote:
Writing the snapshot regularly to the log would cause a lot of IO ops and make it more difficult to parse I think.

you can still write the data after the game if you want.
anyway, if you want to do it the wrong way, i dont wanna argue.

fffmmm wrote:
I tried to find a way to get the reclaim for everyone, but sadly I only found a function to get it for ones own economy, but not for everyones eco.

then maybe its not possible, i was just assuming since gazui can do it, hotstats should be able to also.

Statistics: Posted by rootbeer23 — 18 Jan 2013, 18:10

Re: export hotstats to html

2013-01-18T17:58:10+02:00

For me the ascii format, is the format of freedom.
This is awesome

Statistics: Posted by SeraphimLeftNut — 18 Jan 2013, 17:58

Re: export hotstats to html

2013-01-18T17:51:48+02:00

rootbeer23 wrote:
i would prefer to see mass reclamation and mass income from extractors and fabricators seperately.

in dump.lua, you should write a single line of LOG() for each tick that is just a collection of values, instead of
dumping the whole table, including the keys (of the lua associative array).
i.e. dump a csv with the values
s["resources"]["massin"]["total"], s["resources"]["massin"]["rate"], ...

finally i suggest to parse the replay.log with python and thus make the parser integral part of falobby and then add a new tab to falobby that shows the resulting html.

JSON objects are easily parsed using python, thus I think it's a good choice (Even easier to parse than CSV).
Writing the snapshot regularly to the log would cause a lot of IO ops and make it more difficult to parse I think.

I tried to find a way to get the reclaim for everyone, but sadly I only found a function to get it for ones own economy, but not for everyones eco.

Statistics: Posted by fffmmm — 18 Jan 2013, 17:51

Re: export hotstats to html

2013-01-18T17:41:12+02:00

or better yet, save the values in a flat array while accumulating the scores instead of copying the scoreData into the historical array. that whould cut down on RAM waste by several factors.

Statistics: Posted by rootbeer23 — 18 Jan 2013, 17:41

Re: export hotstats to html

2013-01-18T16:59:12+02:00

i would prefer to see mass reclamation and mass income from extractors and fabricators seperately.

in dump.lua, you should write a single line of LOG() for each tick that is just a collection of values, instead of
dumping the whole table, including the keys (of the lua associative array).
i.e. dump a csv with the values
s["resources"]["massin"]["total"], s["resources"]["massin"]["rate"], ...

finally i suggest to parse the replay.log with python and thus make the parser integral part of falobby and then add a new tab to falobby that shows the resulting html.

Statistics: Posted by rootbeer23 — 18 Jan 2013, 16:59

Re: export hotstats to html

2013-01-18T15:47:29+02:00

SeraphimLeftNut wrote:
The key missing statistics are mass and energy being spent as a function of time. Incomes are nice, but usage is much more interesting.
Looks like the time step can't be less than 0.5 seconds based on your suggestion, but that is already pretty good.

There also seems to be an upper limit on log size that the program is able to handle, which excludes long setons games.

The x hr, x min, x sec label also complicates data import from the source file. In places where you have 1 hour 30 seconds, it would be nicer if it said 1 hour, 0 minutes, 30 seconds.

I updated it a little:
-you can now export the data to a CSV-File (Not sure if the data in the CSV is organized that well... Let me know if that needs to be adjusted)
-it now acquires a stats snapshot every second instead of every 10 seconds
-added a chart for the mass spending rate and the energy spending rate (overcharges look great on that one)
-changed the way it prints the time so that it always prints it in the format Xh Xm Xs

peanot wrote:
Thanks, I will see if I can get an interactive prototype running soon.

Oh also, can someone tell me where I can find the generated log file? I seem to be missing something obvious here.

You have to enable logging in the lobby (Options->settings->save game logs). The logs can be found in the following directory: C:\ProgramData\FAForever\logs

Statistics: Posted by fffmmm — 18 Jan 2013, 15:47

Re: export hotstats to html

2013-01-18T11:56:18+02:00

Thanks, I will see if I can get an interactive prototype running soon.

Oh also, can someone tell me where I can find the generated log file? I seem to be missing something obvious here.

Statistics: Posted by peanot — 18 Jan 2013, 11:56

Re: export hotstats to html

2013-01-18T06:08:36+02:00

The key missing statistics are mass and energy being spent as a function of time. Incomes are nice, but usage is much more interesting.
Looks like the time step can't be less than 0.5 seconds based on your suggestion, but that is already pretty good.

There also seems to be an upper limit on log size that the program is able to handle, which excludes long setons games.

The x hr, x min, x sec label also complicates data import from the source file. In places where you have 1 hour 30 seconds, it would be nicer if it said 1 hour, 0 minutes, 30 seconds.

Statistics: Posted by SeraphimLeftNut — 18 Jan 2013, 06:08

Re: export hotstats to html

2013-01-18T00:44:18+02:00

Thanks for the pointers and for the hotstats modification!
I am very glad I do not have to figure out lua. That will probably save me many hours.

@DilliDalli:
I am afraid I do not have enough time for a(nother) "proper job".
Studying is KIND OF a fulltime job, and I have enough personal projects.
(Though, if LOTS of money was involved... hehehe)

But yeah, I think I will see tomorrow or maybe over the weekend what I can figure out. If it is super easy to decode that JSON output into internal objects(and it should be), then I could be drawing the first graphs within a few hours.

Again, the question really is what kind of graphs or other features would be useful.
But yeah, not trying to pressure anyone here, I know I just asked a few hours ago.

Statistics: Posted by peanot — 18 Jan 2013, 00:44

Re: export hotstats to html

2013-01-18T00:28:13+02:00

peanot wrote:
Oh and also, let me know if you mind that I am using your thread for this fffmmm, if so I will happily take this some place else.
Also, if you yourself want to do something like that, we probably should communicate and not both do the same thing.
Though I guess competition can be a good thing?

You can of course use this thread in any way you want: I don't claim a monopoly on it
If you plan to do do your tool from scratch, use this modified version of hotstats: https://www.dropbox.com/s/7cmyot5l27i4bdu/hotstats.zip
It dumps the objects as JSON String/Object which should massively reduce the time neccessary to parse and arrange the data (usualy all it takes is a JSON.parse(string) - depends on the language you're gonna use) and it takes a snapshot every second instead of every 10 seconds.

PS: to get the correct mass/energy rate you will have to multiply the stored rates by 10
PS2: this one can't be parsed by the tool I originally posed as I dump it in some custom format (didn't quite think that one trough b4 I started it)

~fffmmm

Statistics: Posted by fffmmm — 18 Jan 2013, 00:28

Re: export hotstats to html

2013-01-18T00:06:01+02:00

Programming? Sounds like you need to speak to Zep if you want a proper job

Statistics: Posted by Softly — 18 Jan 2013, 00:06