How should FFAs be rated?

[Softly]

So there was some discussion lately on whether or not FFAs should be ranked, with the motivation being that they don't necessarily change points "accurately", for some value of accurately.

My feeling was that we dodged the issue of how we want the rating system to work for FFAs, and whether or not it is actually working that way at the moment. So, I decided to investigate.

One caveat: I am assuming that scoring from games works the same way now as it did ~4 years ago (probably true though).

What I found is that the points you get in an FFA, depend on where you came in the FFA (obviously). Also, where you come in the FFA depends on a) whether or not you won (duh), and b) how many ACUs you killed (wtf?!?).

Example: 4 player FFA between players A, B, C, D
Player A wins, killing players B and C.
Player B also kills player D.

Then the rankings are as follows: 1st: A, 2nd: B, 3rd: C + D.
Assuming the players have a rating deviation of 60 initially (roughly what most players will have) and have 1500 mean, then under the current rating settings the new ratings are:
A => mu=1515.849, sigma=59.863
B => mu=1502.564, sigma=59.620
C => mu=1490.748, sigma=59.579
D => mu=1490.748, sigma=59.579

Compared to the case where B,C,D are counted as "equal" seconds, in which case the ratings would be:
A => mu=1511.560, sigma=59.905
B => mu=1496.084, sigma=59.494
C => mu=1496.160, sigma=59.492
D => mu=1496.160, sigma=59.492

My question: Is this the intended behaviour? (I don't think so)

Also, for my own amusement, here's what happens if A and C are 2k rated players in the above example (same sigma).
A 1st, B 2nd, C + D 3rd:
A => mu=2006.133, sigma=60.181
B => mu=1515.351, sigma=59.665
C => mu=1974.190, sigma=59.592
D => mu=1504.326, sigma=59.598

and A 1st, B,C,D 2nd:
A => mu=2003.636, sigma=60.325
B => mu=1508.795, sigma=59.543
C => mu=1978.739, sigma=59.539
D => mu=1508.831, sigma=59.546

In each case C seems to get savaged, and B + D profit nicely

What do people think?

[Softly]

Also the code (in python3.5) for people who want to try this at home:

Code: Select all

#testing trueskill for FFAs
import trueskill
from trueskill import Rating
trueskill.setup(mu=1500, sigma=500, beta=240, tau=10, draw_probability=0.10)

r1 = Rating(mu=2000,sigma=60)
r2 = Rating(mu=1500,sigma=60)
   
print(trueskill.rate([[r1],[r2],[r1],[r2]],ranks=[0,1,2,2]))

trueskill settings taken from: https://github.com/FAForever/server/blo ... fig.py#L16

[JaggedAppliance]

One caveat: I am assuming that scoring from games works the same way now as it did ~4 years ago (probably true though).

It was changed a bit about 3 weeks ago: https://github.com/FAForever/server/com ... 2c16ce7eca
Here's the thread with the info on why it was changed: viewtopic.php?f=45&t=11698&hilit=axle

As for your post, I had no idea kills decided where you placed in an FFA, that sounds bizarre.

[Softly]

For clarity, when I say "scoring from games" I mean how the game reports the score to the server. Last I checked (years ago) it was 1 point per ACU kill, -1 per death, and 5 for surviving.

This had some odd consequences, for example ctrl+k in a 1v1 to kill the opponent would lead you to win (your score is 0, the opponent -1).

In teamgames the team score is aggregated, so everyone on the same team gets the same treatment regardless of who got the kills.

[PhilipJFry]

Why does FFA have to be rated at all?

Spoiler: show

inb4 "diplomacy"

[Morax]

I agree with Philip... FFA is kind of a weird game mode that adds too much variation to the global rating of players.

[Softly]

I agree with Philip... FFA is kind of a weird game mode that adds too much variation to the global rating of players.

What do you mean by variation? If you have some specific cases you want to discuss, let me know and I will post the exact numbers.

Personally, I think the best argument is coming up with results like this:

3 way FFA, two 1.6k (A and B) players and a 1.4k player (C) (as ever, with sigma=60 for all players).

If player C ctrl+ks 10 seconds into the game, they gain points (1.9 points to be exact).

This effect increases with a rating gap, suppose we have A and B on 2k and C on 1k. In that case player C gains 29 points (Nice return for a 10 second ctrl+k).

[Exotic_Retard]

well in ffa you tend to go to the highest threat, so lets say you are lowest rated player in the game, just sit there being useless. and someone else dies. great now you got rating for free. in my case i died second and gained 20 rating. was a long time ago so dont know the details but yeah

another example was when speed2 lost 40 rating cos i built a czar and killed everyone at once. his rating was decided completely by me. i could have killed in any order but i had a nice outcome on his rating. it was me violet, speed and someone else like sheeo. i also got like 30-40 rating

in any case its strange since you get 10x the rating change from an ffa than you get from a teamgame.

[everywhere116]

That's probably because you're on your own and not on a team. I noticed that global 1v1s affect your rank a lot more than teamgames do as well.

[Softly]

Maybe someone can explain why identically ranked players get different numbers of points for the same score?

Code: Select all

#testing trueskill for FFAs
import trueskill
from trueskill import Rating
trueskill.setup(mu=1500, sigma=500, beta=240, tau=10, draw_probability=0.10)

r = Rating(mu=1500,sigma=60)
   
res = trueskill.rate([[r],[r],[r],[r],[r],[r],[r],[r],[r],[r],[r],[r]],ranks=[0,1,1,1,1,1,1,1,1,1,1,1])
for rating in res:
    print(rating)

"""
Results:
(trueskill.Rating(mu=1512.769, sigma=59.718),)
(trueskill.Rating(mu=1498.466, sigma=59.174),)
(trueskill.Rating(mu=1498.578, sigma=59.162),)
(trueskill.Rating(mu=1498.676, sigma=59.153),)
(trueskill.Rating(mu=1498.761, sigma=59.147),)
(trueskill.Rating(mu=1498.834, sigma=59.144),)
(trueskill.Rating(mu=1498.895, sigma=59.144),)
(trueskill.Rating(mu=1498.945, sigma=59.146),)
(trueskill.Rating(mu=1498.985, sigma=59.151),)
(trueskill.Rating(mu=1499.014, sigma=59.158),)
(trueskill.Rating(mu=1499.033, sigma=59.169),)
(trueskill.Rating(mu=1499.043, sigma=59.184),)
"""