[split from visionik AMA] Database Discussion

[ax0lotl]

Also mySql is not the best solution for such a big system ...
Never worked with it: googleBigTable, Hadoop

A relational database should be fine for FAF. Hadoop is for a whole different magnitude of data.

[Dragonfire]

A relational database should be fine for FAF. Hadoop is for a whole different magnitude of data.

I was not clear:
A relational database is not good for everything in faf, e.g. replays.

[nine2]

A relational database is not good for everything in faf, e.g. replays.

I don't see why not. I heard there are only a million replays.

[Ze_PilOt]

A relational database is not good for everything in faf, e.g. replays.

I don't see why not. I heard there are only a million replays.

Its more than That.

Also you are not querying the replay table.

You are querying a table that has a row for every player in the game.

[Aurion]

A relational database is not good for everything in faf, e.g. replays.

I don't see why not. I heard there are only a million replays.

Its more than That.

Also you are not querying the replay table.

You are querying a table that has a row for every player in the game.

Yet it's possible to push all replay data to a for example a document-based database (in addition to the MySQL database) and it would be way easier and faster to query from that because of the lack of relations (you would push user data with the replay data with which you need to filter). This requires quite some storage, because it's not being stored efficiently of course and databases like that like lots of memory too.

Organise a 'Kickstarter' for new map making tool... v60 scmaps, EnvCubes and custom decals and texture support.

Just throwing that one out there lol!

That would be awesome, but do we actually have the knowledge in this community to make something like that?

[Vee]

Also mySql is not the best solution for such a big system ...
Never worked with it: googleBigTable, Hadoop

was not clear:
A relational database is not good for everything in faf, e.g. replays.

Yet it's possible to push all replay data to a for example a document-based database (in addition to the MySQL database) and it would be way easier and faster to query from that because of the lack of relations (you would push user data with the replay data with which you need to filter).

OMG...

[Dragonfire]

OMG...

Can you specify it more?
Why it is bad?
Why it is good?

[Aurion]

Yet it's possible to push all replay data to a for example a document-based database (in addition to the MySQL database) and it would be way easier and faster to query from that because of the lack of relations (you would push user data with the replay data with which you need to filter).

OMG...

What?

[maov]

I doubt that a nosql solution would be faster then mysql running ad hoc queries on 'min rating', 'player', 'map' order by date with limit xxx on one machine only. It all depends on the criteria for the search though; is it static data(write and never update), will the search method change in the future, how far can it be denormalized without forcing large codechanges/data migration, is simple backup important (put the replays in the filesystem).

The only good way to find out is to replicate/create good test data, run a few tests on a "small" machine or you limit it with docker/virtual machine, double the size of the data and see if its still running well. This is again only valid if the system is not doing anything else though so ... I´ve had my share of optimizing DB systems (Postgresql & some MSSQL) and we are running a cassandra setup with spark at my current work, but faf is not really running on a standard setup, you would normally have far more ram, faster disks and dedicated machines (DB on its own machine with fast network).