perf: lazy load MGUMatch/MGUGame fields

[ElectricalBoy]

Summary

See #7398 for context.
Some fields in MGUMatch/MGUGame are not used in certain contexts, e.g., a lot of extradata fields in MGUGame are not needed during match results processings in standings tables. This PR adds lazy loading to such fields to reduce overall memory consumption.

How did you test this change?

https://liquipedia.net/valorant/User:ElectricalBoy/Sandbox2

	Live	PR
Runtime	0.751 s	0.716 s
Memory	41.48 MB	36.38 MB

[mbergen]

Can you test how this affects MatchTable? IIRC that's also using this.

[ElectricalBoy]

Can you test how this affects MatchTable? IIRC that's also using this.

4508f5e

[Rathoz]

While I like the approach, since so many m2g will need to be loaded, it may be worth batching them rather than going back and forth with the database. While the Lua time difference is negligible, the real-time difference can add up. Also, we need to look at not loading the m2g extradata field to reduce memory further

[ElectricalBoy]

While I like the approach, since so many m2g will need to be loaded, it may be worth batching them rather than going back and forth with the database.

because lazy loading is implemented through __index metamethod, it will not do any additional query if match2.match2games field is already populated, i.e., if the consumer knows that it will need m2g then they could preload them by simply adding match2games to lpdb query param
so this pr is specifically for those that we expect to not require m2g

Also, we need to look at not loading the m2g extradata field to reduce memory further

that part is already included

Lua-Modules/lua/wikis/commons/MatchGroup/Util.lua

Lines 586 to 613 in 9d24c8a

	---@param matchId string
	---@return match2game[]
	function MatchRecordMT._fetchGames(matchId)
	local gameRecords = mw.ext.LiquipediaDB.lpdb('match2game', {
	conditions = '[[match2id::' .. matchId .. ']]',
	query = 'date, game, length, map, mode, opponents, patch, resulttype, status, scores,' ..
	'subgroup, type, vod, walkover, winner',
	order = 'match2gameid asc',
	limit = 1000,
	})
	Array.forEach(gameRecords, function (gameRecord, index)
	setmetatable(gameRecord, {
	__index = function (t, k)
	if k == 'extradata' then
	local queriedExtradata = mw.ext.LiquipediaDB.lpdb('match2game', {
	conditions = '[[match2id::' .. matchId .. ']] AND [[match2gameid::' .. index .. ']]',
	query = 'extradata',
	limit = 1,
	})[1].extradata
	t.extradata = queriedExtradata
	return queriedExtradata
	end
	end
	})
	end)
	return gameRecords
	end

[hjpalpha]

won't this increase runtime for anything that needs to access game extradata fields?
it needs to spin up a new mw.ext.LiquipediaDB.lpdb for every time it needs to access extradata of a game (the first time per game)

(same concern for other fileds for which you spin up an additional mw.ext.LiquipediaDB.lpdb)

maybe an option param to tell it that it needs to query them with the original mw.ext.LiquipediaDB.lpdb call?

[ElectricalBoy]

won't this increase runtime for anything that needs to access game extradata fields?
it needs to spin up a new mw.ext.LiquipediaDB.lpdb for every time it needs to access extradata of a game (the first time per game)

depends; if you fetch all the relevant fields from match2 lpdb query from the get go then MatchRecordMT.__index won't be called in the first place, i.e., spinning up another mw.ext.LiquipediaDB.lpdb call can simply be avoided by adding match2games to lpdb query param

__index
This is used when a table access t[key] would return nil.

from Extension:Scribunto/Lua reference manual

[Rathoz]

While I like the approach, since so many m2g will need to be loaded, it may be worth batching them rather than going back and forth with the database.

because lazy loading is implemented through __index metamethod, it will not do any additional query if match2.match2games field is already populated, i.e., if the consumer knows that it will need m2g then they could preload them by simply adding match2games to lpdb query param so this pr is specifically for those that we expect to not require m2g

If we take standings as an example, it will most of the time require m2g, but not m2g.extradata. But because we cannot control which m2g fields we query, we will receive all of them, including extradata, if we fetch m2g. This is obviously a problem in the linked ticket, which is why it's split up here.

However, because we will (always?) need the m2g info, it would be advantageous to batch all the m2g queries (without extradata retrieval) into a single query, rather than running them one by one.

[ElectricalBoy]

If we take standings as an example, it will most of the time require m2g

not for swiss; matches tiebreakers only need m2 (though you need m2o regardless) and games tiebreakers rely on scores stored in m2o (#6720). iirc you only need m2g for rounds tiebreakers

I don't recall what the situation is for ffastandings 🤷

[Rathoz]

Mm ok

[Rathoz]

I would prefer not to keep the entire record in memory, especially when it's not needed. This can easily cause duplication of data in memory (eg. we fetch with m2g and then access it), which can be very painful on some wikis. Can we optimize this to only store data we need later, and clean up information no longer needed?

[Rathoz]

I'm a little concerned at the only 10% memory saving too.