Repository navigation
Add lazy-load guard to detect N+1 relation queries #591
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
27007b4
Add BeforeLazyRelationLoad event and LazyLoadGuard
KalimeroMK c1b49e7
Cover prevented BeforeLazyRelationLoad event
KalimeroMK 0a27a4f
Apply Rector suggestion in EventDispatcherProvider
KalimeroMK 37fc942
Fix double event dispatch in tests, count in strict mode, document event
KalimeroMK 174a71d
Drop Off mode and only/except from LazyLoadGuard
KalimeroMK 39f3a9d
Move lazy load guard to a separate trait without events
KalimeroMK c06c21e
Report lazy load starting from the second load of the same relation
KalimeroMK c0a004d
Merge branch 'master' into lazy-load-guard
KalimeroMK 21b0bf5
Apply Rector suggestion in MagicRelationsTrait
KalimeroMK 4a715b6
Count lazy loads per result set
KalimeroMK b12bf63
Clarify when a lazy load is reported
KalimeroMK ab9e220
Report the count and the relation to eager load
KalimeroMK 2a8dd34
Cover MagicPropertiesTrait usage
KalimeroMK 7425406
Document the report and resetting the guard between tests
KalimeroMK 8e97cf1
Throw a friendly exception when the guard reports in strict mode
KalimeroMK 862e620
Count distinct records of a result set and detect the trait used indi…
KalimeroMK 98c15b1
Name the model in the exception solution
KalimeroMK a891589
Cover re-loaded relations, asArray sets, nested and indirect guards
KalimeroMK 2b0d82f
Document record counting, indirect trait use and resetCounters()
KalimeroMK 64fec4e
Track the records of a result set by id, not in a WeakMap
KalimeroMK 4e3d408
Take the result set and the map once per registered set
KalimeroMK File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| # LazyLoadGuardTrait | ||
|
|
||
| `LazyLoadGuardTrait` allows detecting N+1 queries caused by lazy loading of relations, i.e. reading a relation | ||
| that wasn't eager-loaded by `with()`. | ||
|
|
||
| ```php | ||
| use Yiisoft\ActiveRecord\ActiveRecord; | ||
| use Yiisoft\ActiveRecord\Trait\LazyLoadGuardTrait; | ||
|
|
||
| final class Customer extends ActiveRecord | ||
| { | ||
| use LazyLoadGuardTrait; | ||
| } | ||
| ``` | ||
|
|
||
| Lazy loads are counted by [LazyLoadGuard](../../src/LazyLoadGuard.php) per result set, and reported according to | ||
| the mode starting from the second record of the same result set that reads the relation. One record reading a | ||
| relation isn't an N+1 problem, and neither are records that were each fetched by a query of their own, since there | ||
| is nothing to eager load across them: | ||
|
|
||
| ```php | ||
| // Reported: one query for the customers, then one query per customer for the orders. | ||
| foreach (Customer::query()->all() as $customer) { | ||
| $customer->getOrders(); | ||
| } | ||
|
|
||
| // Not reported: each customer came from a query of its own. | ||
| Customer::query()->findByPk(1)->getOrders(); | ||
| Customer::query()->findByPk(2)->getOrders(); | ||
|
|
||
| // Not reported: one record reading the relation again after it was reset. Records are counted, not reads. | ||
| $customer = Customer::query()->findByPk(1); | ||
| $customer->getOrders(); | ||
| $customer->refresh(); | ||
| $customer->getOrders(); | ||
| ``` | ||
|
|
||
| The model may use the trait through a parent class or through another trait of its own, which is where a | ||
| project-wide base trait usually puts it. | ||
|
|
||
| The mode and the logger are set statically: | ||
|
|
||
| ```php | ||
| use Yiisoft\ActiveRecord\LazyLoadGuard; | ||
| use Yiisoft\ActiveRecord\LazyLoadGuardMode; | ||
|
|
||
| LazyLoadGuard::set(LazyLoadGuardMode::Log, $logger); | ||
| ``` | ||
|
|
||
| | Mode | Behavior | | ||
| |----------|------------------------------------------------------------------------------------| | ||
| | `Log` | Default. Writes a PSR-3 warning; the context carries the model, relation and trace | | ||
| | `Strict` | Throws `LazyLoadException` | | ||
|
|
||
| Both modes report the same text, which names the relation, how many records of the result set read it and the | ||
| `with()` call that would replace them with one query: | ||
|
|
||
| ``` | ||
| Relation "App\Model\Customer::orders" is lazy loaded by 3 records of one result set. | ||
| Eager load it with "with('orders')" to run one query instead of 3. | ||
| ``` | ||
|
|
||
| In `Log` mode the logger is optional; when it's omitted the lazy loads are still counted and readable through | ||
| `LazyLoadGuard::getCounters()`, but nothing is written anywhere. `getCounters()` returns every lazy load per | ||
| relation, including the ones that weren't reported because they belong to result sets of their own. | ||
|
|
||
| The counters live until they're cleared. In long-running workers call `LazyLoadGuard::resetCounters()` after | ||
| each request, so that the result sets of one request aren't counted against the next; it keeps the mode and the | ||
| logger, so they're configured once: | ||
|
|
||
| ```php | ||
| LazyLoadGuard::set(LazyLoadGuardMode::Log, $logger); | ||
|
|
||
| while ($request = $queue->next()) { | ||
| $handler->handle($request); | ||
| LazyLoadGuard::resetCounters(); | ||
| } | ||
| ``` | ||
|
|
||
| In a test suite call `LazyLoadGuard::reset()` between tests, which clears the counters and restores the mode and | ||
| the logger to their defaults as well, so that the setup of one test doesn't leak into the next: | ||
|
|
||
| ```php | ||
| protected function setUp(): void | ||
| { | ||
| LazyLoadGuard::reset(); | ||
| LazyLoadGuard::set(LazyLoadGuardMode::Strict); | ||
| } | ||
| ``` | ||
|
|
||
| The guard is meant for development and testing, e.g. `Strict` in the test suite and `Log` on staging. | ||
|
|
||
| > [!NOTE] | ||
| > The trait overrides `retrieveRelation()` method. If another trait of the model overrides it too, | ||
| > resolve the conflict using `insteadof` and `as` operators. | ||
|
|
||
| Back to [Extending Functionality With Traits](traits.md). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -36,9 +36,7 @@ final class EventDispatcherProvider | |
| */ | ||
| public static function get(string $targetClass): EventDispatcherInterface | ||
| { | ||
| if (!isset(self::$dispatchers[$targetClass])) { | ||
| self::$dispatchers[$targetClass] = new Dispatcher(new Provider(self::getListenersFromAttributes($targetClass))); | ||
| } | ||
| self::$dispatchers[$targetClass] ??= new Dispatcher(new Provider(self::getListenersFromAttributes($targetClass))); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not related? |
||
|
|
||
| return self::$dispatchers[$targetClass]; | ||
| } | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Yiisoft\ActiveRecord\Internal; | ||
|
|
||
| use Yiisoft\ActiveRecord\ActiveRecordInterface; | ||
| use Yiisoft\ActiveRecord\LazyLoadGuard; | ||
|
|
||
| use function count; | ||
| use function spl_object_id; | ||
|
|
||
| /** | ||
| * Counts the records of one result set that lazy loaded a relation. The counts are freed together with the result | ||
| * set they belong to, which the records are the only reference to. | ||
| * | ||
| * @internal | ||
| * | ||
| * @see LazyLoadGuard | ||
| */ | ||
| final class LazyLoadResultSet | ||
| { | ||
| /** | ||
| * @var array[] The ids of the records that lazy loaded the relation `[relation => [record_id => true], ...]`. | ||
| * | ||
| * The records of a result set are known when it's registered, while all of them are alive, so an id of a | ||
| * freed record is never reused within the same result set. | ||
| * | ||
| * @psalm-var array<string, array<int, true>> | ||
| */ | ||
| private array $records = []; | ||
|
|
||
| /** | ||
| * Registers a record that lazy loaded the relation and returns the number of records of this result set that | ||
| * have loaded it. A record that loads the relation again, after {@see ActiveRecordInterface::resetRelation()} | ||
| * or {@see ActiveRecordInterface::refresh()} dropped it, is counted once. | ||
| */ | ||
| public function registerLazyLoad(ActiveRecordInterface $record, string $relation): int | ||
| { | ||
| $this->records[$relation][spl_object_id($record)] = true; | ||
|
|
||
| return count($this->records[$relation]); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace Yiisoft\ActiveRecord; | ||
|
|
||
| use LogicException; | ||
| use Yiisoft\FriendlyException\FriendlyExceptionInterface; | ||
|
|
||
| /** | ||
| * Represents an exception thrown by {@see LazyLoadGuard} in the {@see LazyLoadGuardMode::Strict} mode when several | ||
| * records of one result set lazy load the same relation. | ||
| */ | ||
| final class LazyLoadException extends LogicException implements FriendlyExceptionInterface | ||
| { | ||
| /** | ||
| * @param class-string $modelClass | ||
| */ | ||
| public function __construct( | ||
| string $message, | ||
| private readonly string $modelClass, | ||
| private readonly string $relationName, | ||
| ) { | ||
| parent::__construct($message); | ||
| } | ||
|
|
||
| public function getName(): string | ||
| { | ||
| return 'Relation is lazy loaded by several records of one result set'; | ||
| } | ||
|
|
||
| public function getSolution(): ?string | ||
| { | ||
| return <<<SOLUTION | ||
| Every record runs a query of its own to load the relation. Eager load it with the query that reads | ||
| the records, so that the relation is read with a single query: | ||
|
|
||
| ```php | ||
| \$records = \\{$this->modelClass}::query()->with('$this->relationName')->all(); | ||
| ``` | ||
|
|
||
| When the lazy loads are intentional, switch the guard to `LazyLoadGuardMode::Log` to keep reporting | ||
| them without throwing. | ||
| SOLUTION; | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is concerning that this code will always execute, even when N+1 query detection is not needed.
Why is it needed if you always create
new LazyLoadResultSet()there https://github.com/yiisoft/active-record/pull/591/changes#diff-d6bdeea9079931e23a1a13d2c4a1974b57a13a8f0ec17e84fea80aa6c33be4dfR93There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair point, I measured it to be sure: registering 10,000 unguarded records takes 0.93 ms against 45.71 ms of hydrating them, so it's ~2% in the worst case I could build. If that's still too much, the cheapest fix is an explicit switch checked first thing here. @samdark had me drop the
Offmode in 174a71d, so I'd rather not bring one back on my own — what do you two prefer, always on and cheap, or a switch again?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But that's not consuming anything if not registered, right? So not an issue in production and very little resource consumption in dev/staging. That's acceptable to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For debugging / profiling it is okay
Given this https://github.com/yiisoft/active-record/pull/591/changes#r4244295174, it's better to remove this in production.