Support common application usage: forked workers, shared connections, DATABASE_URL

README.md

@@ -156,6 +156,24 @@ end
   client that mysql2 closed because a running query was interrupted (an `Async` task stopped, `Thread#raise`) is
   replaced the same way, without failing the next call.
 * SQLite: threads and fibers sharing one `SQLite3::Database` take turns; a transaction holds the database until it ends.
+* `pg`: `QuerySql.connect` also takes a connection URL: `QuerySql.connect(ENV.fetch('DATABASE_URL'), size: 5)`.
+* A connection shared with another library (ActiveRecord's `raw_connection`, Sequel, your own code) keeps its
+  settings: the generated queries pass their own type maps (`pg`; the connection's `type_map_for_results`,
+  `type_map_for_queries` and `field_name_type` are left as they are) or query options (`mysql2`: rows are read with
+  `as: :array, cast: true, cast_booleans: false`, whatever the client's or `Mysql2::Client.default_query_options`).
+  On SQLite `results_as_hash` and the other settings of the `SQLite3::Database` are kept too.
+* Forked processes (Puma / Sidekiq / Resque workers, `Process.fork`): connections opened before the fork stay the
+  parent's. connection_pool's fork handling replaces the pooled ones in the child without closing the parent's
+  sessions, and the child lets go of every inherited connection the generated code has used (also ones another
+  thread had checked out at fork time, and ones the child never uses), so its exit does not end them either. A `pg`
+  connection used directly (not from a `ConnectionPool`) reconnects in the child; a `mysql2` client
+  used that way raises `Mysql2::Error::ConnectionError` in the child (open clients in the process that uses them).
+  SQLite: the sqlite3 gem closes writable databases a child inherits, so open the database again in the child.
+  On macOS, connecting from a forked child can crash in libpq's GSSAPI probe; set `gssencmode: 'disable'` (or
+  `PGGSSENCMODE=disable`), as usual for forking `pg` apps there.
+* A `transaction` inside a transaction that another library opened on the same connection (`BEGIN`) is a savepoint
+  on PostgreSQL and SQLite. MySQL has no nested transactions: its `BEGIN` commits the open transaction first (see
+  [Limitations](#limitations)).
 
 ## Supported features
 ### Query annotations
@@ -247,6 +265,9 @@ server; duplicate keys and invalid values raise `Mysql2::Error` instead of being
   elements raises `SQLite3::SQLException: too many SQL variables`; on MySQL the limit is the server's
   `max_allowed_packet`. Load larger sets into a table (`:copyfrom`) and join it instead.
 * `:batchexec` / `:batchone` / `:batchmany` are not atomic on their own: wrap them in `transaction` for all-or-nothing.
+* MySQL: `transaction` on a client inside a transaction that was started with raw SQL or by another library commits
+  that transaction first (MySQL's `BEGIN` does), so a later `ROLLBACK` no longer undoes it. Run the outer work in
+  `transaction` too (inner calls become savepoints), or finish the other transaction first.
 * SQLite: sqlc does not see parameters in `ON CONFLICT ... DO UPDATE SET` (it leaves `sqlc.arg(...)` in the SQL or
   reports fewer parameters than there are `?`), so such queries are generation errors. Use `excluded.<column>`
   (`DO UPDATE SET n = counters.n + excluded.n`) or move the value into `VALUES`.

docs/Coverage.md

@@ -35,6 +35,15 @@ only when the behavior is rejected or decided during generation and generated co
 | Top-level Float JSON parameters, Symbol enum values | [`test_top_level_float_json_parameters_round_trip_exactly`](../tests/end2end_pg.rb#test_top_level_float_json_parameters_round_trip_exactly) | [`test_symbol_enum_value_is_bound_as_its_name`](../tests/end2end_mysql2.rb#test_symbol_enum_value_is_bound_as_its_name) | not SQLite (no enums / JSON encoding) | |
 | Modules sharing a connection keep separate prepared statements | [`test_generator_options_preserve_runtime_values_and_generate_expected_layouts`](../tests/end2end_options.rb#test_generator_options_preserve_runtime_values_and_generate_expected_layouts) | not MySQL (no server-side prepared statements) | not SQLite (statements are per call) | [`test_statements_are_named_after_the_module_and_query_class`](../test/features_test.rb#test_statements_are_named_after_the_module_and_query_class) |
 | SQLite parameters sqlc does not report (`ON CONFLICT ... DO UPDATE SET`) are generation errors | not PostgreSQL | not MySQL | unit: rejected at generation | [`test_sqlite_parameters_sqlc_does_not_report_are_generation_errors`](../test/features_test.rb#test_sqlite_parameters_sqlc_does_not_report_are_generation_errors), [`test_sqlite_placeholders_in_literals_and_comments_are_not_parameters`](../test/features_test.rb#test_sqlite_placeholders_in_literals_and_comments_are_not_parameters) |
+| Forked processes (Puma / Sidekiq workers): pooled connections are replaced in the child without ending the parent's sessions; `pg` direct connections reconnect, `mysql2` direct clients raise, SQLite children open their own database | [`test_a_forked_child_leaves_the_parent_connections_usable`](../tests/end2end_pg.rb#test_a_forked_child_leaves_the_parent_connections_usable) | [`test_a_forked_child_leaves_the_parent_connections_usable`](../tests/end2end_mysql2.rb#test_a_forked_child_leaves_the_parent_connections_usable) | [`test_a_forked_child_opens_its_own_database_and_leaves_the_parent_usable`](../tests/end2end_sqlite3.rb#test_a_forked_child_opens_its_own_database_and_leaves_the_parent_usable) |  |
+| A forked child's plain `exit` leaves the parent's sessions open: a pooled connection another thread has checked out at fork time, a direct connection the child never uses | [pg](../tests/end2end_pg.rb) `test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open` · [mysql2](../tests/end2end_mysql2.rb) `test_a_child_exit_leaves_connections_checked_out_or_unused_at_fork_open` |
+| A connection shared with another library keeps its settings (pg type maps / `field_name_type`, mysql2 query options and `default_query_options`, SQLite `results_as_hash`) | [`test_a_connection_shared_with_another_library_keeps_its_settings`](../tests/end2end_pg.rb#test_a_connection_shared_with_another_library_keeps_its_settings) | [`test_a_client_with_other_query_options_still_returns_typed_rows`](../tests/end2end_mysql2.rb#test_a_client_with_other_query_options_still_returns_typed_rows) | [`test_a_database_shared_with_another_library_keeps_its_settings`](../tests/end2end_sqlite3.rb#test_a_database_shared_with_another_library_keeps_its_settings) |  |
+| `pg`: `QuerySql.connect` takes a connection URL (`DATABASE_URL`) | [`test_connect_accepts_a_connection_url`](../tests/end2end_pg.rb#test_connect_accepts_a_connection_url) | not MySQL (mysql2 takes no URL) | not SQLite (no `connect`) |  |
+| `transaction` inside a transaction opened by another library: a savepoint (PostgreSQL, SQLite); MySQL limitation: commits it | [`test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint`](../tests/end2end_pg.rb#test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint) | [`test_transaction_inside_a_transaction_opened_by_another_library_commits_it`](../tests/end2end_mysql2.rb#test_transaction_inside_a_transaction_opened_by_another_library_commits_it) | [`test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint`](../tests/end2end_sqlite3.rb#test_transaction_inside_a_transaction_opened_by_another_library_is_a_savepoint) |  |
+| Schema-per-tenant `search_path` switching with prepared statements | [`test_queries_follow_the_search_path_of_each_tenant`](../tests/end2end_pg.rb#test_queries_follow_the_search_path_of_each_tenant) | not MySQL (databases, no search path) | not SQLite (no schemas) |  |
+| Primary + read-replica query objects; a write to the read-only replica raises and leaves it usable | [`test_primary_and_read_replica_query_objects`](../tests/end2end_pg.rb#test_primary_and_read_replica_query_objects) | [`test_primary_and_read_replica_query_objects`](../tests/end2end_mysql2.rb#test_primary_and_read_replica_query_objects) | [`test_primary_and_read_replica_query_objects`](../tests/end2end_sqlite3.rb#test_primary_and_read_replica_query_objects) |  |
+| Memoized frozen query objects, subclasses and prepended wrappers forwarding `...` / `**`, `SimpleDelegator`, shared across threads | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_pg.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_mysql2.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) | [`test_memoized_frozen_subclassed_and_wrapped_query_objects`](../tests/end2end_sqlite3.rb#test_memoized_frozen_subclassed_and_wrapped_query_objects) |  |
+| Rows through `Marshal` (Rails.cache) and JSON (`to_h.to_json`) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_pg.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_mysql2.rb#test_rows_round_trip_through_marshal_and_json) | [`test_rows_round_trip_through_marshal_and_json`](../tests/end2end_sqlite3.rb#test_rows_round_trip_through_marshal_and_json) |  |
 
 CI runs `test/coverage_map_test.rb`, which checks every linked test method exists in its referenced `tests/` or `test/` file.
 | Models named like Ruby / driver constants (`Exception`, `Data`, `Thread`, `Set`, `Monitor`, ...) do not shadow them | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_pg.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_mysql2.rb#test_models_named_like_core_constants_do_not_shadow_them) | [`test_models_named_like_core_constants_do_not_shadow_them`](../tests/end2end_sqlite3.rb#test_models_named_like_core_constants_do_not_shadow_them) | |

examples/mysql2/connection.rb

@@ -3,8 +3,42 @@
 # auto-generated by sqlc - do not edit
 
 module Mysql2Codegen
-  # State kept on every Mysql2::Client the first time it is used.
+  # State kept on every Mysql2::Client the first time it is used. Rows are read with their own query options
+  # (as: :array, cast: true, cast_booleans: false), whatever the client's or Mysql2::Client.default_query_options.
   module ConnectionState
+    def self.extended(client) = client.instance_variable_set(:@owner_pid, ::Process.pid)
+
+    # A forked child (Puma / Sidekiq / Resque worker) lets go of every inherited session the generated code has
+    # used, also ones another thread had checked out at fork time and ones it never uses: its exit would end
+    # them for the parent.
+    def self.after_fork = ::ObjectSpace.each_object(self) { |client| client.release_parent_socket if client.inherited? }
+
+    module ForkHook
+      def _fork
+        pid = super
+        ConnectionState.after_fork if pid.zero?
+        pid
+      end
+    end
+    ::Process.singleton_class.prepend(ForkHook)
+
+    # Opened by the parent of a forked process (Puma / Sidekiq workers): its socket is the parent's session.
+    def inherited? = @owner_pid != ::Process.pid
+
+    # connection_pool closes the connections a forked child inherits: point this process's copy of the socket
+    # at /dev/null first, so the QUIT sent by #close does not end the parent's session.
+    def close
+      release_parent_socket if inherited?
+      super
+    end
+
+    def release_parent_socket
+      self.automatic_close = false
+      ::IO.for_fd(socket, autoclose: false).reopen(::IO::NULL)
+    rescue ::StandardError # already closed
+      nil
+    end
+
     attr_accessor :in_transaction
     # The error with which the server rolled back the whole transaction (a deadlock, or a lock wait timeout with
     # innodb_rollback_on_timeout): the transaction's later queries raise instead of running in autocommit mode.
@@ -42,7 +76,7 @@ def ends_transaction?(error)
 
       if @rollback_on_timeout.nil?
         @rollback_on_timeout = begin
-          query('SELECT @@innodb_rollback_on_timeout', as: :array).first.first == 1
+          query('SELECT @@innodb_rollback_on_timeout', as: :array, cast: true).first.first == 1
         rescue ::Mysql2::Error
           false
         end

examples/mysql2/connection.rbs

@@ -7,6 +7,16 @@ module Mysql2Codegen
     def query_running?: () -> bool
     def rolled_back_error: () -> untyped
     def ends_transaction?: (untyped error) -> bool
+    def self.extended: (untyped client) -> void
+    def inherited?: () -> bool
+    def close: () -> void
+    def release_parent_socket: () -> void
+    def self.after_fork: () -> void
+    module ForkHook
+      def _fork: () -> ::Integer
+    end
+    def socket: () -> ::Integer
+    def automatic_close=: (bool) -> bool
   end
 
   class TransactionEnded < ::StandardError