flex_columns

Schema-free, structured storage inside a RDBMS. Use a VARCHAR, TEXT, CLOB, BLOB, or BINARY column in your schema to store structured data in JSON, while still letting you run validations against that data, build methods on top of it, and automatically delegate it to your models. Far more powerful than ActiveRecord's built-in serialization mechanism, flex_columns gives you the freedom of schemaless databases inside a proven RDBMS.

Combined with low_card_tables, allows a RDBMS to represent a wide variety of data efficiently and with a great deal of flexibility — build your projects rapidly and effectively while relying on the most reliable, manageable, proven data engines out there.

Supported platforms:

• Ruby 2.7, 3.0, 3.1 and 3.2 (MRI). Version 2.0.0 requires Ruby 2.7 or newer; earlier versions of this gem supported Ruby 1.8.7 to 2.2 and JRuby 1.7.
• ActiveRecord 6.1, 7.0, 7.1 and 7.2 are tested (6.1 on every Ruby, 7.x on Ruby 3.1+); the gemspec allows 3.0 up to (but not including) 8. (Should be compatible with future versions, as well...just sits on top of the public API of ActiveRecord.)
• Tested against MySQL, PostgreSQL and SQLite 3. (Should be compatible with all RDBMSes supported by ActiveRecord.)

Running the specs

bundle install
FLEX_COLUMNS_TRAVIS_CI_DATABASE_TYPE=sqlite bundle exec rspec

To run the specs against throwaway MySQL and PostgreSQL containers (and SQLite) without installing any database, use script/test-docker (requires Docker):

script/test-docker                          # all databases, ActiveRecord 7.2.3.2 and 6.1.7.10
script/test-docker postgres 7.2.3.2         # one database, one ActiveRecord version
script/test-docker mysql -- spec/flex_columns/system/validations_system_spec.rb

It uses its own database configuration, so your spec_database_config.rb is neither used nor modified. The mysql2 gem needs the MySQL client library on the host (for example brew install mysql-client or apt-get install libmysqlclient-dev).

To run by hand, set FLEX_COLUMNS_AR_TEST_VERSION (for example 6.1.7.10) to run against a specific ActiveRecord version. To use MySQL or PostgreSQL, create spec_database_config.rb (git-ignored) as described in the error message that the specs print when no database is configured. For example, for PostgreSQL:

FLEX_COLUMNS_SPEC_DATABASE_CONFIG = {
  :require => 'pg', :database_gem_name => 'pg',
  :config => { :adapter => 'postgresql', :database => 'myapp_test', :username => 'postgres',
               :host => '127.0.0.1', :min_messages => 'WARNING' }
}

The file takes priority over FLEX_COLUMNS_TRAVIS_CI_DATABASE_TYPE. Delete Gemfile.lock when you switch databases or ActiveRecord versions.

Installing flex_columns

# Gemfile
gem 'flex_columns'

Example

As an example — assume table users has a CLOB column user_attributes:

class User < ActiveRecord::Base
  ...
  flex_column :user_attributes do
    field :locale
    field :comments_display_mode
    field :custom_page_color
    field :nickname
  end
  ...
end

You can now write code like:

user = User.find(...)
user.locale = :fr_FR

case user.comments_display_mode
when 'threaded' then ...
when 'linear' then ...
end

Robust Example

As a snapshot of all possibilities:

# Assume we're storing the JSON in a wholly separate table, so we don't have to load it unless we need it...
class UserDetails < ActiveRecord::Base
  flex_column :user_attributes,
    :compress => 100,        # try compressing any JSON >= 100 bytes, but only store compressed if it's smaller
    :visibility => :private, # attributes are private by default
    :prefix => :ua,          # sets a prefix for methods delegated from the outer class
    :unknown_fields => :delete # if DB contains fields not declared here, delete those keys when saving
  do
    # automatically adds validations requiring a string that's non-nil
    field :locale, :string, :null => false
    # automatically adds validations requiring the value to be one of the listed values
    field :comments_display_mode, :enum => %w{threaded linear collapsed}
    # in the JSON in the database, the key will be 'cpc', not 'custom_page_color', to save space
    field :custom_page_color, :json => :cpc
    field :nickname
    field :visit_count, :integer

    # Use the full gamut of Rails validations -- they will run automatically when saving a User
    validates :custom_page_color, :format => { :with => /^\#[0-9a-f]{6}/i, :message => 'must be a valid HTML hex color' }

    # Define custom methods...
    def french?
      [ :fr_FR, :fr_CA ].include?(locale)
    end

    # +super+ works correctly in all cases
    def visit_count
      super || 0
    end

    # You can also access attributes using Hash syntax
    def increment_visit_count!
      self[:visit_count] += 1
    end
  end
end

# And now transparently include it into our User class...
class User < ActiveRecord::Base
  has_one :user_details

  include_flex_columns_from :user_details
end

...and then you can write code like so:

user = User.find(...)

user.user_attributes.french?   # access directly from the column
user.ua_visit_count            # :prefix prefixed the delegated method names with the desired string

user.visit_count = 'foo'       # sets an invalid value
user.save                      # => false; user isn't valid
user.errors.keys               # => :'user_attributes.visit_count'
user.errors[:'user_attributes.visit_count'] # => [ 'must be a number' ]

There's lots more, too:

• Complete validations support: the flex-column object includes ActiveModel::Validations, so every single Rails validation (or custom validations) will work perfectly
• Bulk operations, for avoiding ActiveRecord instantiation (efficiently operate using raw select_all and activerecord-import or similar systems)
• Transparently compresses JSON data in the column using GZip, if it's typed as binary (BINARY, VARBINARY, CLOB, etc.); you can fully control this, or turn it off if you want
• Happily allows definition and redefinition of flex columns at any time, for full dynamism and compatibility with development mode of Rails
• Rich error hierarchy and detailed exception messages — you will know exactly what went wrong when something goes wrong
• Include flex columns across associations, with control over exactly what's delegated and visibility of those methods (public or private)
• Control whether attribute methods generated are public (default) or private (to encourage encapsulation)
• "Types": automatically adds validations that require fields to comply with database types like :integer, :string, :timestamp, etc.
• Decide whether to preserve (the default) or delete keys from the underlying JSON that aren't defined in the flex column — lets you ensure database data is of the highest quality, or be compatible with any other storage mechanisms

Documentation

Documentation is on the Wiki!