Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Per environment master shard #538

Open
wants to merge 45 commits into
base: master
Choose a base branch
from
Open

Per environment master shard #538

wants to merge 45 commits into from

Conversation

calh
Copy link

@calh calh commented Aug 28, 2019 •
edited
Loading

In many setups, the default master shard might be a different database server in each environment. A single master_shard config setting doesn't fit. This patch first checks the shards.yml for octopus => master_shards => Rails env for a setting, and still falls back to octopus => master_shard if it doesn't exist.

Example:

octopus:
  master_shard: fallback
  master_shards:
    development: dev1
    production: lion
  environments:
    - development
    - production
  development:
    dev1:  . . . 
    dev2:  . . . 
  production:
    lion:  . . . 
    tiger:  . . .

calh added 30 commits June 21, 2019 15:38
@calh
Setup a docker development environment
74145f1
@calh
Start on setting up sample_app -- not working
48090f0 
Running cucumber gives the error:

cannot load such file -- test/unit (LoadError)
@calh
Ooops, revert a couple files.
8e2671d 
These weren't supposed to make it into this PR
@calh
Add DB2 server
c3ac7ca
@calh
Trying to add ibm_db gem to gemspec
ce42246 
This is failing with the stacktrace:

Caused by:
LoadError: cannot load such file -- arel/visitors/bind_visitor
/usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
/usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
/usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
/usr/src/app/.bundle/ruby/2.5.0/gems/ibm_db-4.0.0/lib/active_record/connection_adapters/ibm_db_adapter.rb:13:in `<top (required)>'
/usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
/usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
/usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
/usr/src/app/lib/octopus/proxy_config.rb:245:in `initialize_adapter'
@calh
Adding ibm_db versions for each rails version
75a1ef5
@calh
Install the DB2 ODBC driver
77a3155 
The gem install does this, but the download is slow.  Watching
the wget progress is more satisfying.
@calh
Add a DB2 shard
faaca64
@calh
Appriasal run for ibm_db
28e9190
@calh
Check that @config exists. Fixes thiagopradi#413
3e9f50b 
Error message is:

ActiveRecord::StatementInvalid: NoMethodError: undefined method `[]' for
nil:NilClass:

  from octopus/lib/octopus/abstract_adapter.rb:23:in `octopus_shard'
  from octopus/lib/octopus/abstract_adapter.rb:12:in `instrument'
  from vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.11.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'

This has shown to surface in the ibm_db and sqlserver adapters.
@calh
Check that @config exists. Fixes thiagopradi#413
4fd1129 
Error message is:

ActiveRecord::StatementInvalid: NoMethodError: undefined method `[]' for
nil:NilClass:

  from octopus/lib/octopus/abstract_adapter.rb:23:in `octopus_shard'
  from octopus/lib/octopus/abstract_adapter.rb:12:in `instrument'
  from vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.11.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'

This has shown to surface in the ibm_db and sqlserver adapters.
@calh
Monkey patch IBM_DBAdapter to add @config
5f3bc57 
This allows the InstrumenterDecorator to properly log the shard name
@calh
Ooof, typo in filename
fa21f02
@calh
Adding db2
9da3a89
@calh
Adding DB2
08ad9e0
@calh
Setting up DB2 server image
ee8c9d2
@calh
Add conditional check to allow the ibm_db gem to be optional
4a488a9
@calh
Optionally load the ibm_db gem
66c61c6
@calh
ibm_db support only works in rails 4.2 and 5.0
a70aad5
@calh
Remove ibm_db gem, this is optional now
5ef3e27
@calh
Remove ibm_db from rails 5.1
82205de
@calh
First DB2 test is working
1c658c1
@calh
Note on DB2
7f48d85
@calh
We don't need this with Appraisal
c6e267f
@calh
Tests for DB2
3495359
@calh
Add db2 port/db
1c5b373
@calh
Adding expect, in case I change up the entrypoint/cmd
67dc844
@calh
Create 4 total dbs for shard/replication testing
a96f523
@calh
db:prepare on rails5, run tests against all Appraisals
3dcd867
@calh
Skip migrations with DB2. Random bug in ibm_db.
ea41105 
Running tests individually works, but running the whole
migration_spec.rb file will recreate the issue _almost_
every time, in a random spec.

Here's the full error and stack trace:

Octopus::Migration should send the query to the correct shard
     Failure/Error:
       delegate :adapter_name, :add_transaction_record, :case_sensitive_modifier,
         :type_cast, :to_sql, :quote, :quote_column_name, :quote_table_name,
         :quote_table_name_for_assignment, :supports_migrations?, :table_alias_for,
         :table_exists?, :in_clause_length, :supports_ddl_transactions?,
         :sanitize_limit, :prefetch_primary_key?, :current_database,
         :combine_bind_parameters, :empty_insert_statement_value, :assume_migrated_upto_version,
         :schema_cache, :substitute_at, :internal_string_options_for_primary_key, :lookup_cast_type_from_column,
         :supports_advisory_locks?, :get_advisory_lock, :initialize_internal_metadata_table,
         :release_advisory_lock, :prepare_binds_for_database, :cacheable_query, :column_name_for_operation,
         :prepared_statements, :transaction_state, :create_table, to: :select_connection

     RuntimeError:
       An unexpected error occurred during retrieval of table metadata: uncaught throw :"Fetch Failure: <error message could not be retrieved>"
     # /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:1858:in `rescue in tables'
     # /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:1845:in `tables'
     # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:55:in `table_exists?'
     # ./lib/octopus/proxy.rb:25:in `table_exists?'
     # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/schema_migration.rb:20:in `block in table_exists?'
     # ./gemfiles/.bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/deprecation/reporting.rb:36:in `silence'
     # ./gemfiles/.bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/deprecation/instance_delegator.rb:20:in `silence'
     # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/schema_migration.rb:20:in `table_exists?'
     # /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:48:in `create_table'
     # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:1008:in `initialize_schema_migrations_table'
     # ./lib/octopus/proxy.rb:200:in `initialize_schema_migrations_table'
     # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/migration.rb:1127:in `initialize'
     # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/migration.rb:1017:in `new'
     # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/migration.rb:1017:in `run'
     # ./lib/octopus/migration.rb:187:in `block in run_with_octopus'
     # ./lib/octopus/proxy.rb:86:in `block (2 levels) in run_queries_on_shard'
     # ./lib/octopus/proxy.rb:345:in `using_shard'
     # ./lib/octopus/proxy.rb:85:in `block in run_queries_on_shard'
     # ./lib/octopus/proxy.rb:329:in `keeping_connection_proxy'
     # ./lib/octopus/proxy.rb:84:in `run_queries_on_shard'
     # ./lib/octopus/proxy.rb:93:in `block in send_queries_to_multiple_shards'
     # ./lib/octopus/proxy.rb:92:in `map'
     # ./lib/octopus/proxy.rb:92:in `send_queries_to_multiple_shards'
     # ./lib/octopus/migration.rb:186:in `run_with_octopus'
     # ./spec/support/octopus_helper.rb:46:in `migrate_to_version'
     # ./spec/support/octopus_helper.rb:37:in `migrating_to_version'
     # ./spec/octopus/migration_spec.rb:86:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # UncaughtThrowError:
     #   uncaught throw :"\xA3\xC9\x8E+H\xEA\xC9\vX\x82g\x8B\xAAU\x00\x00\bY\r\x8C\xAAU\x00\x00\x1A\xD0\xC1\xE2\xCF\xE9\x1Dd0\xB3m\x8B\xAAU\x00\x00\x80T\r\x8C\xAAU\x00\x00\x01\x99\xD1o,"
     #   /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:1845:in `fetch_assoc'
calh and others added 15 commits July 18, 2019 11:26
@calh
DB2 doesn't have boolean types
091f9ef
@calh
Adding Appriasals just for DB2.
65d981d 
This keeps the test suite going for most users that won't
be using the ibm_db gem
@calh
Ooops, wrong names
c63e2a0
@calh
Boolean coersion works in rails 4.2 with DB2
86058e1
@calh
Merge pull request #1 from calh/docker-dev-system
fc18f35 
Docker dev system
@calh
Merge pull request #2 from calh/nil_shard_config
cc72467 
Check that @config exists.  Fixes thiagopradi#413
@calh
Merge pull request #3 from calh/ibm_db
2cbfd9e 
DB2 ibm_db gem support
@calh
Issue with stock ibm_db 4.0.0 gem
f241079 
ibmdb/ruby-ibmdb#89

Until that issue is officially resolved by IBM, I'm using my local
fork for testing Octopus.
@calh
Fix for DB2 hanging indefinitely on disconnect with Passenger
9a8b293 
This is an oddball bug, deep in the DB2 C library.  Disconnecting
from the database during the Passenger fork process causes it to
hang indefinitely
@calh
Better condition for skipping disconnects on DB2
8b59e83
@calh
Better cross-adapter method of checking the db type
98de909
@calh
Skip the connected? test with DB2
92b4caf 
Since we don't disconnect from DB2, this spec fails.
@calh
Merge branch 'ibm_db'
d5431c2
@calh
Add a per-environment master_shard option
a4d2a06 
In many setups, the default master shard might be a different
database server in each environment.  A single `master_shard`
config setting doesn't fit.  This patch first checks
the shards.yml for `octopus` => Rails env => master_shard
for a setting, and still falls back to `octopus` => `master_shard`
if it doesn't exist
@calh
Better fix for per-environment master shards
1d07220 
I cowboyed up a bit too fast on the first fix.  This moves the
per-environment master_shard selection to its own yaml key
@wlmcewen wlmcewen mentioned this pull request Sep 5, 2019
Merged
@thiagopradi
Copy link
Owner

@calh - any updates on this PR? is this ready for review / merge?

@calh
Copy link
Author

calh commented Sep 25, 2019

@thiagopradi, yes, I'm using this in production right now as well, and it's a key requirement for how our environments are set up. We have different database server names for development and production, and the master shard setting needs to be there to properly initialize the rails app.

I shouldn't have probably merged in the ibm_db branch along with this PR. The only commit that's needed here is 1d07220. If you wanted to merge this separately from the ibm_db PR I could close & recreate this with just that single commit.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@calh @thiagopradi