The ferry from cmu-is-projects

ferry's Issues

Exporting and importing db functions, triggers, indexes, etc

refer to

active record migration
https://github.com/rails/arel

Exports do not work

After Installing this gem via the bundle and via gem install...I got this error in both situations:

gems/ferry-1.0.0/bin/ferry:10:in 'block (2 levels) in <top (required)>': 
please enter a field for environment and table (RuntimeError)

Demo page gives a 404

The readme says "See the ferry_demo app or our GitHub pages site for guidance on using Ferry!" but the link to the demo (https://github.com/cmu-is-projects/ferry_demo.com) returns a 404.

potential site for gem hosted on github pages

weekend of 9/26-9/28 refactor and sql switcher

my weekend project is going to entail ...

refactor code up to 9.25/26
creating db switcher that takes a sqlite db and makes it postgresql
cleaning up git history
refactor readme to make documentation a little nicer

again - sorry for being really MIA on this. the job search has really been relentless and i've been devoting alot of time to that in place of putting more effort in to 475. hopefully this weekend will have some nice deliverables in it.

Init command for DB setup?

Connecting to ActiveRecord requires we know the DB type, and also the credentials that go along with it depending on the DB type. For now, I'm putting the current type as an argument on the export call, but I think we should look into having a initialize for ferry where they specify what the current db is (and any credentials).
Unless there is a better way to detect it, but I can't figure out how to find it through Active Record because I actually have to connect to the DB first before asking what type it is.

Extraneous requirements

Not sure if this is avoidable, but when trying to include ferry in a project of mine, it requires me to install postgres and mysql.

I see the reason for this, but if I am only using postgres or sqlite (which is what I normally use) it is a pain to install mysql for something I won't use

maybe split it into seperate gems, like ferry-mysql or ferry-pg

revising cli option passing using ruby's built in option parser

ruby require 'optparse'

Accounting for nested data structures in db

yep this is important ...

could be anything from nested json forms that need to be accounted for, nested structures in db columns.

Dumping a Remote Database

Is ferry able to dump a remote database? I have attempted this with a mysql database located on remote server, but I get this error:

ferry --dump phenotype
operating with mysql2
mysqldump: Got error: 1044: Access denied for user ''@'localhost' to database 'phenotypes' when selecting the database
"Complete!"

This is what is in my database.yml for the phenotype database:

phenotype:
adapter: mysql2
encoding: utf8
reconnect: false
database: phenotypes
host: pawn.hss.cmu.edu
username: phenotypes
password: –––––––––
socket: /tmp/mysql.sock

reviews, revisions, repairs

- [ ] Review the code in place - [ ] Make sure old activerecord::relation tasks work - [ ] TESTS!!!

Please comment / add more to the list

Data Visualization

Being able to run a quick command prompt that deploys informative graphs and charts to a designated location could hold large value for ferry and the environment around developers. We will be looking into developing functionality like this with d3 visualizations.

implement rspec test suite

Unix style commands

It would be useful to allow the options to follow a "unix style" syntax vs the comma syntax defined.

for example
ferry --import development table path/to/data.csv vs
ferry --import development,table,path/to/data.csv

This allows for Unix tab completion of paths, and is implemented by other gems like bropages & capistrano

Exporting calculated 'columns'

Any planned support for something like this? May not be in the scope of the gem, as it seems like this focuses on data import/export

Upkeep of GitHub page and documentation revising

please post for suggestions or convo starters for documentation addition / revision / deletion

SQL Dumper & Filler

These objects will allow us to dump an applications entire database to a .sql file, or fill an empty specified database from a .sql file. Some challenges we will face here are ...

how to test whether or not a dump was successful
how to test whether or not a fill was successful
making sure all database knowledge has been dumped / filled (like functions, calculated columns, triggers, indexing patters, etc are carried over)
that filling a database matches the correct schema of an application

psql and mysql installation wizard for db switcher

asks if you want to proceed
takes your input and installs
- where to pull code from to install
- path configuration
- checking for previously installed versions?
- giving the user feedback during installation
- rollback if break

documentation

clean up readme

include link to git page
install
examples
- to_csv
- to_yaml
- import
- db_switch
running tests?

ActiveRecord::NoDatabaseError if current user does not have a postgres role

When I try to run ferry in development I end up with an ActiveRecord::NoDatabaseError (see stack trace below). I solved the issue in a fork and will create a pull request for it. It seems like the problem occurs because ferry tries to use the system user name to access the postgreSQL server.

$ ferry --to_csv development items

operating with postgresql
/home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:898:in `rescue in connect': FATAL:  role "roland" does not exist (ActiveRecord::NoDatabaseError)
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `connect'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:568:in `initialize'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:41:in `new'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:41:in `postgresql_connection'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:435:in `new_connection'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:445:in `checkout_new_connection'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `acquire_connection'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:351:in `block in checkout'
  from /home/roland/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:350:in `checkout'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
  from /home/roland/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:541:in `retrieve_connection'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/activerecord-4.1.6/lib/active_record/connection_handling.rb:87:in `connection'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/ferry-1.0.1/lib/ferry/exporter.rb:11:in `to_csv'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/ferry-1.0.1/bin/ferry:12:in `block (2 levels) in <top (required)>'
  from /home/roland/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/optparse.rb:1359:in `call'
  from /home/roland/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/optparse.rb:1359:in `block in parse_in_order'
  from /home/roland/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/optparse.rb:1346:in `catch'
  from /home/roland/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/optparse.rb:1346:in `parse_in_order'
  from /home/roland/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/optparse.rb:1340:in `order!'
  from /home/roland/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/optparse.rb:1432:in `permute!'
  from /home/roland/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/optparse.rb:1454:in `parse!'
  from /home/roland/.rvm/gems/ruby-2.1.2/gems/ferry-1.0.1/bin/ferry:50:in `<top (required)>'
  from /home/roland/.rvm/gems/ruby-2.1.2/bin/ferry:23:in `load'
  from /home/roland/.rvm/gems/ruby-2.1.2/bin/ferry:23:in `<main>'
  from /home/roland/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15:in `eval'
  from /home/roland/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15:in `<main>'

Rspec also fails with:

$ rspec spec/tests/utilities_tests.rb
Failures:

  1) utility functions #db_connect #postgresql
     Failure/Error: expect(ActiveRecord::Base.connection.adapter_name).to eql('PostgreSQL')
     ActiveRecord::NoDatabaseError:
       FATAL:  role "roland" does not exist
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/postgresql_adapter.rb:898:in `rescue in connect'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `connect'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/postgresql_adapter.rb:568:in `initialize'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/postgresql_adapter.rb:41:in `new'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/postgresql_adapter.rb:41:in `postgresql_connection'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:435:in `new_connection'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:445:in `checkout_new_connection'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `acquire_connection'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:351:in `block in checkout'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:350:in `checkout'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:541:in `retrieve_connection'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
     # /home/roland/.rvm/gems/ruby-2.1.1/gems/activerecord-4.1.8/lib/active_record/connection_handling.rb:87:in `connection'
     # ./tests/utilities_tests.rb:15:in `block (3 levels) in <top (required)>'

ferry --help

setup a menu for things to do when someone runs ferry, ferry -h, or ferry --help

Options for Import

As mentioned when I first merged the import functionality, other importers give the option to validate the data against AR before inserting, and also automatically generating PKs for the data if they do not have any.
I think the PK generation would be simple, just need to query the db for the index of the table. But we'd need a case if the PK is not the default "id" field. Then we can assign that to the record and import it.

For the validation, I think that is best left out for now. The dbs should have the basic type validation, but beyond that it might be a big performance hit if we need to check every row of import data against a model.

Customizable migrations

Creating a customizable script that takes in arguments that a user provides is something I feel is a necessary component of ferry that does not yet exist. With the current $ ferry -init command we create a base file ferry.rake in the tasks directory of the developers Rails app. While this provides some value, it's not very impactful. Digging into how capistrano allows users to configure their deploys in a deploy.rb file is going to be a starting point for us as we look to develop a way to provide our own equivalent of deploy.rb for data migrations and manipulations.

cmu-is-projects / ferry Goto Github PK

ferry's People

Contributors

Stargazers

Watchers

Forkers

ferry's Issues

Recommend Projects

Recommend Topics

Recommend Org