level12 / keg Goto Github PK

Randy Syring [4:52 PM]

Wierd... climate develop db init --clear-first creates the stage schema... climate develop db init doesn't....

[4:52]
@tonto: yes, its is a bit unintuative

[4:54]
The reason is that the code to create the schemas only gets called for the "prep_empty" event: https://github.com/level12/keg/blob/master/keg/db/dialect_ops.py#L65

GitHub
level12/keg
keg - Keg: more than Flask

Nick Zaccardi [4:55 PM]
I see that.

Randy Syring [4:55 PM]
Which only gets called on db_clear(): https://github.com/level12/keg/blob/master/keg/db/__init__.py#L109

GitHub
level12/keg
keg - Keg: more than Flask

Nick Zaccardi [4:55 PM]
It should probably get called both, times, yea?

Randy Syring [4:56 PM]
The logic could be cleaned up, yes. But, it's not that simple.

[4:58]
My assumption has always been that the DB starts off ready to create objects as needed, i.e. the database and schemas are already in place. So, it's kind of due to the way that we clear the PostgreSQL schemas of their objects (by dropping the schemas) that we even need to worry about this.

[4:58]
That is why it's called prep_empty() and tied to the clear step.

Nick Zaccardi [4:58 PM]
What about when we first create the database?

[4:59]
I suppose my point is, what is the correct way to create a brand new environment?

Randy Syring [4:59 PM]
in the past, i've always assumed that initial creation of the DB (by the developer) would leave the DB in such a state that all objects could be created by the application.

[4:59]
Alembic?

[5:00]
default DB state + Alembic run should make it ready?

Nick Zaccardi [5:00 PM]
When I create the database it doesn't create all the schemas, it only creates one.

Randy Syring [5:00 PM]
yes, my assumption breaks down in those environments.

Nick Zaccardi [5:00 PM]
So what I am hearing you say is, the provision step should create it.

[5:01]
That is fine, I was falsely under the impression that db init did that.

Randy Syring [5:01 PM]
That seems reasonable to me...but I could also see adjusting Keg to be smart about creating schemas that don't exist that it knows it needs.

Nick Zaccardi [5:01 PM]
Right.

Randy Syring [5:01 PM]
In fact, that would probably be better...I'm only explaining why it works the way it works now, not saying that is the best way.

We used to have to use SecretStorage-Setup a package I wrote to link in some system libraries to virtualenvs.

However, with the latest python packages available, it seems like there are just two packages to be installed on an Ubuntu system (I tested 14.04) to get the keyring to work using the dbus secretstorage subsystem.

• keyring: Keg already includes this a dependency
• SecretStorage: but keyring now includes this as a dependency on Linux
• dbus-python: can be pip installed

So, for any Keg application that wants to use Keyring substitution on Linux, just pip install dbus-python.

• document this in the readme at least
• remove keyring help messages that talk about SecretStorage-Setup, adjust messages as needed
• remove SecreteStorage-Setup repo or, at least, put up notice of deprecation

When using KegApp.command, which references a KegGroup instance, the command being decorated should be wrapped in some kind of error handler by default.

• Turn on/off exception handling using app.debug. --no-error-handler should also be passed on the command level for this purpose.
  • By default, devs using these commands would skip error handling due to app.debug = True
  • In production, error handling would be enabled by default which is good for things like scheduled tasks.
  • If a dev had to troubleshoot in production, --no-error-handler could be passed to avoid spamming other devs with notifications.
• --show-errors: to make the command put the stack trace on stderr (but still run the commands error handling)

I init keg.db.db in keg.db which means it's really hard to customize KegSQLAlchemy/SQLAlchemy classes by subclassing and overriding. This proved especially true when trying to setup custom json serialization methods for a customer project.

I believe keg.db.db would be better served being a local proxy like flask.request and redirecting that to the real instance on the app. Then the app would need to have a setup method for that class.

Travis doesn't support it. However, there was some discussion on distutils list a while back where someone was asking for help in writing a go script to spin up a windows VM on Azure for testing.

self.engine.echo = True should be a config setting for Keg for debugging and testing.

It would be nice, if debug=False that the output would go to the log file and if testing=True that it would go out to the screen.

The application will load once and then load again. Is this expected behavior?

https://www.reddit.com/r/Python/comments/3nctlm/what_python_tools_should_i_be_using_on_every/cvn0ybf

parser.add_argument('-v', '--verbose', action='count', default=0)
parser.add_argument('-q', '--quiet', action='count', default=0)

logging_level = logging.WARN + 10*args.quiet - 10*args.verbose

# script -vv -> DEBUG
# script -v -> INFO
# script -> WARNING
# script -q -> ERROR
# script -qq -> CRITICAL
# script -qqq -> no logging at all

Pull over the issues first.

https://github.com/fengsp/flask-session ?

This is what I did to get an acceptable solution of managing debug from the CLI:

import os
import pathlib
import click
from flask.cli import FlaskGroup, pass_script_info

import app

@click.group(cls=FlaskGroup, create_app=app.cli_create)
@click.option('--debug/--no-debug', default=False)
@pass_script_info
def main(info, debug):
    # This should probably look at FLASK_DEBUG before overriding it when the 
    # debug flag is not passed to the CLI
    os.environ['FLASK_DEBUG'] = '1' if debug else '0'

You can find the issues with setting up debugging by reading this code: https://github.com/pallets/flask/blob/master/flask/cli.py#L550-L556 and ultimate https://github.com/pallets/flask/blob/master/flask/helpers.py#L59-L63.

Basically, the only way to kick the debugger on is to set the FLASK_DEBUG environment variable. This main will run prior to the run command, so it would be possible to write a simple function to parse/load the configuration here and then set FLASK_DEBUG based on that. This seemed like much less hassle and "worked" well enough for me.

Invocation would be app --debug run to turn on the debugger.

It would be really helpful if <app> develop run would auto reload when a change is made in a file.

Maybe this already exists?

On 03/23/2015 05:30 PM, Marc Schlaich wrote:

Hey Randy,

in the upcoming pytest-cov 2.0 you can easily create different data files for each tox environment and automatically combine them at the end of the tox run. See here for an example project:
https://github.com/schlamar/pytest-cov/tree/2.0/example-tox-project

Unfortunately, there is no release date yet as I'm pretty busy right now. However, the 2.0 branch should be quite stable, so you could give it a try.

Marc

The name of an applications configuration file is not always obvious. Consider creating docs to help explain what that should be.

The config variables DEVELOPER_USERNAME, DEVELOPER_EMAIL, and DEVELOPER_PASSWORD are defaults that will be needed in the config for super-user setup. At some point, we should connect KEG_EMAIL_OVERRIDE_TO to DEVELOPER_EMAIL.

https://gitter.im

The default profile is currently controlled from both config files and the _DEFAULT_PROFILE environment variable.

But, when we run tests, in keg.testing, ContextManager hard-codes the profile name as 'TestProfile.' Furthermore, CLIBase and Click's CliRunner create the app without any explicit testing profile. So, in reality, when doing CLI Tests locally, the DevProfile is likely being used.

Need to make this more consistent.

Would like to have a decorator API that matches what Flask allows.

/home/rsyring/.virtualenvs/racebetter/lib/python3.5/site-packages/keg/db/init.py:4: ExtDeprecationWarning: Importing flask.ext.sqlalchemy is deprecated, use flask_sqlalchemy instead.
from flask.ext.sqlalchemy import SQLAlchemy

https://github.com/level12/keg/blob/master/keg/web.py#L155 can be moved out of the if so that it auto_populate will always run.

There are always database changes and alemibic integrates well with SQLAlchemy. Alas, integrate Alembic right into the system.

Remove code that is now in SecretStorage-Setup.

class PayeeFormBaseView(FISBaseView):
    blueprint = admin
    def post(self, payee):
       ...

    def get(self):
        ...

class PayeeFormContract(PayeeFormBaseView):
    rule('/payee-forms/<int:payee_id>/contract', post=True)
    ....

The above doesn't work as expected. It ends up not creating the route for PayeeFormContract.

1. Remove coveralls
2. Add circle CI
3. remove gitter (#65)
4. remove pypi downloads per month

The SQL in db.dialects for clearing and prepping a postgresql database doesn't work when the username has a dash, e.g. "racebetter-beta".

The whole point of having an init() is to have a method that is safe to override without having to worry about passing parameters up the stack or call super().

The whole point of having the init() happen separate from the app setup was to facilitate being able to use the app as a decorator. When using a factory setup method, you can't get at the app instance to use it as a decorator, but Flask requires the app instance to use as a decorator. So when the app gets bigger and you want to move some of your stuff around, the app setup just gets ugly IMO.

Options:

• make the class object itself have whatever methods on it we need to decorate things and then just save them for assignment to the app instance once it's created.
• rename init() to something else like start()
• maybe the init name isn't so helpful and we should have a "on_ready() contract" where the name would be more indicative of what we really are trying to accomplish.

• CircleCI
• Codecov

https://github.com/level12/keg/blob/master/keg/web.py#L154 will interpret an empty string as False and call self.render().

For example... assume template_which_is_empty.html exists but is empty. This logic will fail.

class View(keg.web.BaseView)
    @keg.web.route('/route')
    def route_func(self):
        return flask.render_template('template_which_is_empty.html')

Click does not recommend this, as it causes subtle bugs. Click 5.0 and above issue a landslide of warnings when it detects this.

SQLite requires a flag to be set to enforce FK constraints. It would be nice if Keg could provide a setting to enable this behavior automatically when connected to a SQLite database.

http://stackoverflow.com/a/15542046

@event.listens_for(sa.engine.Engine, 'connect')
def _set_sqlite_pragma(connection, conn_record):
    from sqlite3 import Connection as SQLite3Connection
    if isinstance(connection, SQLite3Connection):
        cursor = connection.cursor()
        cursor.execute("PRAGMA foreign_keys=ON;")
        cursor.close()

I did the following for RaceBetter:

import logging
from logging.config import dictConfig
from logging.handlers import RotatingFileHandler
import os.path as osp

import keg.logging
from pythonjsonlogger import jsonlogger


class Logging(keg.logging.Logging):

    def dict_config(self):
        log_config = self.config.get('LOGGING_CONFIG')
        if log_config:
            dictConfig(log_config)

    def init_app(self):
        super().init_app()
        self.dict_config()
        self.init_json_log()

    def init_json_log(self):
        log_fname = self.log_fname.replace('.', '_json.')
        log_fpath = osp.join(self.log_dpath, log_fname)

        log_max = self.config['KEG_LOG_MAX_BYTES']
        log_backups = self.config['KEG_LOG_MAX_BACKUPS']

        file_handler = RotatingFileHandler(log_fpath, maxBytes=log_max, backupCount=log_backups,
                                           encoding='utf-8')
        file_handler.setLevel(logging.INFO)

        include_fields = '%(asctime) %(pathname) %(funcName) %(lineno) %(message) %(levelname)' \
            ' %(name)s %(process) %(processName) %(message)'
        formatter = jsonlogger.JsonFormatter(include_fields)
        file_handler.setFormatter(formatter)

        for logger in self.loggers:
            logger.addHandler(file_handler)

        # Make the handler easier to get to by other code for custom logger/handler setups like
        # we did for the heartbeat CLI command.
        self.json_file_handler = file_handler

Seems like it could have been easier. I also had to set KEG_LOG_MAX_BYTES to a really high level to turn off file rotation by Python so the system logrotate could handle it.

No one is monitoring that channel, we should remove the badge.

Ported from Bitbucket

When the next Flask comes out, this function is defined on Flask and won't be needed on KegApp.

A byte string doesn't not get picked up by the keyring implementation and probably should.

https://github.com/level12/keg/blob/master/keg/keyring.py#L53

See: http://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap08.html

Specifically:

Environment variable names used by the utilities in the Shell and Utilities volume of IEEE Std 1003.1-2001 consist solely of uppercase letters, digits, and the '_' (underscore)

This means all dotted import names (like those used in the tests) must be replaced with an underscore.

Right now, we are assuming a Keg app can run without Blinker. However, since we are kicking off our testing prep with a signal, it's safe to assume Blinker will need to be installed.

There are some test fails/errors in Windows for the keg tests. At least one is path related. I need to determine if others are related to the config file I copied from the template, or if something else is happening.

# defined here so it can be pickled
class CustomType(object):
    def __init__(self):
        self._count = 0

    def count(self):
        self._count += 1
        return self._count

class SessionView(BaseView):
    blueprint = testing_blueprint
    url = '/test-session'

    @route(get=True)
    def custom(self):
        if 'custom_type' not in flask.session:
            flask.session['custom_type'] = CustomType()

        return '{}'.format(flask.session['custom_type'].count())

class TestSession(object):
    def test_custom_type(self):
        ta1, _ = login()
        ta2, _ = login()
        assert ta1.get('/test-session/custom').body == '1' # Fails with a value of b'2'

With a vanilla Flask app this is not the case:

import flask
from flask_webtest import TestApp

app = flask.Flask(__name__)
app.config['SECRET_KEY'] = 'test'


# defined here so it can be pickled
class CounterType(object):
    def __init__(self, start):
        self._count = start

    def count(self):
        self._count += 1
        return self._count

    def __repr__(self):
        return self._count


@app.route('/count')
def count():
    flask.session['hits'] = CounterType(flask.session.get('hits', 0)).count()
    return '{}'.format(flask.session['hits'])


class TestSession(object):

    def test(self):
        counter = lambda ta:  ta.get('/count').body

        ta_1 = TestApp(app)

        assert ta_1.get('/count').body == b'1' 

        ta_2 = TestApp(app)
        assert counter(ta_2) == b'1'
        assert counter(ta_2) == b'2'


if __name__ == '__main__':
    app.run(debug=True)

1. In the Flask world, init_app() is used used for manager objects that are Globally instantiated but work in the context of multiple flask apps. Since the Logging instance doesn't really do that, each Logging instance is dedicated to a single app instance, all it's init could happen in __init__().
2. But, each Logging instance is operating on the Global root logger by default, with some help from clear_keg_handlers() to make sure we don't create duplicate handlers. In essence, this is creating a last-app-loaded-wins situation, which may not be desirable.

explicit is better than implicit

There are many generic ways to inject code before a response is generated, but I can't find any way to insert a response "middleware" (i.e. a way to modify responses for a certain class hierarchy).

Until #32 is settled, Click's upper bound needs to be < 5.0 to prevent a landslide of warnings.