miserlou / zappa-django-utils Goto Github PK

View Code? Open in Web Editor NEW

144.0 11.0 19.0 45 KB

A handy Django application to supercharge your Zappa deployments.

Home Page: https://www.zappa.io

License: MIT License

Python 100.00%

zappa-django-utils's Introduction

zappa-django-utils 0.4.1: final release!

Final Release

Thanks to everyone who has supported this package of Zappa utilities for Django! Many of these utilities have become deprecated as new features have been added to AWS, or split into their own projects such a django-s3-sqlite: https://github.com/flipperpa/django-s3-sqlite

We're making one final release and will not be accepting further issues or pull requests.

Description

Small utilities for making Zappa deployments slightly easier for Django applications.

This project was inspired by Edgar Roman's Zappa Django Guide.

Installation

Install via pip:

$ pip install zappa-django-utils

Add to your installed apps:

INSTALLED_APPS += ['zappa_django_utils']

Usage

Using an S3-Backed Database Engine - DEPRECATED!

Use django-s3-sqlite instead, as it has an updated SQLite driver compatible with current versions of Django:

https://github.com/flipperpa/django-s3-sqlite

Creating a Postgres Database

Once you have your RDS set up, your VPC/Subnet/Security Groups set up, and your DATABASES setting set up, you can create the database with:

$ zappa manage <stage> create_pg_db

Then you're ready to python manage.py makemigrations and zappa update; zappa manage <stage> migrate!

Creating a Default Admin User

You'll probably need a default user to manage your application with, so you can now:

$ zappa manage <stage> create_admin_user

Or you can pass some arguments:

$ zappa manage <stage> create_admin_user one two three

This will internally make this call:

User.objects.create_superuser('one', 'two', 'three')

Now log in and immediately change the admin user's email and password.

Creating/Dropping a Postgres Schema

You can create a Postgres schema with:

$ zappa manage create_pg_schema

and drop it with:

$ zappa manage drop_pg_schema

License

zappa-django-utils's People

Contributors

Stargazers

Watchers

Forkers

jjorissen52 mbeacom edgarroman pizzapanther dangayle grantmcconnaughey flipperpa mwalker formulka jnoortheen luzfcb tutzy1 kevinjfq tolufash casualuser adambrdn ehsandeihimi dhawalagrawal devops-factory-rd

zappa-django-utils's Issues

Error: Please define stage 'create_pg_db' in your Zappa settings.

Intro: I am deploying a django app to AWS lambda severlessly using zappa. My RDS instance has a postgres database. I am watching rich jones djangocon video on how to deploy django app severlessly using zappa. So far I have managed to reach the part where I need to add a database to my project. I have already done pip install zappa-django-utils and added it to my INSTALLED_APPS. Now when I try to run

zappa manage create_pg_db production I get the error:

Error: Please define stage 'create_pg_db' in your Zappa settings.

I even tried zappa manage create_pg_db I am still getting the same error

Below is how my zappa_settings.json file looks:

{
    "production": {
        "aws_region": "us-east-1",
        "django_settings": "Cool.settings",
        "profile_name": "default",
        "project_name": "cool",
        "runtime": "python3.6",
        "s3_bucket": "cool-7dsfsdf5",
        "project_directory": "/tmp/code",
        "slim_handler": true,
        "vpc config": {
            "SubnetIds": [
                "subnet-3132ss13b",
                "subnet-321321319",
                "subnet-2c2313223",
                "subnet-5ljlkjljd",
                "subnet-132121357",
                "subnet-f925f9c7"
            ],
            "SecurityGroupIds": [
                "sg-a9asdasd"
            ]
        }
    },
    "production_ap_northeast_1": {
        "aws_region": "ap-northeast-1",
        "extends": "production"
    },
    "production_ap_northeast_2": {
        "aws_region": "ap-northeast-2",
        "extends": "production"
    },
    ... All regions..
}

How do I take it from here

[ERROR] CommandError: Unknown command: 'create_pg_db'

I followed the tutorial on https://romandc.com/zappa-django-guide/walk_database/
and pip installed the package, added the value to installed zappa, configured the aws aurora postgres then when I do

zappa manage project_name create_pg_db

I get

[ERROR] CommandError: Unknown command: 'create_pg_db'
Traceback (most recent call last):
  File "/var/task/handler.py", line 609, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 243, in lambda_handler
    return handler.handler(event, context)
  File "/var/task/handler.py", line 408, in handler
    management.call_command(*event['manage'].split(' '))
  File "/var/task/django/core/management/init.py", line 105, in call_command
    raise CommandError("Unknown command: %r" % command_name)[END]

did I miss anything, what can I do to fix this?

Dropping PG DB

Generally dangerous but super helpful when prototyping a new service.

If I'll create a PR to add this helper function, would it be merged? :)

s3sqlite: check cache before downloading

[Feature Request] Additional options upon first DB creation

I use the HStoreField to store JSON strings in the database. In order to do so you have to connect to your newly created Django database and run the query:

CREATE EXTENSION IF NOT EXISTS hstore;

It would be great to offer a command line option for this (e.g. --hstore)

s3sqlite OperationalError

I have set up a simple Django REST API which I deployed using Zappa. I also want to use the s3sqlite database as described in your blog. The default django url works just fine, but when I go to https://****.execute-api.eu-central-1.amazonaws.com/dev/users/ or try to login as admin I get:


Environment:


Request Method: POST
Request URL: https://****.execute-api.eu-central-1.amazonaws.com/dev/api-auth/login/

Django Version: 2.0.6
Python Version: 3.6.1
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'rest_framework',
 'storages',
 'zappa_django_utils']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback:

File "/var/task/django/db/backends/utils.py" in _execute
  85.                 return self.cursor.execute(sql, params)

File "/var/task/django/db/backends/sqlite3/base.py" in execute
  303.         return Database.Cursor.execute(self, query, params)

The above exception (no such table: auth_user) was the direct cause of the following exception:

File "/var/task/django/core/handlers/exception.py" in inner
  35.             response = get_response(request)

File "/var/task/django/core/handlers/base.py" in _get_response
  128.                 response = self.process_exception_by_middleware(e, request)

File "/var/task/django/core/handlers/base.py" in _get_response
  126.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/var/task/django/views/generic/base.py" in view
  69.             return self.dispatch(request, *args, **kwargs)

File "/var/task/django/utils/decorators.py" in _wrapper
  62.             return bound_func(*args, **kwargs)

File "/var/task/django/views/decorators/debug.py" in sensitive_post_parameters_wrapper
  76.             return view(request, *args, **kwargs)

File "/var/task/django/utils/decorators.py" in bound_func
  58.                 return func.__get__(self, type(self))(*args2, **kwargs2)

File "/var/task/django/utils/decorators.py" in _wrapper
  62.             return bound_func(*args, **kwargs)

File "/var/task/django/utils/decorators.py" in _wrapped_view
  142.                     response = view_func(request, *args, **kwargs)

File "/var/task/django/utils/decorators.py" in bound_func
  58.                 return func.__get__(self, type(self))(*args2, **kwargs2)

File "/var/task/django/utils/decorators.py" in _wrapper
  62.             return bound_func(*args, **kwargs)

File "/var/task/django/views/decorators/cache.py" in _wrapped_view_func
  44.         response = view_func(request, *args, **kwargs)

File "/var/task/django/utils/decorators.py" in bound_func
  58.                 return func.__get__(self, type(self))(*args2, **kwargs2)

File "/var/task/django/contrib/auth/views.py" in dispatch
  66.         return super().dispatch(request, *args, **kwargs)

File "/var/task/django/views/generic/base.py" in dispatch
  89.         return handler(request, *args, **kwargs)

File "/var/task/django/views/generic/edit.py" in post
  141.         if form.is_valid():

File "/var/task/django/forms/forms.py" in is_valid
  179.         return self.is_bound and not self.errors

File "/var/task/django/forms/forms.py" in errors
  174.             self.full_clean()

File "/var/task/django/forms/forms.py" in full_clean
  377.         self._clean_form()

File "/var/task/django/forms/forms.py" in _clean_form
  404.             cleaned_data = self.clean()

File "/var/task/django/contrib/auth/forms.py" in clean
  195.             self.user_cache = authenticate(self.request, username=username, password=password)

File "/var/task/django/contrib/auth/__init__.py" in authenticate
  70.             user = _authenticate_with_backend(backend, backend_path, request, credentials)

File "/var/task/django/contrib/auth/__init__.py" in _authenticate_with_backend
  116.     return backend.authenticate(*args, **credentials)

File "/var/task/django/contrib/auth/backends.py" in authenticate
  16.             user = UserModel._default_manager.get_by_natural_key(username)

File "/var/task/django/contrib/auth/base_user.py" in get_by_natural_key
  44.         return self.get(**{self.model.USERNAME_FIELD: username})

File "/var/task/django/db/models/manager.py" in manager_method
  82.                 return getattr(self.get_queryset(), name)(*args, **kwargs)

File "/var/task/django/db/models/query.py" in get
  397.         num = len(clone)

File "/var/task/django/db/models/query.py" in __len__
  254.         self._fetch_all()

File "/var/task/django/db/models/query.py" in _fetch_all
  1179.             self._result_cache = list(self._iterable_class(self))

File "/var/task/django/db/models/query.py" in __iter__
  53.         results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)

File "/var/task/django/db/models/sql/compiler.py" in execute_sql
  1068.             cursor.execute(sql, params)

File "/var/task/django/db/backends/utils.py" in execute
  100.             return super().execute(sql, params)

File "/var/task/django/db/backends/utils.py" in execute
  68.         return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)

File "/var/task/django/db/backends/utils.py" in _execute_with_wrappers
  77.         return executor(sql, params, many, context)

File "/var/task/django/db/backends/utils.py" in _execute
  85.                 return self.cursor.execute(sql, params)

File "/var/task/django/db/utils.py" in __exit__
  89.                 raise dj_exc_value.with_traceback(traceback) from exc_value

File "/var/task/django/db/backends/utils.py" in _execute
  85.                 return self.cursor.execute(sql, params)

File "/var/task/django/db/backends/sqlite3/base.py" in execute
  303.         return Database.Cursor.execute(self, query, params)

Exception Type: OperationalError at /api-auth/login/
Exception Value: no such table: auth_user

Am I missing something?

Vacuum SQLite database occasionally

To keep the DB size as small as possible, a VACUUM should be issued occasionally. This is more a note to self than anything else:

        # If running Zappa with S3 SQLite, perform a vacuum to keep the database size small.
        if settings.DATABASES['default']['ENGINE'] == 'zappa_django_utils.db.backends.s3sqlite':
            cursor = connection.cursor()
            cursor.execute('VACUUM;')
            cursor.close()

Does Zappa incur data transfer costs through API Gateway?

It looks to me that my Zappa Site is being server through API Gateway based on the logs, which uses the data transfer model of EC2 (1st GB free then $0.09 for next 9.99 TB). Will Zappa incur these charges?

DB Migrations

I'm using this package for PyTexas 2017's DB (https://www.pytexas.org/2017/) on Heroku since the data is light and doesn't change very much. Anyways I recently tried to do some migrations on the DB. I still had my web container running and started up a separate container to run the migrations. Everything ran fine but the web container never got the updated DB with the new columns. I also tried going into maintenance mode so when I ran the migration nothing would interfere. It still wouldn't stick. Finally, I downloaded the DB from S3 and ran the migrations on my local machine and re-uploaded the database. I also turned off versioning for the S3 bucket.

So my question is migration something special that I need to work around? Or is versioning a problem?

Are multiple containers writing to the DB a problem? I'm assuming not since Amazon Lambda scales infinitely. I'm guessing 2 Heroku containers accessing a DB at once shouldn't be a problem?

s3sqlite: check db hash has changed before uploading

Create MySQL Database

We can create a PG DB, we should throw a bone to the MySQL/Aurora people too..

Password not displayed after creating default admin user

When creating a new superuser with the default credentials, a random password is calculated in create_admin_user.py but it is not displayed. The random password is not reproducible, which makes this function useless.

The default password should be shown with the username when the function is complete.

create_pg_db.py: can't connect to a database which doesn't exist

In this line you're attempting to connect to a database which doesn't yet exist.

zappa-django-utils/zappa_django_utils/management/commands/create_pg_db.py

Line 17 in 61bb3d5

con = connect(dbname=dbname, user=user, host=host, password=password)

Instead you should rather connect to the postgres database.

s3sqlite: Python 3 Support

Related: Miserlou/lambda-packages#66

Tests! Tests! Tests!

Needs tests.

s3sqlite database acts like its empty

Just wondering if i have to do anything to update my s3sqlite database after deploying with zappa. I have a sqlite database in an aws bucket that ive been using for local development without any problems. Then once i deployed my project with Zappa, a django project, the database acts as if it is empty.

Create VPC For Internet Access + Database Access

Data loss in SQLite occurring occasionally: DB reverts to prior state

I am running s3sqlite on Amazon S3 with Zappa on AWS Lambda. I'm using Django 2.0 and Wagtail 2.0. It is all bleeding edge stuff - Wagtail 2.0 is straight off master - so this issue will be for collecting data and brainstorming a solution.

I have everything running at https://PyPhilly.org/ Occasionally, when I create a new page in Wagtail, it'll appear for a minute, but disappear a minute later, like the DB has reverted to a previous state. This may be because the database is in Lambda memory, but isn't being PUT back to S3 in the close() method here: https://github.com/Miserlou/zappa-django-utils/blob/master/zappa_django_utils/db/backends/s3sqlite/base.py#L58

Is anyone else experiencing issues like this? Does anyone know if there are times when Django's DB Engine won't call the close() method, which might be causing the DB to disappear from memory, rather than being written back to S3?

I started work on a P.R. to write after each execute() method, but that's going to be problematic for pages with multiple inserts, especially as the database size grows. Getting and putting the DB file back and forth as it grows seems like begging for trouble.

I've just had a P.R. merged which should make seeing problems in zappa tail a bit easier here: https://github.com/Miserlou/zappa-django-utils/pull/15/files#diff-8dac6662b0890c87282111cf1c72ec12L78 (see line 78 of the diff). I'll update this issue with more information as I find it.

Does anyone know what kind of performance implications there would be as the database grows in size with Amazon's infrastructure set up between S3 and Lambda? What if it grows to be 50 MB? 500 MB? The DB I'm working with is < 1MB, and I haven't tested scaling (yet).

Any thoughts, feedback, or data points would be appreciated!