cfpb / back-end Goto Github PK

I'd like to suggest that we merge this repository with the front-end repo. This does a few things:

1. Emphasizes that we are one team
2. Reduces the need for duplication across the two repos
3. Makes it easier to find development guidance

To do this, I recommend renaming front-end to development and placing all future documentation and best-practice guidelines in that repo. We can keep this repo open until the current issues are all closed out, but all future discussions can take place in the development repo.

What do you all think?

How do we generally design our APIs? How do they return data from different methods (GET, POST, PUT, etc)? How do they error? How do we secure our writable APIs?

These are all the questions I have off the top of my head regarding API design that I think we should work to standardize. As far as best practices go, I think this is a pretty good general rundown.

The front-ends have standardized on a single setup.sh file that invokes npm, bower, and grunt. In regulations-site, we've integrated that with the Python build, so that a pip install will run the frontendbuild.sh script.

I'd like @Ooblioob and @rosskarchner's thoughts on whether this is particularly useful for delivery, as well as the rest of the team's thoughts on how we can make this reusable. The build_ext and bdist_egg overrides are required for build_frontend to be invoked by pip, and those overrides make it a bit harder to just wrap this in a separate package that can be imported.

It's not impossible — that package just provides those overrides too. But this could make the process more complex for any projects that need to also override parts of setuptools. Of course, those cases might be doing something so completely different it doesn't really matter.

Thoughts?

Django logging should be standardized across all django apps.

• Log locations
• Formats (verbose, standard ,etc)
• Logs (django, django.request, django.security, etc)

This may also help with "Where are the Logs" in the ops model!

@hkeeler

Update repo description to something like A repository for the discussion and publication of the CFPB back-end team standards

We should standardize our back-end naming conventions (and perhaps have a conversation with FEWDs about standardizing variable/function/etc conventions as a whole).

As we're mostly a Python shop, PEP8 should provide some guidance.

So, yesterday we noticed that clicking on the "About" link from anywhere in eRegulations would take you instead to this page:

http://www.consumerfinance.gov/retirement/before-you-claim/about/

Which is a placeholder page in the retirement app.

This happened because both regulations-site and retirement have a URL with the name "About", and our header in regulations-site simply calls it by that name ({% url 'about' %}).

In the overall CFPB Django project, the urls.py contains the eRegs urls:

    # eRegulations
    url(r'^eregs-api/', include('regcore.urls')),
    url(r'^eregulations/', include('regulations.urls')),

Before the retirement urls:

if 'retirement_api' in settings.INSTALLED_APPS:
    urlpatterns += patterns('',
         url(r'^retirement/', include('retirement_api.urls')),
)

And so the retirement URL named "about" is overriding the eRegs URL named "about".

Django has a handy way to prevent this from happening, URL Namespaces.

Instead of the above URL configuration, we'd use:

    url(r'^eregs-api/', include('regcore.urls', namespace='regcore')),
    url(r'^eregulations/', include('regulations.urls', namespace='regulations')),
    url(r'^retirement/', include('retirement_api.urls', namespace='retirement')),

And then in our templates and all our URL-name matching lookups:

{% url 'regulations:about' %} 

Admittedly, that's a tall-order for some projects (regulations-site among them). It's something I'm going to work toward, and I think it's something we all should work on whenever we touch a new package.

I do think all our new development should presume that an app will be namespaced.

Thoughts?

We have many Django projects, pretty much all of them handling Django settings file(s) very differently. We need to come up with some form of standardization that we can apply across existing projects and can create skeleton for in Django project/app templates.

The issues, as I see them:

• Handling environment-specific settings (local_settings.py files, with sample local_settings.py files for deployment and local development? Using 12 Factor-style environment variables for all local settings?)
• Handling the distinction between our apps running in their own standalone Django project or as one of more apps within a larger Django project. How do we handling settings that could work in both places?

What other settings-related issues have people encountered that we should consider?

3 years or so ago, I cam back from DjangoCon, full or religion. The religion of breaking our Django projects out into per-app repositories.

Lately, I've been thinking that this has created more problems than it solves. It's complicates everything from developer workstation setup to deployment. It doesn't even seem like a common way to work with Django. It's been a trail of tears.

The direction I've been thinking about advocating for V1 is:

• New apps are always incubated in the project.
• consider "graduating" an app if:
  • we could use it in multiple projects (say, cf.gov and intranet)
  • This is a little squishy, but if an app (or set of apps) is big enough, and largely independent of a particular site (like eregs)
  • Or if there is external demand for it-- someone outside the bureau tells us they want use a particular app (or want to contribute to a particular app)

Because we didn't really get a lot of movement after the last meeting, I'd like to focus on one particular area: Django.

I've seen the term "normal Django app" bandied about as if it had meaning. If it does, I'm not entirely sure what that meaning is in our particular context, and I think it'd be something very useful to have defined before the new backend fellows start in January.

I don't really think this needs to be a survey of existing Django apps so much as a set of guides to what our pragmatic ideal might be. This is what someone would read if they're creating new apps.

I think this will focus in on a few specific areas (in no particular order):

• Configuration

  Is it handled in the same way we've already discussed (#1)?

• Basic app structure

  Generally (as far as I've seen), we create apps that provide APIs. What does that look like organizationally? Is it just basic Django? Are there any particular concerns we should be aware of?

• Common dependencies

  What dpeendencies do we already have available for common tasks (like Django Rest Framework for API creation). I don't want to discourage experimentation with new libraries, etc, but I want to make it easy to say, I'm creating an API, Django Rest Framework is already installed in production and that's what a bunch of other projects are using, I'll use that.

• Common patterns

  Building on the last point, would it be useful to have a some common patterns? Again, there are only so many ways to create an API with Django, so is there a general pattern we can define for that? I'm also thinking of Mike Brown's presentation yesterday and the need to ensure you know what your foreign keys have been up to.

• Fixtures and testing

  Is there anything particular about our approach to testing Django apps beyond plain Django unit tests? I know we use mock relatively frequently, and mocking can be somewhat confusing to get your head around at first.

• Security best-practices

  What are the things we should be aware of and things we should be doing in all our Django apps?

• Integration with our Django project

  This isn't, in my mind, a step-by-step for how to integrate into a larger Django project (and how we handle shared resources, etc), but rather a collection of concerns and possible approaches to consider.

• Front end integration (#5)

  This is a problem every project that provides a front-end is going to struggle with. How do we do it?

• Internationalization (#9)

  We should be doing this whereever possible. Is this just straight-up Django internationalization?

• Source control (#8)

• Performance/Good ORMing

Some very special and unique snowflakes (cough_eRegs_cough) might be too esoteric for some or all of these to apply, but I'm looking for useful starting points for someone starting to create a Django app at CFPB. It would be nice to put some of this into a Django template, but in my experience, every software project is that little bit different. I think written guides empower a developer to approach their particular project needs with our approach to Django in mind.

Thoughts? In today's meeting, I'd like to ask for two things:

1. Agreement on areas of Django, like the list above, where we can draw guidelines around what a normal CFPB Django app should be like.
2. Volunteers to tackle those areas.

Specifics about each of these areas can be discussed in issues to preserve the discussion for reference later.

We kind of touched on this in #5, but I'm wondering about the case where we have a normal python project (i.e. no strong front-end needs), as is the case with a lot of our legacy code. I'd like for us to have the convention that all our python projects are installable, because that makes including them in a Django project very easy, but some of our internal projects don't have that convention.

I'd like to add a simple setup.py to each of these repos, but want to throw this idea out there in case there are strong feelings about it. Here's a sample, based on eRegs and Python Hosted's simple example:

from setuptools import setup, find_packages
setup(
    name = "some_project_name_here",
    version = "0.1.0", 
    license = "public domain", 
    packages = find_packages(),
)

There are a host of other fields we could include (long description, authors, etc), but honestly I don't find much value in them and they're likely to get stale quickly. Thoughts?

How many of our projects are doing internationalization? I know @higs4281 has done some fantastic work on "Before You Claim" in this regard. I'm sure most of our projects are at least doing a little bit of l10n, even if it only relates to timezones.

Should we have a discussion about i18n and l10n as it relates to what we do on the back-end, with an eye toward generating some basic guidance?

Is there currently a standard around use of VMs (via Vagrant or other tools) to set up repeatable development environments? This might be helpful when setting up a new machine or in cases where different projects have conflicting requirements. Are there preferences around using VMs vs. Docker containers as an alternative way to set up repeatable and shareable environments?

This is the repo where we, back-end developers at CFPB, can discuss and write the standards and best practices for the work that we do. Our organization creates that logical separation between front-end and back-end development, so that's why the name "back-end" was chosen for this repo.

But it doesn't have to be that way. Does anyone have a suggestion that would also capture the wide variety of work that falls under our purview?

I'd be partial to "software development", but that seems to imply developers that don't share our roll don't develop software, which I don't think is fair.

"back-end" is also consistent with the existing front-end repository.

In the Dev COP discussion last week, cross-team code review was brought up as a nice-to-have, but often times non-team members don't necessarily know where it's appropriate to jump in. I think we can improve that one the back-end (and break down some of the barriers between our teams) with a Python style guide. This is also something that's come up in other conversations I've had over the past few weeks, and in a couple of code reviews I've participated in.

I know we generally try to follow PEP8, I know, but there are some idioms and conventions we might want to adopt for our own use that aren't covered by PEP8. Plus it's always nice to document the rationale we might have, as opposed to simply "because PEP8" — it opens it up to fully-informed reconsideration later.

An example that's more verbose than I'd anticipate us creating is Google's Python Style Guide. What's nice about Google's is it breaks down the rules into their pros and cons and decision-making.

There are three main concerns I'd have a style guide address, consistency, continuity, and readability.

• Consistency and continuity

  We have project teams that may pick up a project another team last worked one, and that was perhaps created by a different team altogether. We've got a lot of code that was writen by individuals that're no longer here, and we have people who've joined since they left.

  This state of affairs is likely to continue, and so ensuring some consistency in our style across projects can help mitigate some of the overhead that comes from this sort of churn. Our code should be as easy as possible for others to maintain and extend as needed, long after we're gone.

• Readability

  Part of what makes code easier for others to maintain and extend is its readability. With a shared understanding of, say, the way lambdas are used, you're less likely to need to spend a lot of time understanding a five-line lambda that's used to sort a list of dictionaries (example from my own past).

  In other words, you can spend less time trying to understand the mundane problems being solved, and more time understanding the way the more interesting problems are being solved.

I think this would also permit some more cross-team code reviews, because if we have a consistent style as a discipline, it means we can focus on more of the interesting bits of code, and less on variable naming, use of list comprehensions, that may vary from team to team, and than one of us from a different team might not be aware of.

I'm interested in what everyone thinks, if you have any favorite resources for Python style, or if you have strong feelings one way or the other about conventions or idioms we use, should be using, should not be using, etc. I would like to spend most of our meeting today discussing this.

Here are some resources I find helpful:

• PEP8
• Google's Python Style Guide
• The Hitchiker's Guide to Python

Ultimately consistency and continuity are the core of what this whole effort to create back-end standards is all about. To get that effort moving again, I want to go back down and start at the bottom, the way we're writing our code, and then we can go from there.

I also realize not all of us write Python, and perhaps as part of this we can come up with some language-agnostic conventions that would help across languages. I'd welcome the input of those of you working on non-Python projects here.

Based on a few conversations I've been involved with recently, we need to define what a "normal" Django app is at CFPB, how it works, and what it includes.

To help with my work on #5, I've created a Django app skeleton that includes some rudimentary front-end stuff (still an area that needs work and collaboration from @cfpb/front-end-team-admin). It is wrapped up with a script to make it easy to create a new CFPB Django app, along with some test scaffolding (Travis, basic test requirements, flake8, and tox for running multiple Python environment tests locally).

This is all in an installable Python package.

I'd welcome any thoughts.

The skeleton is here:
https://github.com/willbarton/cfpb-django-app

And the resulting project is here:
https://github.com/willbarton/myapp