hjwp / book-tdd-web-dev-python Goto Github PK

Following the instructions in chapter one you say we can ignore db.sqlite3.
However you never did a syncdb before so the file does not exist unless I missed it, so it might confuse someone to not find a file which should be there..

For this section of the book:

##############################################################################
Switching to Using Unix Sockets
When we want to serve both staging and live, we can’t have both servers trying to use port 8000. We could decide to allocate different ports, but that’s a bit arbitrary, and it would be dangerously easy to get it wrong and start the staging server on the live port, or vice versa.

A better solution is to use Unix domain sockets—​they’re like files on disk, but can be used by Nginx and Gunicorn to talk to each other. We’ll put our sockets in /tmp. Let’s change the proxy settings in Nginx:

server: /etc/nginx/sites-available/superlists-staging.ottg.eu
server {
listen 80;
server_name superlists-staging.ottg.eu;

location /static {
    alias /home/elspeth/sites/superlists-staging.ottg.eu/static;
}

location / {
    proxy_pass http://unix:/tmp/superlists-staging.ottg.eu.socket;
}

}
Now we restart Gunicorn, but this time telling it to listen on a socket instead of on the default port:

elspeth@server:$ sudo systemctl reload nginx
elspeth@server:$ ./virtualenv/bin/gunicorn --bind
unix:/tmp/superlists-staging.ottg.eu.socket superlists.wsgi:application
And again, we rerun the functional test again, to make sure things still pass:

$ STAGING_SERVER=superlists-staging.ottg.eu python manage.py test functional_tests
[...]
OK
Hooray, a change that went without a hitch for once! Moving on.
##############################################################################

I was not so lucky. I was able to create the socket without issue, but the pages wouldn't load. I was seeing the following error logs in nginx:

2018/05/11 16:02:12 [crit] 8944#0: *1 connect() to unix:/tmp/superlists-staging.somesite.com.socket failed (2: No such file or directory) while connecting to upstream, client: 111.111.111.111, se
rver: superlists-staging.somesite.com.socket, request: "GET /favicon.ico HTTP/1.1", upstream: "http://unix:/tmp/superlists-staging.somesite.com.socket:/favicon.ico", host: "superlists-staging.somesite.com", referrer: "http://superlists-staging.somesite.com/"

"No such file or directory" - indicated that nginx couldn't see the socket even though the socket was definitely there:

[nginxuser@otg-staging-server ~]$ ls /tmp/
superlists-staging.somesite.com.socket
systemd-private-xxxxxxxxxxxxxxx-chronyd.service-z1HGQH
systemd-private-xxxxxxxxxxxxxxx-nginx.service-SCaqlr

I found the following on the fedora wiki:
http://fedoraproject.org/wiki/Features/ServicesPrivateTmp

This meant that every service sees a completely different /tmp and can only see its own files in that directory, so nginx was unable to see the gunicorn /tmp files.

To fix it I created a directory called /var/sockets with appropriate permissions and replaced the unix:/tmp/ with unix:/var/sockets/ in both the nginx config file and the gunicorn bind command.

The errors went away and I was able to browse to the website using the new socket configuration.

This is an issue that is probably limited to CentOS 7. (P.S. I had to create a custom policy for selinux as well using the output from "sudo grep nginx /var/log/audit/audit.log | audit2allow -m nginx" following the instructions found here: http://nts.strzibny.name/allowing-nginx-to-use-a-pumaunicorn-unix-socket-with-selinux/)

There is some question of how to do this in the comments section with a few people unsure of what steps are necessary. One user suggested installing Homebrew and running "brew install geckodriver".

It's useful to know how to place an executable in /usr/local/bin oneself and straightforward to do. Would you like me to make a PR with instructions?

With Python3, the URL is updated to https://diveintopython3.problemsolving.io/.

In chapter_10_production_readiness.asciidoc, when the requirements.txt is created by adding, there's still an old Django version displayed even though in the next sourcecode section, the Django version is 4.2.11:

[subs="specialcharacters,quotes"]
----
$ *pip freeze | grep -i django*
Django==4.2.7 << HERE

$ *pip freeze | grep -i django >> requirements.txt*
$ *pip freeze | grep -i gunicorn >> requirements.txt*
$ *pip freeze | grep -i whitenoise >> requirements.txt*
----

It should be this:

https://hg.mozilla.org/mozilla-central/file/tip/testing/geckodriver

In ch11l032, this is introduced, to test the HTML5 validation:

self.wait_for(lambda: self.browser.find_elements_by_css_selector(
 '#id_text:invalid'
 ))

However, it isn't testing anything.

Because we are using find_elementS_by_css_selector(...) if the element is not found an empty list will be returned and no exception will be raised. Therefore our wait_for immediately returns after the first try and we didn't test anything (try making it fail on purpose by looking for a different id, it won't fail).

To fix that, the solution is simple, remove the S. That way it'll either find that one element, or raise a NoSuchElementException, which is a WebDriverException, so our wait_for works as expected 🙂

While running through this chapter I noticed a small issue that I thought I would flag with the path reference towards the end of the chapter:

https://github.com/hjwp/Book-TDD-Web-Dev-Python/blame/master/chapter_javascript.asciidoc#L737-L738

I believe these should be:

<script src="../static/jquery-3.3.1.min.js"></script>
<script src="../static/list.js"></script>

In Chapter 23: Test Isolation, and “Listening to Your Tests” in the first code snippter after Moving Down to the Forms Layer it says:

class NewListForm(models.Form):

Which was irritating, as there is no models.Form (PyCharm was complaining). It wasn't a big issue, as it is corrected a few pages later, but it took me a few minutes because I thought I missed something.

asciidoctor will warn of the following errors when make chapter_10_production_readiness.html is ran:

asciidoctor: WARNING: chapter_10_production_readiness.asciidoc: line 374: callout list item index: expected 1, got 3
asciidoctor: WARNING: chapter_10_production_readiness.asciidoc: line 376: callout list item index: expected 2, got 4

When a source block is annotated by callout numbers, same numbers should be used in the annotation text section. But this is not the case in chapter_10_production_readiness.asciidoc:

.Dockerfile (ch09l005)
====
[source,dockerfile]
----
FROM python:slim

RUN python -m venv /venv
ENV PATH="/venv/bin:$PATH"

COPY requirements.txt requirements.txt  <1>
RUN pip install -r requirements.txt  <2>

COPY src /src

WORKDIR /src

CMD python manage.py runserver
----
====

<3> We COPY our requirements file in, just like the src folder.

<4> Now instead of just installing Django, we install all our dependencies
  by pointing pip at the _requirements.txt_ using the `-r` flag.
  Notice the `-r`.

Changelist

• Change <3> We COPY our requirements file in... into <1> We COPY our requirements file in...
• Change <4> Now instead of just installing Django... into <2> Now instead of just installing Django...

Instead of importing

from django.core.urlresolvers import reverse

we could import

from django.urls import reverse

Importing from urlresolvers is only kept for backwards compatibility, and it isn't working
in Django 2.0 anymore.
See: https://docs.djangoproject.com/en/1.11/ref/urlresolvers/

I wanted to take a closer look into the repository for some possible contributions by cloning it to my local machine and followed README for setup.

But I noticed right away that make install can't be ran without errors:

$ make install
which uv && uv venv .venv || python -m venv .venv
/bin/bash: line 1: python: command not found
make: *** [Makefile:50: .venv/bin] Error 127

Shouldn't python -m venv .venv be python3 -m venv .venv? I do have python set as an alias for python3 in my .bashrc but it isn't used because the Bash used by make (SHELL := /bin/bash defined in Makefile) is a non-interactive shell.

It's a bit confusing when you write:

item.save()
item.full_clean()

while the name of the tests is test_cannot_save_empty_list_item
Because in the end... the item is saved. It can be verified by putting the following right after the with block:

self.assertEqual(Item.objects.count(), 0)

Two potential solutions

• Maybe just swapping the order of the 2 lines might make things less confusing :)

  item.full_clean()
  item.save()

• Another solution would be to simply rename the tests from test_cannot_save_empty_list_item to test_cannot_validate_empty_list_item
  And removing the call to save() alltogether.

Hi Harry. Firstly, thanks for writing this book, I am learning a lot.

I went through the Spike for implementing a Persona Login and encountered an error, then went through and did the "real" (test-driven) implentation and encountered the same error.

MultiValueDictKeyError at /accounts/login
""assertion'"

It has something to do with the following code:
user = authenticate(assertion=request.POST["assertion"])
Unfortunately I don't know why this is happening.

When I run the functional tests, I get
selenium.common.exceptions.TimeoutException: Message: ''

When make book.html, Error displayed:

asciidoctor: ERROR: book.asciidoc: line 18: include file not found: /Users/swot/offline/Book-TDD-Web-Dev-Python/part1.harry.asciidoc
asciidoctor: ERROR: book.asciidoc: line 30: include file not found: /Users/swot/offline/Book-TDD-Web-Dev-Python/part2.harry.asciidoc
asciidoctor: ERROR: book.asciidoc: line 42: include file not found: /Users/swot/offline/Book-TDD-Web-Dev-Python/part3.harry.asciidoc

I am stucked on chapter with sockets.
I did everything according to the guide:

/etc/nginx/sites-avalible/malomalsky.studio

server {
listen 80;
server_name malomalsky.studio;

location /static/ {
    alias /home/malomalsky/sites/malomalsky.studio/static;
    }

location / {
    proxy_pass http://unix:/tmp/malomalsky.studio.socket;
    }
}

../virtualenv/bin/gunicorn --bind unix:/tmp/malomalsky.studio.socket superlists.wsgi:application

Gunicorn works without any errors, but i have 502 Bad request error.

Ubunty 18

In

elspeth@server:$ set -a; source .env; set +a

It would be nice to explain the set -a; & set +a. I found out by googling (https://stackoverflow.com/a/43267603/4490991)

First, the Deasnakes PPA is actually the Deadsnakes PPA, and second, the current link is not working:

https://launchpad.net/~fkrull/+archive/ubuntu/deadsnakes

That link gives me an error. I was able to go here:

https://launchpad.net/~deadsnakes

But I am uncertain whether that is the intended destination.

The end line of function _update_settings

append(settings_path, '\nfrom .secret_key import SECRET_KEY')

will fail silently bacause of the newline character('\n'). Should change to

run(f"echo '\nfrom .secret_key import SECRET_KEY' >> {settings_path}")

Explains in fabric/issues/1598

In chapter 5 you do the following just to make the test pass and say that is not what we want..
Maybe you should make it more clear to NOT do that because at that point the test is passing, is that what you want?
If you want to have this intermediate step then you can at least say that the functional test is still NOT passing, so the job is clearly not done yet..

from django.http import HttpResponse
from django.shortcuts import render

def home_page(request):
    if request.method == 'POST':
        return HttpResponse(request.POST['item_text'])
    return render(request, 'home.html')

And by the way the following seems a bit excessive to me, it basically uses the same logic that can be used in production.. I know the client will be introduced later but it looks like at this point I have to know already how to implement the solution to actually write a test to prove it..

self.assertIn('A new list item', response.content.decode())
expected_html = render_to_string(
    'home.html',
    {'new_item_text':  'A new list item'}
)
self.assertEqual(response.content.decode(), expected_html)

In chapter04 you suggest to commit a change to the functional test (git commit -am 'Functional test now checks..')

Which it's fine, but maybe it suggests people that it's ok to commit broken tests, which should be avoided.
I would commit it only leaving a "skip" maybe or otherwise don't commit it at all and wait for it to pass first..

Hello, I'm reading early release book and there was something I encounter that's not really your fault but I think it's probably worth mentioning because it took me a couple hours to figure out how to do bug and it could save some people a lot of time.

Early in the book, you show a screenshot of how are your project folder should look and there's a file geckodriver.log in it.

I couldn't figure out how to get this to generate for the life of me until I found one of my other projects that did have it. so i deleted the file, ran it and got a log!

Anyways, it turns out after downgrading selenium the log file generated again so something in their new updates. I guess does not automatically generate logs which might be confusing to some users!

certainly was for me.

cheers

In the online version of the book (Third Edition Early Release), in the section on Installing Django and Selenium, I think that quotation marks might be needed around the Python command when invoking the interpreter with -c.

Currently, the book suggests this command:

(.venv) $ python -c from selenium import webdriver; webdriver.Firefox()

I had to add quotation marks to make it run:

(.venv) $ python -c "from selenium import webdriver; webdriver.Firefox()"

I tried on Windows 10 (Git Bash and PowerShell), Linux (bash), and Mac (zsh). Quotation marks were needed in each case.

I don't know if quotation marks are strictly necessary in all cases though, as the docs only say that "it is usually advised to quote command in its entirety".

I'm looking forward to the Third Edition! It's great to have access to the early release as it becomes available.

In chapter 08, "Debugging Tips" part, the error logs of Nginx was written as

/var/log/error.log

But on Ubuntu, the Nginx error log should be

/var/log/nginx/error.log

There are typo errors on "Map your domains to the server" chapter in server-quickstart.md file.

The word configration in the sentence "Once you have a domain, you need to set up a couple of A-records in its DNS configration,... "

This is minor but an uppercase is missing in "...you may need to wait. some registrars..."

In Chapter 2. Extending Our Functional Test Using the unittest Module the functional_tests.py looks like the following:

from selenium import webdriver
import unittest

class NewVisitorTest(unittest.TestCase):  1

    def setUp(self):  3
        self.browser = webdriver.Firefox()

    def tearDown(self):  3
        self.browser.quit()

    def test_can_start_a_list_and_retrieve_it_later(self):  2
        # Edith has heard about a cool new online to-do app. She goes
        # to check out its homepage
        self.browser.get('http://localhost:8000')

        # She notices the page title and header mention to-do lists
        self.assertIn('To-Do', self.browser.title)  4
        self.fail('Finish the test!')  5

        # She is invited to enter a to-do item straight away
        [...rest of comments as before]

if __name__ == '__main__':  6
    unittest.main(warnings='ignore')  7

I found that self.browser.quit() does not close firefox, and instead had to use self.browser.close()

Reading the documentation for selenium, it seems like this is not expected and quit() should have worked.

I am using MacOS Sierra, python3.6, selenium 3.5.0, geckodriver 0.18.0, FF 52.1.0 (64-bit)

When creating the .gitignore file on Windows OS the "" are included in the .gitignore which causes git to not ignore the files.

The following lines in Chapter 1 need to be changed to accomodate Windows users:

From:

$ echo "db.sqlite3" >> .gitignore
$ echo "geckodriver.log" >> .gitignore
$ echo "virtualenv" >> .gitignore

To:

$ echo db.sqlite3 >> .gitignore
$ echo geckodriver.log >> .gitignore
$ echo virtualenv >> .gitignore

import random
chars = 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)'
key = ''.join(random.SystemRandom().choice(chars) for _ in range(50))

The above method for secret key generation is good that it's cryptographically secure and fairly strong, but I would advocate for a different method:

import os
import base64
key = base64.b64encode(os.urandom(256 // 8))

The current method is a bit unclear as to how much entropy there is in the key (currently 50 * log2(50) = 282 bits). Drawing random bytes directly from os.urandom (the same CSPRNG that random.SystemRandom uses) then base64-encoding them provides a very clear indication of strength: whatever the argument to urandom is (in bytes, //8 for bits).

There's a mismatch between the prerequisite and preface version listings.
pre-requisite-installations says it was tested on Python 3.12 and might require editing or produce different messages and error logs for other Python versions, whereas the preface notes the book was updated for use with Python 3.11.

The image embedded on the pre-requisite-installations page shows the 3.11 installer as well.

In chapter 5 I'm not sure why you really need database migrations.

So far there were no models and 0 data in the database (and you were using sqlite anyway) so a syncdb would be perfectly enough.
And if you're db is messed up and you're still playing around you can also just remove the .db file..

Also because (how I worked at least) we had only one migration per release, if for any single tiny change there is a different migration it gets very messy and slow..
Maybe you could at least say that migrations are not actually necessary now but as a general practice is good to get used using them?

Another broken link for you, on http://www.obeythetestinggoat.com/book/chapter_philosophy_and_refactoring.html#_on_refactoring

But, as Kent Beck puts it in Test-Driven Development: By Example, links to http://www.obeythetestinggoat.com/book/chapter_philosophy_and_refactoring.html#tddbe.

Hi,

The section: 'What to do next' -> 'Test for Graceful Degradation' mentions Persona.

At first, I figured I missed something somewhere but after a quick google search I realized it was a leftover from the first edition

Solution could as simple as removing the section.

In chapter 5 you use a redirect to re-render the page with the desired post values.
A redirect is usually a more generic way to avoid code duplication, but in this case also just doing this would be enough, and would not have the overhead of a redirect.

def home_page(request):
    template_values = {}
    if request.method == 'POST':
        item_text = request.POST['item_text']
        Item.objects.create(text=item_text)
        template_values['new_item_text'] = item_text

    return render(request, 'home.html', template_values)

In chapter 6.2, in the line

Later on, we’ll refactor out a general wait_for

the later on links to chapter 6.2, but is should link to chapter 11.2

Right there, just under The Git version control system, some words ceased to exist. There were not erased, only never pushed.

This is available for any platform, at http://git-scm.com/. On Windows, this comes with the

In chapter04 (I tried to comment directly in the code but the asciidoc doesn't let me by the way) there is this:

    self.assertTrue(any(row.text) == '1: Buy peacock feathers' for row in rows)

It might be also written like this I think, which would be even easier adding another variable for the text in the rows.

 self.assertIn('Buy peacock feathers', [row.text for row in rows])

Chapter http://www.obeythetestinggoat.com/book/chapter_post_and_database.html#_the_django_orm_and_our_first_model:

later on links to http://www.obeythetestinggoat.com/book/chapter_post_and_database.html#rewrite-model-test which returns to the top of the current page.

[chapter advanced forms] should display a different string, shouldn't it?

In the deployment guide after the nginx installation we perform steps to remove the default symlink and set up a proxy pass to django app on port 8000 using a command:
elspeth@server:$ sudo ln -s ../sites-available/$SITENAME /etc/nginx/sites-enabled/$SITENAME

However nginx did not render the relative symlink. My suggestion would be to replace it with:
elspeth@server:$ sudo ln -s etc/nginx/sites-available/$SITENAME /etc/nginx/sites-enabled/$SITENAME

And remove the relative paths in commands in general or add a pwd before them as it can take to reproduce the desired effect otherwise.

Hey @hjwp,

I hope you are doing well.

I have sent you an email explaining in great detail my issue with FT up until Chapter 7.

I am at Chapter 9 and I cannot deploy now the site due to the current issue.

Can you please check your email and provide me with feedback?

Cheers.

My code:

def new_list(request):
	"""
	item.objects.create 是创建Item对象的简化方式无需再掉用.save()方法
	"""
	form = ItemForm(data=request.POST)
	if form.is_valid():
		list_ = List()
		print(request)
		list_.owner = request.user
		list_.save()
		form.save(for_list=list_)
		return redirect(list_)
	else:
		return render(request, 'home.html', {"form": form})

        @patch('lists.models.List')
	@patch('lists.forms.ItemForm')
	def test_list_owner_is_saved_if_user_is_authenticated(self, mockItemFormClass, mockListClass):
		user = User.objects.create(email='[email protected]')
		self.client.force_login(user)
		#print(settings.AUTHENTICATION_BACKENDS[0])
		self.client.post('/lists/new', data={'text': 'new item'})
		
		mock_list = mockListClass.return_value
		self.assertEqual(mock_list.owner, user)

Description

On page 24 in the pdf version of the book, we have just added a failing unit test (test_bad_maths). And the step to run the new test is missing relative location of manage.py.

The line:

$ python3 manage.py test

Should be:

$ python3 ../manage.py test

Fantastic book so far (thank you!), though just wanted to point out a minor inconsistency with the chapter title and URL numbering that I've just discovered.

The left hand navigational menu item on the obeythetestinggoat.com site for '13. Dipping Our Toes, Very Tentatively, into JavaScript' points to http://www.obeythetestinggoat.com/book/chapter_14.html, whilst '14. Deploying Our New Code' points to: http://www.obeythetestinggoat.com/book/chapter_13.html.

The same behaviour can be seen on the TOC page too (http://www.obeythetestinggoat.com/pages/book.html#toc).

Nothing too serious, tough though you might like to know!

In chapter 3 one of the test methods is called

def test_home_page_returns_correct_html(self):

this is fine except that it might suggest that you are checking the whole HTML generated, while instead you are checking for the parts you're interested in.

Maybe there could be even two tests:

• valid_html (which checks for starting and ending html)
• contains_to_do (check that the string To-Do is part of the body)

to make it even more clear

I would like to add next chapter link at the bottom of this page https://www.obeythetestinggoat.com/book/chapter_unit_test_first_view.html
to make it easier to get to the next page as suggested by the other user. Is it ok that I go ahead?

It's not that important but I just was curious to get the tests running.
First problem was that also cssselect was not installed (so it should be added to tests-requirements.txt), but after that I get this error:

'output:\n', "fatal: '/home/andrea/projects/book_original/source/chapter_11/superlists' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n")

I've done a git submodule update --init
but I suspect I need write access to these repos as well is that correct?

In ch07l006, there is an useless assertion that sneaked in.

Basically, that test:

self.assertNotIn('make a fly', page_text)

is useless because in test_multiple_users_can_start_lists_at_different_urls Edith did not create a second list item with ...make a fly. So that particular assertion would never fail and just brings a little bit of confusion.

If you have updated Python to 3.7 version, you will get the following error when trying to do the python3 manage.py runserver command.

"SyntaxError: Generator expression must be parenthesized"