/static/card_group_main.html returns a 404 error

In QueryMatcher.__init__(), there is still a reference to card_info.sane_title(). This needs to be updated:

    if 'kingdom' in args:
        self.kingdom_restrict = [card_info.sane_title(k)
                                 for k in args['kingdom'].split(',')]

Maybe more generally, search for other references to card_info that are left over from commit 7d4c46a.

When searching for more than one card in a kingdom, inclusion of a space after the comma separating two card names results in an "internal server error". Ideally, the application should ignore such spaces in this situation.

With no space, Highway,Saboteur, we get a successful search: http://councilroom.com/search_result?p1_name=Qvist&p2_name=&kingdom=Highway%2CSaboteur

With a space between the card names, Highway, Saboteur, we get an internal server error: http://councilroom.com/search_result?p1_name=Qvist&p2_name=&kingdom=Highway%2C+Saboteur

These two URLs should return the same results.

This was reported by Qvist on http://forum.dominionstrategy.com/index.php?topic=6305.0

I notice that https://www.councilroom.com doesn't work, only http://www.councilroom.com.

% nmap www.councilroom.com 
Starting Nmap 7.80 ( https://nmap.org ) at 2023-04-10 21:57 PDT
Nmap scan report for www.councilroom.com (107.23.137.191)
Host is up (0.079s latency).
rDNS record for 107.23.137.191: ec2-107-23-137-191.compute-1.amazonaws.com
Not shown: 998 filtered ports
PORT   STATE SERVICE
22/tcp open  ssh
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 6.72 seconds

I don't suppose you'd be open to enabling HTTPS for the sake of security and privacy? (E.g., ISPs are known to inject ads and tracking into all unencrypted connections.)

I'd be happy to help either in terms of implementation or just advice.

Looking at the README it seems like the Python app is probably just running on a raw VPS somewhere. If that's the case, it's easy to set up HTTPS without even making application changes: just install Caddy and use it as a reverse proxy with this /etc/caddy/Caddyfile:

{
    auto_https disable_redirects
}

https://www.councilroom.com:443 {
    reverse_proxy localhost:80
}

The proxy takes care of solving HTTP-01 challenges and managing the appropriate certs, so you're basically good to go.

If the architecture is something different, there are of course other options that might be better suited. If you already have an nginx or Apache setup, you can set up Certbot to handle the provisioning and renewals, then it's just a matter of updating your proxy config to pull in the cert files from disk. Or if this is on EC2 and you'd prefer to use an Amazon solution, ALBs and classic ELBs will terminate TLS for you via ACM, which can be renewed automatically if your records are in Route 53 as they seem like they might be:

% dog councilroom.com 
SOA councilroom.com. 6m20s A "ns-685.awsdns-21.net." "awsdns-hostmaster.amazon.com." 1 2h00m00s 15m00s 14d0h00m00s 1d0h00m00s

The main disadvantage of that is just that you end up paying around $20/month on load balancer infrastructure for questionable benefit. Anyway, up to you, just thought I would raise the question since I didn't see other discussions about it here or on the mailing list.

Traceback from gunicorn log:

Traceback (most recent call last):
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 420, in _delegate
    return handle_class(cls)
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 396, in handle_class
    return tocall(*args)
  File "/srv/councilroom/councilroom_prod/frontend.py", line 607, in GET
    target_inds = map(self.str_card_index, targets)
  File "/srv/councilroom/councilroom_prod/frontend.py", line 544, in str_card_index
    return str(card.index)
NameError: global name 'card' is not defined

Under way

Observed when visiting /goals

No matter what I put in the fields, I always get the following result:

No data available in table
Showing 0 to 0 of 0 entries

Underway

The update.py script was killed when the system ran out of memory on 2012-12-19:

Dec 19 03:29:30 ip-10-0-0-236 kernel: [11571472.649735] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Dec 19 03:29:30 ip-10-0-0-236 kernel: [11571472.649990] [25456]  1001 25456   272242   216302   0       0             0 python
Dec 19 03:29:30 ip-10-0-0-236 kernel: [11571472.649994] [25501]     0 25501   106609    66276   0       0             0 gunicorn
Dec 19 03:29:30 ip-10-0-0-236 kernel: [11571472.649997] [25537]     0 25537   164450     8463   0       0             0 gunicorn
Dec 19 03:29:30 ip-10-0-0-236 kernel: [11571472.650001] [25547]     0 25547    82500    65622   0       0             0 gunicorn
Dec 19 03:29:30 ip-10-0-0-236 kernel: [11571472.650005] Out of memory: Kill process 25456 (python) score 334 or sacrifice child
Dec 19 03:29:30 ip-10-0-0-236 kernel: [11571472.650018] Killed process 25456 (python) total-vm:1088968kB, anon-rss:865208kB, file-rss:0kB

It died during the optimal_card_ratios step.

Should be able to clone the repository to a generic Ubuntu/Debian instance, run this fabfile, and have a fully configured and functioning instance of the application.

Traceback (most recent call last):
File "/srv/councilroom/councilroom_working/.venv/local/lib/python2.7/site-packages/web/application.py", line 239, in process
return self.handle()
File "/srv/councilroom/councilroom_working/.venv/local/lib/python2.7/site-packages/web/application.py", line 230, in handle
return self._delegate(fn, self.fvars, args)
File "/srv/councilroom/councilroom_working/.venv/local/lib/python2.7/site-packages/web/application.py", line 420, in _delegate
return handle_class(cls)
File "/srv/councilroom/councilroom_working/.venv/local/lib/python2.7/site-packages/web/application.py", line 396, in handle_class
return tocall(*args)
File "/srv/councilroom/councilroom_working/frontend.py", line 340, in GET
ret += (query_matcher.GameMatcher(g, qm).display_game_snippet() +
File "/srv/councilroom/councilroom_working/query_matcher.py", line 133, in display_game_snippet
self._display_supply())
File "/srv/councilroom/councilroom_working/query_matcher.py", line 126, in _display_supply
_render_card(card, freq, self.query_matcher.kingdom_restrict))
File "/srv/councilroom/councilroom_working/query_matcher.py", line 20, in _render_card
plural = dominioncards.pluralize(freq)
AttributeError: 'module' object has no attribute 'pluralize'

From Rob's post in councilroom-dev:

I suspect because a few of the pages are coded very poorly and traverse a whole users set of games to compute stats (popular buys per player, the player page, and the player vs other player record page all do this). When a a similarly poorly coded search engine comes around (these pages are blocked by robots.txt, but some search engines don't honor it), it just keeps putting too many requests on those uber slow pages and councilroom eventually runs out of memory and dies. The decision to do a scan of all the players games came when players had a few hundred game max, but now that there are crazy people with 10k games, and so the design is horribly broken. If someone pre-computed those pages, I am pretty sure a 486 could serve councilroom fine.

Not sure what is required to be precalculated in advance, but it is distinctly slow for users with large game counts.

From Rob's post in councilroom-dev:

I suspect because a few of the pages are coded very poorly and traverse a whole users set of games to compute stats (popular buys per player, the player page, and the player vs other player record page all do this). When a a similarly poorly coded search engine comes around (these pages are blocked by robots.txt, but some search engines don't honor it), it just keeps putting too many requests on those uber slow pages and councilroom eventually runs out of memory and dies. The decision to do a scan of all the players games came when players had a few hundred game max, but now that there are crazy people with 10k games, and so the design is horribly broken. If someone pre-computed those pages, I am pretty sure a 486 could serve councilroom fine.

Not sure what is required to be precalculated in advance, but it is distinctly slow for users with large game counts.

This probably will involve having all Python logging go to syslog, and having syslog on the workers forward to syslog on the master site.

When the update.py script is run from within IPython, each additional run results in an additional set of logging handlers being added, which results in duplication of each message.

The site returns an internal server error when searching for, say, rrenaud on the /popular_buys page.

This is the traceback from the gunicorn log:

Traceback (most recent call last):
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 420, in _delegate
    return handle_class(cls)
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 396, in handle_class
    return tocall(*args)
  File "/srv/councilroom/councilroom_prod/frontend.py", line 75, in GET
    return render.buy_template(stats, player_buy_summary)
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/template.py", line 881, in __call__
    return BaseTemplate.__call__(self, *a, **kw)
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/template.py", line 808, in __call__
    return self.t(*a, **kw)
  File "buy_template.html", line 130, in __template__
    </tbody>
  File "/srv/councilroom/councilroom_prod/stats.py", line 112, in render_std_devs
    return u'%.2f' % (self.mean() / self.sample_std_dev())
  File "/srv/councilroom/councilroom_prod/stats.py", line 115, in mean
    return self.mvs1.mean() - self.mvs2.mean()
  File "/srv/councilroom/councilroom_prod/stats.py", line 115, in mean
    return self.mvs1.mean() - self.mvs2.mean()
  File "/srv/councilroom/councilroom_prod/stats.py", line 40, in mean
    return self.sum / self.freq
ZeroDivisionError: integer division or modulo by zero

Recent commit have introduced a new error that occurs on visiting the game logs (http://councilroom.com/game?game_id=game-20130119-160107-0c7d61c4.html):

Error annotating game, tell [email protected]!
Exception:'int' object is not callable
  File "/srv/councilroom/councilroom_prod/frontend.py", line 447, in GET
    return annotate_game.annotate_game(contents, game_id, debug)
  File "/srv/councilroom/councilroom_prod/annotate_game.py", line 56, in annotate_game
    encoded = game_state.encode_game_state()
  File "/srv/councilroom/councilroom_prod/game.py", line 516, in encode_game_state
    'money': self.cur_turn.money() if self.cur_turn else 0,

From Rob's post in councilroom-dev:

I suspect because a few of the pages are coded very poorly and traverse a whole users set of games to compute stats (popular buys per player, the player page, and the player vs other player record page all do this). When a a similarly poorly coded search engine comes around (these pages are blocked by robots.txt, but some search engines don't honor it), it just keeps putting too many requests on those uber slow pages and councilroom eventually runs out of memory and dies. The decision to do a scan of all the players games came when players had a few hundred game max, but now that there are crazy people with 10k games, and so the design is horribly broken. If someone pre-computed those pages, I am pretty sure a 486 could serve councilroom fine.

Not sure what is required to be precalculated in advance, but it is distinctly slow for users with large game counts.

Usage grows pretty large. See if there is a reason it has to be that way.

Investigate why the winner in this game was awarded a PurplePileDriver goal, while the winner in this game was not.

But only if it is unclaimed. And while we are at it, they should also register their instance name appropriately.

This appears on visiting the page:

No stats for "Minion" and "Gold".

Urls like http://example.com/game?game_id=game-20120521-020825-2658beb6.html are returning error messages like:

could not find game game-20120521-020825-2658beb6.html

The following URL should contain a graph, but does not:

http://councilroom.com/win_rate_diff_accum.html?cards=Cost%3D%3D5%26%26Actions%3D%3D1

The following URL should contain a graph, but does not:

http://councilroom.com/win_weighted_accum_turn.html?cards=Silver,Cost%3D%3D3%20%26%26%20Actions%3E%3D1%20%26%26%20Cards%3E%3D1

gunicorn log:

Traceback (most recent call last):
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 420, in _delegate
    return handle_class(cls)
  File "/srv/councilroom/councilroom_prod/.venv-prod/local/lib/python2.7/site-packages/web/application.py", line 396, in handle_class
    return tocall(*args)
  File "/srv/councilroom/councilroom_prod/frontend.py", line 127, in GET
    opening['cards'].sort(key=dominioncards.cost, reverse=True)
AttributeError: 'module' object has no attribute 'cost'

The Score and Money Graphs do not appear on pages like this:
http://councilroom.mccllstr.com/game?game_id=game-20120722-142757-f1048654.html

The scraper script should now:

1. Retrieve the day's archive from isotropic
2. Store a verbatim copy in the S3 bucket
3. Create a Celery task instance to repackage it

The Celery task should:

1. Retrieve the file from S3
2. Decompress it
3. Recompress each game, individually
4. Insert the compressed game into MongoDB
5. Create a Celery task instance to parse and insert the game

Some of the error pages still say to email/blame Rob when things go wrong.

The following images are missing:

• DoublePileDriver.png
• TriplePileDriver.png
• Vintner.png

Have this object stored in MongoDB instead, so we can run the analyze.py script on a worker.

The new indexes module centralizes the setup of MongoDB indexes and creates a centralized script that can be run to get the DB into the appropriate configuration.

From Rob's post in councilroom-dev:

I suspect because a few of the pages are coded very poorly and traverse a whole users set of games to compute stats (popular buys per player, the player page, and the player vs other player record page all do this). When a a similarly poorly coded search engine comes around (these pages are blocked by robots.txt, but some search engines don't honor it), it just keeps putting too many requests on those uber slow pages and councilroom eventually runs out of memory and dies. The decision to do a scan of all the players games came when players had a few hundred game max, but now that there are crazy people with 10k games, and so the design is horribly broken. If someone pre-computed those pages, I am pretty sure a 486 could serve councilroom fine.