Current Default:

// service root@b893aa9d262c:/CartoDB-SQL-API# cat /etc/cartodb/config.js
  "user_from_host": "^(.*)\\.cartodb\\.com$",

Seems it should be

module.exports.user_from_host = '^([^\\.]+)\\.';

@mnzaki so upon more investigation of trying to run the Analysis Tools - the following issue seems to be related to the configuration of the Batch API in mapsapi service. To reproduce try the following for the user:

[email protected] I'll message you the pass

Error is:

{"type":"analysis","message":"Unable to enqueue SQL API batch job","analysis":{"id":"a1","type":"moran"}}

Current analysis config block has:

"analysis": {
    "batch": {
      "endpoint": "http://127.0.0.1:8080/api/v2/sql/job",
      "hostHeaderTemplate": "{{=it.username}}.localhost.lan"
    },
  },

I think it should be

"analysis": {
    "batch": {
      "endpoint": "http://0.0.0.0:8080/api/v2/sql/job",
      "hostHeaderTemplate": "{{=it.username}}.localhost.lan"
    },
  },

I'm not sure exactly what hostHeaderTemplate should be but I suspect it doesn't matter yet since we are using the IP to reference the host for now. What do you think?

Also when you first load the map you will see the client services error too.

createlang utility has been removed from postgresql 10.x
use CREATE EXTENSION instead. see:

• https://www.postgresql.org/docs/9.6/sql-createlanguage.html
• https://www.postgresql.org/docs/9.6/app-createlang.html

in the analysis logs I am seeing:

{  
  "name":"camshaft",
  "hostname":"a11804d63940",
  "pid":1,
  "level":30,
  "msg":"analysis:limits_error",
  "time":"2019-02-19T16:52:08.563Z",
  "v":0
}

saw something similar in #3

• export organisation from old carto
• try to import into new carto

• add end point tests using bash with https://github.com/bats-core/bats-core

Hey, @sabman -- I pulled this down and built it, but I'm a bit confused about what are valid params to have in the .env file. I tried using the following:

ENVIRONMENT=development
DOMAINS=sabman.localhost
EMAIL=<MY EMAIL>
PUBLIC_HOST=sabman.localhost
PUBLIC_PROTOCOL=https
PUBLIC_PORT=443
DEFAULT_USER=nick
DEFAULT_PASSWORD=abc123def
POSTGRES_USER=postgres

When I build it with those values and bring the cluster up, the router container dies, with what I'm assuming is a failure message back from letsencrypt:

cartodb-router     | An unexpected error occurred:
cartodb-router     | The request message was malformed :: Error creating new authz :: Name does not end in a public suffix
cartodb-router exited with code 1

Do I need a fully qualified domain name in the .env file for it to work?

CartoDB uses Varnish to cache http requests, we need to add it to our services.

For architecture see:

• https://gis.stackexchange.com/questions/95243/performance-tuning-on-cartodb
• https://carto.com/developers/on-premises/guides/builder/architecture/

For example usage in a single server Dockerfile:

• https://github.com/sverhoeven/docker-cartodb/blob/master/Dockerfile#L117-L126
• https://github.com/sverhoeven/docker-cartodb/blob/master/Dockerfile#L227-L234

Possibly useful for configuring in microservices environment:

• https://github.com/webkul/magento2-varnish-docker-compose
• https://cloudkul.com/blog/magento-2-and-varnish-cache-integration-with-docker-compose/

We need a way to install SSL certificates via letsecrypt and renewal via certbot.
Maybe something along the lines of https://github.com/wmnnd/nginx-certbot

cartodb-editor outputs Password: permanently. This seems to be a problem with bundle exec rake db:create or bundle exec rake db:migrate.
Do you have to do something else than setting all the environment variables (including POSTGRES_PASSWORD)?

maybe something like: https://github.com/vegasbrianc/prometheus

After standing up the containers I ran docker-compose logs
.
.
.
[2018-11-20T12:03:21.373012 #85] DEBUG -- : Sequel::Postgres::Database (1140.0ms) CREATE DATABASE "cartodb_dev_user_62b6ab17-e862-44cb-8b81-a9e3ae30a19a_db"
cartodb-editor | WITH TEMPLATE = template_postgis
cartodb-editor | OWNER = postgres
cartodb-editor | ENCODING = 'UTF8'
cartodb-editor | CONNECTION LIMIT=-1
cartodb-editor | I, [2018-11-20T12:03:21.373135 #85] INFO -- : (1.140186s) CREATE DATABASE "cartodb_dev_user_62b6ab17-e862-44cb-8b81-a9e3ae30a19a_db"
cartodb-editor | WITH TEMPLATE = template_postgis
cartodb-editor | OWNER = postgres
cartodb-editor | ENCODING = 'UTF8'
cartodb-editor | CONNECTION LIMIT=-1
cartodb-editor | D, [2018-11-20T12:03:21.387069 #85] DEBUG -- : Sequel::Postgres::Database (0.5ms) SET standard_conforming_strings = ON
cartodb-editor | rake aborted!
cartodb-editor | PG::Error: ERROR: function postgis_scripts_installed() does not exist
cartodb-editor | LINE 1: SELECT postgis_scripts_installed()
cartodb-editor | ^
cartodb-editor | HINT: No function matches the given name and argument types. You might need to add explicit type casts.
cartodb-editor | QUERY: SELECT postgis_scripts_installed()
.
.
.
SELECT count() FROM pg_extension WHERE extname='postgis'
cartodb-editor |
cartodb-editor | I, [2018-11-20T12:03:21.429966 #85] INFO -- : (0.001023s)
cartodb-editor | SELECT count() FROM pg_extension WHERE extname='postgis'
cartodb-editor |
cartodb-editor | D, [2018-11-20T12:03:21.535599 #85] DEBUG -- : Sequel::Postgres::Database (105.4ms) CREATE EXTENSION postgis FROM unpackaged
cartodb-editor | E, [2018-11-20T12:03:21.535677 #85] ERROR -- : PG::Error: ERROR: function postgis_scripts_installed() does not exist
cartodb-editor | LINE 1: SELECT postgis_scripts_installed()
cartodb-editor | ^
cartodb-editor | HINT: No function matches the given name and argument types. You might need to add explicit type casts.
cartodb-editor | QUERY: SELECT postgis_scripts_installed()
cartodb-editor | CONTEXT: PL/pgSQL function postgis_major_version_check() line 24 at SQL statement: CREATE EXTENSION postgis FROM unpackaged
cartodb-editor | D, [2018-11-20T12:03:21.536434 #85] DEBUG -- : Sequel::Postgres::Database (0.6ms) ROLLBACK
cartodb-editor | I, [2018-11-20T12:03:21.536483 #85] INFO -- : (0.000674s) ROLLBACK
cartodb-editor | D, [2018-11-20T12:03:21.537007 #85] DEBUG -- : Sequel::Postgres::Database (0.3ms) ROLLBACK
.
.
.
I also checked the postgis container to see what extension are installed in the various databases
template_postgis
List of installed extensions
Name | Version | Schema | Description
---------+---------+------------+------------------------------
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language

carto_db_development
List of installed extensions
Name | Version | Schema | Description
-----------+---------+------------+---------------------------------------------------------------------
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
postgis | 2.5.0 | public | PostGIS geometry, geography, and raster spatial types and functions
uuid-ossp | 1.1 | public | generate universally unique identifiers (UUIDs)

cartodb_dev_user_eff62147-fd46-4571-baf5-7d32c9988267_db
List of installed extensions
Name | Version | Schema | Description
---------+---------+------------+------------------------------
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
(1 row)

Any pointer/s what could be the cause of this.

@mnzaki tried the updates with the new postigis_template seems to still be missing crankshaft and client services but could be that it's still using an older schema