>>> execfile('cartoframes/context.py')
>>> execfile('cartoframes/examples.py')
>>> base_url = EXAMPLE_BASE_URL
>>> api_key = EXAMPLE_API_KEY
>>> session = None
>>> creds = None
>>> creds = Credentials(creds=creds, key=api_key, base_url=base_url)
>>> auth_client = APIKeyAuthClient(base_url=creds.base_url(),api_key=creds.key(),session=session)
>>> sql_client = SQLClient(auth_client)
>>> res = sql_client.send("select current_schemas('f')", **DEFAULT_SQL_ARGS)
>>> res['rows'][0]['current_schemas']
u'{public,cartodb}'
>>> type(res['rows'][0]['current_schemas'])
<type 'unicode'>

Wouldn't it be nice if running a query returned the proper type? In this example, it should be a string array instead of a string. Is this feasible?

cc @danicarrion

Hello! I've been trying to do cc.read and cc.query using cartoframes. When I try and run it I get this error message:

<ipython-input-54-d98adbbbcfad> in <module>()
----> 1 assigned = cc.read('us_counties_assigned')
      2 unassigned = cc.read('us_counties_unassigned')

/Users/mspichiger/venv/lib/python2.7/site-packages/cartoframes/context.pyc in read(self, table_name, limit, index, decode_geom, shared_user)
    192                 raise ValueError("`limit` parameter must an integer >= 0")
    193 
--> 194         return self.query(query, decode_geom=decode_geom)
    195 
    196     def write(self, df, table_name, temp_dir=CACHE_DIR, overwrite=False,

/Users/mspichiger/venv/lib/python2.7/site-packages/cartoframes/context.pyc in query(self, query, table_name, decode_geom)
    728                 query,
    729                 skipfields='the_geom_webmercator',
--> 730                 **DEFAULT_SQL_ARGS)
    731             if 'error' in select_res:
    732                 raise CartoException(str(select_res['error']))

/Users/mspichiger/venv/lib/python2.7/site-packages/carto/sql.pyc in send(self, sql, parse_json, do_post, format, **request_args)
     84             return self.auth_client.get_response_data(resp, parse_json)
     85         except Exception as e:
---> 86             raise CartoException(e)
     87 
     88 

CartoException: Unterminated string starting at: line 1 column 1349219 (char 1349218)

The confusing part is that every time I run the command I get a new column value. In the example I gave the number is 1349219, but I also get 2981871 (char 2981870), 1266002 (char 1266001), and many more seemingly random columns. The dataset I'm trying to read is about 6.6 MB and has 18 columns. I've included the table here:

us_counties_assigned.csv.zip

Let me know if there's anything I can do to help.

There are some useful params in the SQL API, such as skipfields which allows for some columns to not be included in the API response (e.g., give me all columns except for the_geom_webmercator).

cc @alrocar

It would be nice to use a custom user-agent (instead of requests' default one), so we are able to better track usage/errors in our platform logs.

A method to create jobs would be needed. Right now the only job that the BatchSQLManager has is all, which permites to read the existing jobs in the CARTO account.

The current version of cartodb via pip

pip install cartodb

FileImport:

from cartodb import CartoDBAPIKey, CartoDBException, FileImport

yields this error:

ImportError: cannot import name FileImport

Installing dev version:

pip install -e git+git://github.com/CartoDB/cartodb-python.git#egg=cartodb

...works fine.

Hello,

I see there's a sync table dev branch. Curious if there are plans to merge it into master.

Thanks!

At this moment one can only check the status of an import on the Sync Manager, which is strange because it should be possible to check the status of any import, both from local and remote resources. I mean once a local resource has been uploaded it should return the control and with the import identifier to be able to check the status for the rest of the process happening at the CARTO instance.

The script when passing a --map should only return with a pretty format map details, a second --export command flag could actually do the download. That would be awesome

It'd be nice to return on the console a link to the dataset

If you try to import a dataset with a non-organization user, it imports the dataset correctly, but then it raises an error:

from carto.auth import APIKeyAuthClient
from carto.datasets import DatasetManager
auth_client = APIKeyAuthClient(api_key="APIKEY", base_url="https://NONORGUSER.carto.com/")
dataset_manager = DatasetManager(auth_client)
dataset = dataset_manager.create(DATAURL)

This is the error (I've omitted non-relevant information).

.../carto-python/carto/resources.py:90: FutureWarning: This is part of a non-public CARTO API and may change in the future. Take this into account if you are using this in a production environment
  warnings.warn('This is part of a non-public CARTO API and may change in the future. Take this into account if you are using this in a production environment', FutureWarning)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../carto-python/carto/datasets.py", line 213, in create
    table = TableManager(self.client).get(import_job.table_id)
...
  File ".../carto-python/carto/users.py", line 64, in __init__
    organization-enabled APIKeyAuthClient"))
carto.exceptions.CartoException: User management requires an                                    organization-enabled APIKeyAuthClient

The problem seems to be that, after the import, the table information is requested (with Table class) which contains a permission (PermissionFieldclass) which in turn has an owner ( Userclass), But the User class only support organization users.

I've managed to avoid the problem with this patch:

diff --git a/carto/users.py b/carto/users.py
index f23f12a..b1e89e3 100644
--- a/carto/users.py
+++ b/carto/users.py
@@ -26,6 +26,7 @@ from .resources import Manager, WarnResource

 API_VERSION = "v1"
 API_ENDPOINT = "api/{api_version}/organization/{organization}/users/"
+API_NONORG_ENDPOINT = "api/{api_version}/users/"


 class User(WarnResource):
@@ -60,15 +61,16 @@ class User(WarnResource):
         :param auth_client: Auth client
         """
         if auth_client.organization is None:
-            raise CartoException(_("User management requires an \
-                                   organization-enabled APIKeyAuthClient"))
+            self._api_endpoint = API_NONORG_ENDPOINT
+        else:
+            self._api_endpoint = API_ENDPOINT

         super(User, self).__init__(auth_client)

     def get_collection_endpoint(self):
         """
         """
-        return API_ENDPOINT.format(api_version=API_VERSION,
+        return self._api_endpoint.format(api_version=API_VERSION,
                                    organization=self.client.organization)

     def get_resource_endpoint(self):
@@ -100,7 +102,7 @@ class UserManager(Manager):
     def get_collection_endpoint(self):
         """
         """
-        return API_ENDPOINT.format(api_version=API_VERSION,
+        return self._api_endpoint.format(api_version=API_VERSION,
                                    organization=self.client.organization)

     def get_resource_endpoint(self, resource_id):

Hi,

I have a csv that I want to upload to my account. I am using the overwrite arg in cartoframes.

cc.write(pd.read_csv('table_ouput.csv'), 'table', overwrite=True, temp_dir="")

How can I do the equivalent using the carto-python library directly?

This code does not save the attributes

dataset.license = "whatever"
dataset.attributions = "whatever"

Hi!

We are having some problems with this library in OnPremise CARTO instances with invalid certificates and without the chance of use http, only https.

For fixing this, the APIKeyAuthClient class should have a verify (or something like that) attribute that refers to the verify parameter in requests. We'll need to propagate this change to BasicAuthClient, also for setting the using of the verify parameter in the send method.

We can do that couple of PRs, but we need to now why you convert every boolean to a string here, because the verify parameter needs to be a boolean.

Regards.

Synchronization table metadata is wrong. Running something like this:

dm = DatasetManager(auth_client)
dsets = dm.all()

tables = [{
    'name': table.name, 
    'privacy' : table.privacy,
    'created': table.created_at, 
    'synchronization': table.synchronization,
    'geometry': table.table.geometry_types
} for table in dsets]

tables_df = json_normalize(tables)
tables_df.head()

All tables presents None values:

But as you can see in my dashboard, there are sync tables:

Do you think you could add a license file to this package just to firm up any potential legal issues associated with its use?

Thanks!

Running into an issue when using cartoframes to read a table. Based on the traceback, I think it has to do with the carto python library SQL client and parsing a problematic json.

@andy-esch and I were able to get a dataframe back when we query with a limit (cc.query('select * from joined_full limit 1000). But once we try cc.query('select * from joined_full limit 100000), we get the error again. So this could indicate a problem with a few rows of the dataset or with some sort of query limit. Any ideas on what's happening here would be helpful!

If you need access to the joined_full dataset to help diagnose, just let me know.

Error:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
/anaconda3/lib/python3.6/site-packages/carto/sql.py in send(self, sql, parse_json, do_post, format, **request_args)
     83 
---> 84             return self.auth_client.get_response_data(resp, parse_json)
     85         except Exception as e:

/anaconda3/lib/python3.6/site-packages/pyrestcli/auth.py in get_response_data(self, response, parse_json)
     46             if parse_json:
---> 47                 return response.json()
     48             return response.content

/anaconda3/lib/python3.6/site-packages/requests/models.py in json(self, **kwargs)
    891                     pass
--> 892         return complexjson.loads(self.text, **kwargs)
    893 

/anaconda3/lib/python3.6/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    353             parse_constant is None and object_pairs_hook is None and not kw):
--> 354         return _default_decoder.decode(s)
    355     if cls is None:

/anaconda3/lib/python3.6/json/decoder.py in decode(self, s, _w)
    338         """
--> 339         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    340         end = _w(s, end).end()

/anaconda3/lib/python3.6/json/decoder.py in raw_decode(self, s, idx)
    354         try:
--> 355             obj, end = self.scan_once(s, idx)
    356         except StopIteration as err:

JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 50479105 (char 50479104)

During handling of the above exception, another exception occurred:

CartoException                            Traceback (most recent call last)
<ipython-input-25-6bdb9cec3095> in <module>()
----> 1 road_segments = cc.read('joined_full')

~/proj/cartoframes/cartoframes/context.py in read(self, table_name, limit, index, decode_geom, shared_user)
    200                 raise ValueError("`limit` parameter must an integer >= 0")
    201 
--> 202         return self.query(query, decode_geom=decode_geom)
    203 
    204     @utils.temp_ignore_warnings

~/proj/cartoframes/cartoframes/context.py in query(self, query, table_name, decode_geom)
    760                 query,
    761                 skipfields='the_geom_webmercator',
--> 762                 **DEFAULT_SQL_ARGS)
    763             if 'error' in select_res:
    764                 raise CartoException(str(select_res['error']))

/anaconda3/lib/python3.6/site-packages/carto/sql.py in send(self, sql, parse_json, do_post, format, **request_args)
     84             return self.auth_client.get_response_data(resp, parse_json)
     85         except Exception as e:
---> 86             raise CartoException(e)
     87 
     88 

CartoException: Expecting property name enclosed in double quotes: line 1 column 50479105 (char 50479104)

The script does not show a header on the output and all my tables appear as not cartodbfied.

This is more a issue of me not knowing what I'm doing. How do I add data to more than one column at the time? And how do I use variables in the sql statement?
I have the variables "date_object, location, user and text". I've tried this with no luck:

cl.sql("INSERT INTO tweets (date, location, user, text) VALUES (%s, %s, %s, %s)", (date_object, location, user, text))

This however works, but it only lets me update one column at the time and no use of variables.

cl.sql("INSERT INTO tweets (text) VALUES ('inserted text')")

I would really like to get this up and running. I'm trying to feed tweets from Twitters streaming api to make a live map.
I would be very thankful for any kind of help.
Thanks
Martin

For those cases when you're creating a file-like object in memory, to avoid having to write and read from a real file.

Deleting datasets is an important task, and this library should support it. The call is a 'DELETE' request to api/v1/viz.

Like Builder, a call to DatasetManager.delete should error/warn if a dataset is connected to a map. I prefer an Error in the Python case. Adding a force=False|True will let users force the operation regardless. As far as I know, there's metadata in the api/v1/viz GET that lists the dependents of a table.

Right now all exceptions are CartoException, which makes it hard to handle specific exceptions but let others be raised. E.g., If I want to print a special message about limits being hit, I would have to use the error string which could change without notice in future updates to the platform.

Context

The VisualizationManager lets us access many fields to get information about a CARTO account's maps, but the fields below don't return values even if values should exist—for example my sync tables return 'None' for the synchronizations field.

Can we update carto-python to return these values?
If it's easier to split these into separate issues please let me know

• active_child
• active_layer_id
• attributions
• children
• connector
• description
• display_name
• external_source
• license
• next_id
• prev_id
• source
• synchronization
• title
• transition_options
• user

Additional Information

Here's the script I used to test: carto_account_datasets_info_all.zip

These are all of the available fields:

• active_child
• active_layer_id
• attributions
• auth_tokens
• children
• client
• connector
• created_at
• delete
• description
• display_name
• external_source
• fields
• get_collection_endpoint
• get_id
• get_resource_endpoint
• id
• kind
• license
• liked
• likes
• locked
• map_id
• name
• next_id
• permission
• prev_id
• privacy
• refresh
• save
• send
• source
• stats
• synchronization
• table
• tags
• title
• transition_options
• type
• update_from_dict
• updated_at
• url
• user
• uses_builder_features

as with the rest of carto credential details, the user name should be taken as an option using an environment variable as a default like CARTO_USER

I would be great to support the Auth API

https://carto.com/developers/auth-api/reference/

This little gist runs:

class ApiKey(Resource):
    name = CharField()
    token = CharField()
    type = CharField()
    created_at = DateTimeField()
    updated_at = DateTimeField()
    grants = Field()

    class Meta:
        id_field = "name"
        collection_endpoint = "api/v3/api_keys/"


class ApiKeyManager(Manager):
    resource_class = ApiKey
    json_collection_attribute = "result"

keys_manager = ApiKeyManager(auth_client)
keys_manager.all()
keys_manager.get('Master')
grants =  [
    {
        "type": "apis",
        "apis": ["sql", "maps"]
    },
    {
        "type": "database",
        "tables" : []
    }
]
key = self.keys_manager.create(name="myname", grants=grants)

Would be possible a complete implementation?

When using the class VisualizationManager, the object table does not return the tables of the map but the client object.

I'm using the next code to get the table object values:

auth_client = APIKeyAuthClient(CARTO_BASE_URL, CARTO_API_KEY, organization)


visualization_manager = VisualizationManager(auth_client)

mapa = visualization_manager.get("mapName")
printer.pprint(mapa.__dict__)


printer.pprint(mapa.table.__dict__)

The result of python is:

{   'active_layer_id': None,
    'client': <carto.auth.APIKeyAuthClient object at 0x10f509950>,
    'created_at': u'2016-12-03T23:45:11+00:00',
    'description': None,
    'display_name': None,
    'id': u'87e6f95c-b9b2-11e6-8d20-0e3ebc282e83',
    'liked': False,
    'likes': 0,
    'locked': False,
    'map_id': u'7ed57ea1-2958-4577-bcfa-446572a1e7d2',
    'name': u'Stores',
    'table': <carto.tables.Table object at 0x10f517f50>,
    'title': None,
    'updated_at': u'2016-12-05T18:55:58+00:00',
    'url': u'https://cartoworkshops.carto.com/u/carto-workshops/viz/87e6f95c-b9b2-11e6-8d20-0e3ebc282e83/map'}
{   'client': <carto.auth.APIKeyAuthClient object at 0x10f509950>}

As far as I understand, baseurls in the v1.0.0 version of this module need to be structured like this:

https://eschbacher.carto.com/api/

Other pieces of carto's stack don't include the /api/ piece, like carto(db).js's support for on-prem, etc.: https://github.com/CartoDB/carto-workshop/blob/master/05-apis/exercises/maps_api_guide.md#custom-api-endpoints

To be consistent with these tools, it'd be nice to have a cleaner way of sending the baseurl to other carto services without having to do string operations.

cc @danicarrion

Hi there. Great work on this library.
I wanted to use privacy param in my code like following

fi = FileImport(import_file, cl, {"privacy":"public"})

but I could not make it work. I talked to Javier and he said this parm is not supported. Is this true? Is there anyway to include this param to the library?

Thank you for you nice work!

Hiroo

The script does not help a lot on reading the queries we are running at this moment. Please format the output 😓

I got a start on this but stopped short of making oauth work since I don't currently need it. Mostly making relative imports more explicit and dealing with modules being rearranged (eg parse_qsl is now in urllib.parse). The work's all in a81422e.

As of 0.6 you can specify an hostnamebut not a port. This makes it harder to test offline.

The output of the table_info script should be more compact, use tables to separate field names and indexes from their types and definitions respectively, etc.

Adding an .is_valid() method would be pretty useful for establishing if an application is good-to-go before starting to perform operations. I suppose this could be pulled off a bare-bones API request, but it would be useful to have a purpose-built method for doing this instead.

CartoDB/cartoframes#592 (comment)

First, some context :_)

I'm going to implement CartoDB/cartoframes/issues/170, which will provide easy access to example datasets and maps from CARTO Frames. Those examples will be loaded from a CARTO account with example maps and datasets.

CARTO Frames always checks if the provided user and API key are valid for authentication, and that's a problem for a test account. Unless we dropped the requirement for an API key (and that's something that we all want to avoid) we must use the soon-to-be-released Auth API default public API key. Although Auth API is not feature-complete yet, today we've released an improvement that allows validating any pair of user and API key: CartoDB/cartodb/pull/13564.

Currently, APIKeyAuthClient is based on api_key parameter, and, as the migration to Auth API will be gradual, that shouldn't change soon.

As I need a client to access CARTO based on Auth API header to check validation, I propose implementing a AuthAPIKeyClient that extends APIKeyAuthClient. Its responsability is the same than APIKeyAuthClient (providing an authenticated way to send requests to CARTO), but using the header instead of the parameter.

Then, I'd move the is_authenticated method from CARTO Frames to carto-python, because it seems useful.

I'd like to create a new class, AuthAPIKeyClient, because currently, send doesn't have a way to send headers. In order to avoid creating a new class, we should add a "use auth header flag" somewhere and infest the code with ifs, which doesn't seem too clean.

Please keep in mind that I've just begun programming Python, so all of this might be bullshit :-) I wanted to share my approach before beginning coding.

What do you think, @andy-esch @alrocar @danicarrion ?

PS: the implementation of is_authenticated will be based in the improved Auth API endpoint (CartoDB/cartodb/pull/13564), removing the query to information_schema.tables.

Inserting text with unicode characters throws a UnicodeEncodeError:

INSERT INTO foo (the_geom, standard_resolution_url, username, caption, created_time, link) VALUES (ST_GeomFromText('POINT(-106.388 39.64517)', 4326),'http://distilleryimage11.instagram.com/1dfcf9d863ae11e180c9123138016265_7.jpg','fernandezmaia', 'Buen día! Hoy empiezo en Golden Peak, parece que el destino hace que nieve cada vez que yo madrugo! Va a ser un laaaargo día', to_timestamp(1330613527), 'http://instagr.am/p/HokIZBrPc_/')
Traceback (most recent call last):
File "cartoImport.py", line 45, in
cl.sql(sql)
File "/usr/local/lib/python2.7/dist-packages/cartodb/cartodb.py", line 67, in sql
p = urllib.urlencode({'q': sql})
File "/usr/lib/python2.7/urllib.py", line 1312, in urlencode
v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 254: ordinal not in range(128)

Context

The VisualizationManager lets us access many fields to get information about a CARTO account's maps, but the fields below don't return any information.

Can we update carto-python to return these values?
If it's easier to split these into separate issues please let me know

• active_child
• active_layer_id
• attributions
• children
• display_name
• external_source
• kind
• license
• liked
• likes
• locked
• map_id
• name
• next_id
• parent_id
• prev_id
• permission
• privacy
• related_tables
• source
• stats
• synchronization
• tags
• title
• transition_options
• type
• uses_builder_features

Additional Information

Here's the script I used to test: carto_account_maps_info_all.zip

Here's the dataset generated by the test showing blank fields: maps_all_info2_save

These are all of the available fields:

• active_child
• active_layer_id
• attributions
• children
• client
• created_at
• delete
• description
• display_name
• export
• external_source
• fields
• get_collection_endpoint
• get_id
• get_resource_endpoint
• id
• kind
• license
• liked
• likes
• locked
• map_id
• name
• next_id
• parent_id
• permission
• prev_id
• privacy
• refresh
• related_tables
• save
• send
• source
• stats
• synchronization
• tags
• title
• transition_options
• type
• update_from_dict
• updated_at
• url
• uses_builder_features

Looks like the last change to 1.0.0. was not merged

[env] $ python sql_batch_api_jobs.py create --query 'select version()'
Traceback (most recent call last):
  File "sql_batch_api_jobs.py", line 55, in <module>
    createJob = batchSQLClient.create(args.query)
  File "/home/jsanz/src/sdks/carto-python/examples/env/src/carto/carto/sql.py", line 87, in create
    data = self.send(self.api_url, http_method="POST", json_body={"query": sql_query}, http_header=header)
  File "/home/jsanz/src/sdks/carto-python/examples/env/src/carto/carto/sql.py", line 76, in send
    data = self.client.send(url, http_method=http_method, http_headers=http_header, json=json_body)
  File "/home/jsanz/src/sdks/carto-python/examples/env/src/carto/carto/auth.py", line 42, in send
    return super(APIKeyAuthClient, self).send(relative_path, http_method, **requests_args)
  File "/home/jsanz/src/sdks/carto-python/examples/env/local/lib/python2.7/site-packages/pyrestcli/auth.py", line 34, in send
    return self.session.request(http_method, url, proxies=self.proxies, **requests_args)
TypeError: request() got an unexpected keyword argument 'http_headers'
[env] $ git pull
Already up-to-date.

I just spent a couple of hours discovering that this library only works with Python 2. I only tried it with 2.7 so I don't know which particular versions of 2 it works with.

The library installs without any problem on Python 3.5, and can be imported, but doesn't work.

It'd be great to have the following exceptions added so we can handle errors better on the application side:

• CartoAuthError
• CartoSQLError (or something like that) for syntax problems in query, 'relation does not exist', and others
• CartoQuotaError if over quota on LDS, DO, etc. which for now happen on SQL API but maybe this will change in the future?
• CartoDependentVisError if someone tries to delete a dataset that has dependent visualizations
• I'm sure there are others

To be able to handle these, we have to inspect the string, which is not a stable way to handle exceptions.

cc @alrocar

The carto-etl script prepares insert sql queries that when going through the SQLClient produce this error in the sql-api:

WARNING:carto-etl:Chunk #4: Retrying ([u'syntax error at or near "+"'])

Seems that the query is encoded in this way:

insert+into+INF_SIPS_CARTO+%28the_geom%2Cc...

Looks like copy pasted from another example and it's not clear what the query parameter is. It's the database pid?

h.

Since #90, cartoframes client param is overwritten. In cartoframes we set it here for use in all major cartoframes methods.

I don't know the best course of action for resolving this situation, but I'm happy to help because we are no longer able to track cartoframes usage.

cc @alrocar

Since CartoDB is not supporting OAuth anymore, references at documentation and source code should be removed.

Another type of message (or even no message at all) seems more appropriate.

The current behavior results in lots of pink warning messages in cartoframes whenever a users uses CartoContext.execute:

cc @alrocar