Giter VIP home page Giter VIP logo

rcatmaid's Introduction

Travis-CI Build Status Release Version DOI natverse Docs R-CMD-check

catmaid

This package provides access to the CATMAID API for R users. At present it provides low level functions for appropriately authenticated GET/POST requests, optionally parsing JSON responses. There are also intermediate level functions that retrieve skeleton (i.e. neuron) information, connectivity information for one or more neurons as well as a number of other API endpoints. Finally, there is a high level function to convert neurons to the representation of the nat (NeuroAnatomy Toolbox) R package, enabling a wide variety of analyses.

It is strongly recommended to read through the instructions below, the package overview documentation and then skim the reference documentation index , which groups the available functions into useful categories.

Quick start

# install
if (!require("remotes")) install.packages("remotes")
# nb repo is rcatmaid, but R package name is catmaid
remotes::install_github("natverse/rcatmaid")

# use 
library(catmaid)

# general help starting point
?catmaid

Example

This example is based on EM tracing data from the Drosophila first instar larva by Albert Cardona and colleagues as submitted to Virtual Fly Brain and visible at https://catmaid.virtualflybrain.org.

This produces a 3D plot of first and second order olfactory neurons coloured according to the peripheral odorant receptor.

library(catmaid)
# Specify VFB CAMAID server containing data
conn=catmaid_login(server="https://l1em.catmaid.virtualflybrain.org")

# fetch olfactory receptor neurons
orns=read.neurons.catmaid("name:ORN (left|right)", .progress='text')
# calculate some useful metadata
orns[,'Or']= factor(sub(" ORN.*", "", orns[,'name']))
orns[,'side']= factor(sub(".* ORN ", "", orns[,'name']))

# repeat for their PN partners, note use of search by annotation
pns=read.neurons.catmaid("ORN PNs", .progress='text')
pns[,'Or']= factor(sub(" PN.*", "", pns[,'name']))
pns[,'side']= factor(sub(".*(left|right)", "\\1", pns[,'name']))
# plot, colouring by odorant receptor
plot3d(orns, col=Or)
# note that we plot somata with a radius of 1500 nm
plot3d(pns, col=Or, soma=1500)

Which produces the following output plot:

Larval PN

Authentication

You will obviously need to have the login details of a valid CATMAID instance to try this out. As of December 2015 CATMAID is moving to token based authentication. For this you will need to get an API token when you are logged into the CATMAID web client in your browser. See http://catmaid.github.io/dev/api.html#api-token for details.

Once you have the login information you can use the catmaid_login function to authenticate. The minimal information is your server URL and your CATMAID token.

catmaid_login(server="https://mycatmaidserver.org/catmaidroot",
              authname="Calvin",authpassword="hobbes",
              token="9944b09199c62bcf9418ad846dd0e4bbdfc6ee4b")

Note that the CATMAID servers that I am aware of require two layers of password protection, an outer HTTP auth type user/password combination as well as an inner CATMAID-specific token based login. The outer HTTP auth type user/password combination may be specific to you or generic to the project.

Setting environment variables in your .Renviron file

It is recommended that you set your login details by including code like this in in your .Renviron file:

catmaid_server="https://mycatmaidserver.org/catmaidroot"
catmaid_token="9944b09199c62bcf9418ad846dd0e4bbdfc6ee4b"

# additional security for mycatmaidserver.org/catmaidroot page
catmaid_authname="Calvin"
catmaid_authpassword="hobbes"

In this way authentication will happen transparently as required by all functions that interact with the specified CATMAID server.

Be sure to leave one blank line at the end of the .Renviron file, or it will not work. Note that catmaid_server rather than catmaid.server is now the preferred form of specifying environment variables (some shells do not like variables with periods in their name). Note also that the use of package options in your .Rprofile file is still possible but now deprecated.

Cached authentication

Whether you use options in your .Renviron as described above or you login explicitly at the start of a session using catmaid_login the access credentials will be cached for the rest of the session. You can still authenticate explicitly to a different CATMAID server (using catmaid_login) if you wish.

Multiple servers

If you use more than one CATMAID server but always do so in different sessions or rmarkdown scripts then you can save an appropriate .Renviron file in the project folder.

If you need to talk to more than one CATMAID server in a single session then you must use catmaid_login to login into each server

# log in to default server specified in .Renviron/.Rprofile
conn1=catmaid_login()
# log into another server, presumably with different credentials
conn2=catmaid_login(server='https://my.otherserver.com', ...)

and then use the returned connection objects with any calls you make e.g.

# fetch neuron from server 1
n1=read.neuron(123, conn=conn1)
# fetch neuron from server 2
n2=read.neuron(123, conn=conn2)

n.b. you must use connection objects to talk to both servers because if no connection object is specified, the last connection will be re-used.

Installation

Currently there isn't a released version on CRAN but can use the devtools package to install the development version:

if (!require("remotes")) install.packages("remotes")
remotes::install_github("natverse/rcatmaid")

Note: Windows users need Rtools and devtools to install this way.

Acknowledgements

Originally based on python code presently visible at:

by Albert Cardona and Philipp Schlegel. Released under the GPL-3 license.

rcatmaid's People

Contributors

alexanderbates avatar jefferis avatar mmc46 avatar sridharjagannathan avatar zhihaozheng avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

rcatmaid's Issues

fix treenode table

Changed at some point to

(?P<project_id>\d+)/treenode/table/(?P<skid>\d+)/content

connectors.neuronlist should handle neurons without connectors

for example in the following, some neurons do not have connectors and this results in an error

> cc=connectors(fulln)
 Hide Traceback

 Rerun with Debug
 Error: All inputs to rbind.fill must be data.frames 
4 stop("All inputs to rbind.fill must be data.frames", call. = FALSE) 
3 rbind.fill(dfs) at catmaid_nat.R#194
2 connectors.neuronlist(fulln) at catmaid_nat.R#173
1 connectors(fulln) 

catmaid_query_connected fails

Looks like the response from catmaid includes additional elements since I wrote this function.

> catmaid_query_connected(skid=2568465)
 Show Traceback

 Rerun with Debug
 Error in df[df$syn.count >= minimum_synapses, ] : 
  incorrect number of dimensions In addition: Warning messages:
1: In data.frame(union_reviewed = c(NA, NA, NA, NA, NA, NA, NA, NA,  :
  row names were found from a short variable and have been discarded
2: In data.frame(union_reviewed = c(NA, NA, NA, NA, NA, NA, NA, NA,  :
  row names were found from a short variable and have been discarded

bug in catmaid_connection_setenv()

Error in names(export_vector) = paste0("catmaid.", vars_to_export) : 
  'names' attribute [7] must be the same length as the vector [5]

when some values are NULL

XXX header in 1st column of connector data.frame

First column of the connector data.frame for neuron objects downloaded from CATMAID has variable name 'XXX' (unknown identifier). Also in 'connectors' documentation. I believe the number is 'treenode id'. If it is an outgoing connectors, the number is pre-synaptic treenode id; for incoming connectors, it is post-synaptic treenode id.

logical and in annotation queries

something like this:

pid=1
aids=c(9646158, 11442737)

post_data=list(pid=pid)
post_data[sprintf("neuron_query_by_annotation[%d]", seq_along(aids))]=as.list(aids)
catmaid_fetch("1/neuron/query-by-annotations", body=post_data)

add catmaidneuron class

give these neurons an extra class (on top of neuron) so that we can e.g. plot connectors as well

Bug in catmaid_query_by_neuronname for large n

>  zz=catmaid_query_by_neuronname(".+", maxresults = 1e5)
Error in attr(res2, "annotations") = lapply(res$entities, "[[", "annotations") : 
  attempt to set an attribute on NULL

Probably a rare failure rather than something special about processing many records.

logical and in annotation queries

something like this:

pid=1
aids=c(9646158, 11442737)

post_data=list(pid=pid)
post_data[sprintf("neuron_query_by_annotation[%d]", seq_along(aids))]=as.list(aids)
catmaid_fetch("1/neuron/query-by-annotations", body=post_data)

cache parsed neuron data

not sure of a good strategy yet for this? One simple thing would be to hash the returned json and at least save ourselves the trouble of re-parsing.

as a little test, something like this:

read.neurons.catmaid(<42pnids>)

breaks down to about

  • 20% GET
  • 20% parse json to R list of lists
  • 20% list2df (i.e. parse json list structures to data.frames)
  • 40% parse data.frame to neuron

So it looks like this strategy could give a 3-5x speedup, which sounds interesting. But then the question is where would do this. If we insert something in catmaid_fetch we could make something very general and save the json parsing. But if we worked with read.neuron.catmaid, we should be able to save everything.

Another strategy would be to cache the request itself – this could involve catmaid_fetch again and a hash of the url/post data along with some kind of timestamp checking.

catmaid_query_connected: skid error and warning

In this example there is an inappropriate warning and the skids look wrong.

> catmaid_query_connected(7527710)
$outgoing
    skid name syn.count union_reviewed num_nodes
20    20   NA        70             NA      2977
15    15   NA        43             NA      3798
80    80   NA        36             NA      5184
125  125   NA        36             NA      3769
45    45   NA        34             NA      2577
120  120   NA        34             NA      5173
60    60   NA        20             NA      1813
105  105   NA        11             NA      4397
10    10   NA         9             NA      8319
50    50   NA         9             NA      3984
35    35   NA         8             NA      4057
40    40   NA         8             NA      7340
110  110   NA         8             NA      4228
95    95   NA         7             NA      5674
100  100   NA         6             NA      5829
5      5   NA         5             NA      3731
25    25   NA         3             NA      2088
75    75   NA         3             NA      5080
140  140   NA         3             NA      2409
55    55   NA         2             NA      1332
65    65   NA         2             NA      3629
85    85   NA         2             NA      6223
115  115   NA         2             NA      6293
30    30   NA         1             NA      4058
70    70   NA         1             NA      6210
90    90   NA         1             NA      2327
130  130   NA         1             NA      7408
135  135   NA         1             NA      2824

$incoming
   skid name syn.count union_reviewed num_nodes
35   35   NA        12             NA      8319
20   20   NA        11             NA       732
30   30   NA         9             NA      4058
5     5   NA         7             NA      4397
25   25   NA         6             NA      5173
45   45   NA         5             NA      7340
15   15   NA         4             NA      5829
10   10   NA         1             NA      5184
40   40   NA         1             NA      6293

Warning messages:
1: In data.frame(union_reviewed = c(NA, NA, NA, NA, NA, NA, NA, NA,  :
  row names were found from a short variable and have been discarded
2: In data.frame(union_reviewed = c(NA, NA, NA, NA, NA, NA, NA, NA,  :
  row names were found from a short variable and have been discarded

npop3d fails to remove connectors

this is a bug in plot3d.catmaidneuron which should return the rgl ids of the connectors (just as nat::plot3d.neuron does for the soma).

404 Error: catmaid_get_connected

When I try to use catmaid_get_connected, I get the error:

Error in catmaid_fetch(path, connectivity_post, include_headers = F, ...) :
client error: (404) Not Found

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.