Giter VIP home page Giter VIP logo

parabible-server-2's Introduction

Parabible header image

Parabible Server

This is the Parabible backend. It is written in typescript for the Deno runtime.

It implements the Parabible API documented here (WIP).

This version of Parabible is currently still under development. It is mostly implemented, but there are a number of things remaining to be done (in no particular order):

  • Implement /health endpoint (checks access to db and returns 200 "OK")
  • Implement /module endpoint (lists modules—eventually, will supply additional details given a moduleId)
  • Implement /word endpoint (returns data on individual words [by moduleId])
  • Implement /text endpoint (returns bible text with reference(s) and module(s))
  • Implement /termSearch endpoint (return matching verses given search terms)
  • Implement /highlight endpoint (chapter highlights)
    • Still need to write tests for highlight
  • Figure out how to handle embedded notes (all kinds: text critical [SBLGNT], commentary [NET], general footnotes [e.g., ESV])
  • Consider a termSearchCount endpoint (for suggesting changes to search filters)
  • Finish documenting the API
  • Standardize error codes (include in the API)
  • Handle cases where incorrect parameters being supplied (refactor + break out require/allow code for parameters)
  • Render to html/json? (based on request headers)
  • Clean up typescript
  • Limit execution time of requests
    • There also seems to be a random spike in time taken for some CH queries. We should track that down and fix...
  • Get to 100% test coverage (it's not a big codebase)
  • Figure out how to serve extra resources (like dictionaries)
  • Figure out how to handle more complicated search concepts:
    • Semantic domain subqueries (allow search on any level of the semantic domain)
    • Participant awareness (i.e. understanding of antecedents of relative pronouns [data is available!])

There's also tons of other work on the data pipeline and the client:

  • Data Pipline (this is probably the easiest area to get involved in and have meaningful impact):
    • Enriching data to parity with prod
    • Doing an initial import of the apostolic fathers(!)
    • Splitting out into a meaningful structure on github
  • Client:
    • The old client has just been hacked onto the new backend so it's got lots of issues to deal with...
    • Consider supporting natural language queries...

Running on Docker

In order to run the server, you will need to have access to the Parabible database (TODO). The easiest way to run the entire stack is to use the docker-compose file supplied here (TODO).

To build and run the image:

$ docker build .
$ docker run -e PORT=3000 \
             -e CLICKHOUSE_USER=admin \
             -e CLICKHOUSE_PASSWORD=password \
             -e CLICKHOUSE_URL=http://localhost:8123 \
             image_hash

You will need to provide the container with the necessary environmental variables:

Variable Name Description
PORT The port to listen on (3000)
CLICKHOUSE_USER User to authenticate with on Clickhouse (admin)
CLICKHOUSE_PASSWORD Password to authenticate with on Clickhouse (toor)
CLICKHOUSE_URL URL and port where Clickhouse can be reached (http://localhost:8123)

Optional variables:

Variable Name Description
MAX_EXECUTION_TIME Maximum time to wait for a response from the API (5s)

parabible-server-2's People

Contributors

jcuenod avatar

Stargazers

avatar avatar avatar

Watchers

avatar avatar

parabible-server-2's Issues

Adjacency Search

With a sliding window function, we should be able to do "adjacency" searches in parabible. This will not force an order of wids (not a sequence search), but it is a useful first step. The relevant "WINDOW" function docs for clickhouse are pasted below. First, though, here's a mocked up query that does something useful:

select * from (
SELECT
    groupArray(rid) OVER w1 AS rids,
    groupArray(wid) OVER w1 AS wids,
    groupArray(text) OVER w1 AS words,
    groupArray(lexeme) OVER w1 AS lexemes,
    groupArrayIf(lexeme, lexeme='ἐκκλησία') OVER w1 AS w1,
    groupArrayIf(lexeme, lexeme='παροικέω') OVER w1 AS w2
FROM word_features
WINDOW
    w1 AS (PARTITION BY module_id ORDER BY wid ASC Rows BETWEEN 2 PRECEDING AND 2 FOLLOWING)
)
where length(w1) > 0 and length(w2) > 0
LIMIT 10

-- first_value and last_value respect the frame
SELECT
    groupArray(value) OVER w1 AS frame_values_1,
    first_value(value) OVER w1 AS first_value_1,
    last_value(value) OVER w1 AS last_value_1,
    groupArray(value) OVER w2 AS frame_values_2,
    first_value(value) OVER w2 AS first_value_2,
    last_value(value) OVER w2 AS last_value_2
FROM wf_frame
WINDOW
    w1 AS (PARTITION BY part_key ORDER BY order ASC),
    w2 AS (PARTITION BY part_key ORDER BY order ASC Rows BETWEEN 1 PRECEDING AND CURRENT ROW)
ORDER BY
    part_key ASC,
    value ASC;
┌─frame_values_1─┬─first_value_1─┬─last_value_1─┬─frame_values_2─┬─first_value_2─┬─last_value_2─┐
│ [1]            │             1 │            1 │ [1]            │             1 │            1 │
│ [1,2]          │             1 │            2 │ [1,2]          │             1 │            2 │
│ [1,2,3]        │             1 │            3 │ [2,3]          │             2 │            3 │
│ [1,2,3,4]      │             1 │            4 │ [3,4]          │             3 │            4 │
│ [1,2,3,4,5]    │             1 │            5 │ [4,5]          │             4 │            5 │
└────────────────┴───────────────┴──────────────┴────────────────┴───────────────┴──────────────┘

-- second value within the frame
SELECT
    groupArray(value) OVER w1 AS frame_values_1,
    nth_value(value, 2) OVER w1 AS second_value
FROM wf_frame
WINDOW w1 AS (PARTITION BY part_key ORDER BY order ASC Rows BETWEEN 3 PRECEDING AND CURRENT ROW)
ORDER BY
    part_key ASC,
    value ASC
┌─frame_values_1─┬─second_value─┐
│ [1]            │            0 │
│ [1,2]          │            2 │
│ [1,2,3]        │            2 │
│ [1,2,3,4]      │            2 │
│ [2,3,4,5]      │            3 │
└────────────────┴──────────────┘

-- second value within the frame + Null for missing values
SELECT
    groupArray(value) OVER w1 AS frame_values_1,
    nth_value(toNullable(value), 2) OVER w1 AS second_value
FROM wf_frame
WINDOW w1 AS (PARTITION BY part_key ORDER BY order ASC Rows BETWEEN 3 PRECEDING AND CURRENT ROW)
ORDER BY
    part_key ASC,
    value ASC
┌─frame_values_1─┬─second_value─┐
│ [1]            │         ᴺᵁᴸᴸ │
│ [1,2]          │            2 │
│ [1,2,3]        │            2 │
│ [1,2,3,4]      │            2 │
│ [2,3,4,5]      │            3 │
└────────────────┴──────────────┘
```sql

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.