Directlink: Going from relational databases to databases with relations with Neo4j and Spring Data

@startuml
' see https://gist.github.com/QuantumGhost/0955a45383a0b6c0bc24f9654b3cb561

!define Table(name,desc) class name as "desc" << (T,#FFAAAA) >>
!define primary_key(x) <b>x</b>
!define unique(x) <color:green>x</color>
!define not_null(x) <u>x</u>

hide methods
hide stereotypes

Table(artists, "artists\n(Artists that have been played)") {
primary_key(id) INTEGER
not_null(unique(name)) VARCHAR[255]
}

Table(genres, "genres\n(Genres that have been played)") {
primary_key(id) INTEGER
not_null(unique(name)) VARCHAR[255]
}

Table(tracks, "tracks\n(Partially normalized track data)") {
primary_key(id) INTEGER
not_null(artist_id) INTEGER
not_null(track_id) INTEGER
name VARCHAR[4000]
}

Table(plays, "plays\n(The actualy play data)") {
primary_key(id) INTEGER
not_null(track_id) INTEGER
played_on DATETIME
}

' relationships
' one-to-one relationship
artists "1"-->"*" tracks : "A track has been released by an artist"
' one to may relationship
genres "1"-->"*" tracks : "A track has a genre"
' many to many relationship
' Add mark if you like
tracks "1" --> "*" plays : "A tracks has been played some times"
@enduml

From our getting started guide:

Neo4j is an open-source NoSQL native graph database which provides an ACID-compliant transactional backend for your applications. With development starting in 2003, it has been publicly available since 2007.

While a relational database management system (RDBMs) stores relations between tuples (hence the tables itself in a RDBMs are called relations), a graph database is a database designed to treat the relationships between data as a first-class citizen in the data model.

The Spring Framework has been been around since 2002 and is one of the oldest Enterprise Java Frameworks actively used, maintained and developed. Spring Boot is ten years younger. It dates back to a ticket "Improve support for containerless application" which already describes important goals for the first release in April 2014.

• Fast start for all kind of development wit hSpring

• No generation of code or configuration

• Easily configurable from the outside

• Consistent component model

• Lots of non-functional feature out of the box, like metrics, health checks and so on

• Improve the developer experience with other Spring related projects

The deployment scenario most often used with Spring Boot applications are self-contained Jars, including batteries. That is: Having all needed libraries with them, including a servlet container or similar when the application is a web-application of some kind.

Let’s quote the Spring Data site:

Spring Data’s mission is to provide a familiar and consistent, Spring-based programming model for data access while still retaining the special traits of the underlying data store.

It makes it easy to use data access technologies, relational and non-relational databases, map-reduce frameworks, and cloud-based data services.

Spring Data itself is an umbrella project with support for several, quite different datastores, reaching from classic RDBMs-systems over document- and key-value-stores to cloud based services. Non-relational datastores certainly include Neo4j.

At the core of Spring Data lives the repository. There are several sources for the repository pattern. One is from Martin Fowlers Patterns of Enterprise Application Architecture.

Mediates between the domain and data mapping layers using a collection-like interface for accessing domain objects.

Further down the road we’ll see why the distinction between domain and data mapping layer is important: When discussing the relationship between Spring Data Neo4j (SDN) and Object Graph Mapping (OGM).

You’ll find the repository pattern also prominent in Domain Driven Design (DDD). That reference is very nicely explained by our own Mark Needham here DDD: Repository pattern.

Regardless whether you’re using a relational database or a graph database, you can access your aggregate roots in a consistent way. However, you still have to think yourself how build and create those aggregate roots.

Spring Data repositories and the entities defined therein also support events, auditing and more. Some people fancy the dynamic query derivation from repository method names a lot.

For nearly every store, Spring Data also provides more low level access patterns, often in the form of a XXXTemplate or XXXOperations. We will also dive into that.

Spring Data relies on Springs Dependency Injection mechanism and brings in some dependencies. It can be used without Spring Boot, but Spring Boot does a lot of useful autoconfiguration.

Neo4j OGM stands for Object Graph Mapping and is used to mapped nodes, their properties and relationships return from a graph to Java objects. While it is much easier to map Nodes and their relationships from a graph database to a network of Java objects than mapping rows returned from a relational database to objects (See Object-relational impedance mismatch), there are still edge cases:

• Neo4j can be used without a scheme. How to map basically arbitrary nodes to Objects?

• Cypher and Neo4j provide great means to do all kinds of projections. How to map does?

• And most important: How to deal with possible endless paths between nodes?

We’ll address all of those points.

• statsdb: Plain java module that contains a Java DSL generated by jOOQ for the relational schema described here.

• etl: Some stored procedures for Neo4j that implement an "extract, transform and load" mechanism, connecting PostgreSQL and Neo4j

• charts: A revised version of bootiful-databases. For your reference, an english and a german talk on that.

• knowledge: Finally, the Spring Boot and SDN based project that uses Neo4j to explore the relationship between artists, their tracks and albums.

• Java 11+

• Maven is bundled with our repositories

• Docker (Community edition) or a version that is bundled with your OS.

• Java-IDE of your choice

To run the modules of this project, you have to have PostgreSQL database with some defined schemas up and running. There’s a Docker module and a Docker Compose file to help you with that.

Please run (cd docker && docker-compose -f docker-compose.yml -f docker-compose.default.yml up) from the root of this project. To stop the processes, use (cd docker && docker-compose stop). This brings up both a PostgreSQL instance as well as a Neo4j instance with APOC already installed. The modules itself can be build without running databases. statsdb uses docker-maven-plugin to bring up PostgreSQL for generating jOOOQ-Classes, etl uses Testcontainers to do the same for PostgresSQL during integration tests. In addition it uses Neo4j test harness for an in-memory Neo4j instance.