Giter VIP home page Giter VIP logo

marten's Introduction

Marten

.NET Transactional Document DB and Event Store on PostgreSQL

Discord Twitter Follow Windows Build Status Linux Build status Nuget Package Nuget

marten logo

The Marten library provides .NET developers with the ability to use the proven PostgreSQL database engine and its fantastic JSON support as a fully fledged document database. The Marten team believes that a document database has far reaching benefits for developer productivity over relational databases with or without an ORM tool.

Marten also provides .NET developers with an ACID-compliant event store with user-defined projections against event streams.

Access docs here and v3.x docs here.

Support Plans

JasperFx logo

While Marten is open source, JasperFx Software offers paid support and consulting contracts for Marten.

Help us keep working on this project ๐Ÿ’š

Become a Sponsor on GitHub by sponsoring monthly or one time.

Past Sponsors

.NET on AWS

Working with the Code

Before getting started you will need the following in your environment:

1. .NET SDK 8.0+

Available here

2. PostgreSQL 12 or above database

The fastest possible way to develop with Marten is to run PostgreSQL in a Docker container. Assuming that you have Docker running on your local box, type: docker-compose up or dotnet run --framework net6.0 -- init-db at the command line to spin up a Postgresql database withThe default Marten test configuration tries to find this database if no PostgreSQL database connection string is explicitly configured following the steps below:

PLV8

If you'd like to use Patching Api you need to enable the PLV8 extension inside of PostgreSQL for running JavaScript stored procedures for the nascent projection support.

Ensure the following:

  • The login you are using to connect to your database is a member of the postgres role
  • An environment variable of marten_testing_database is set to the connection string for the database you want to use as a testbed. (See the Npgsql documentation for more information about PostgreSQL connection strings ).

Help with PSQL/PLV8

  • On Windows, see this link for pre-built binaries of PLV8
  • On *nix, check marten-local-db for a Docker based PostgreSQL instance including PLV8.

Test Config Customization

Some of our tests are run against a particular PostgreSQL version. If you'd like to run different database versions, you can do it by setting POSTGRES_IMAGE env variables, for instance:

POSTGRES_IMAGE=postgres:15.3-alpine docker compose up

Tests explorer should be able to detect database version automatically, but if it's not able to do it, you can enforce it by setting postgresql_version to a specific one (e.g.)

postgresql_version=15.3

Once you have the codebase and the connection string file, run the build command or use the dotnet CLI to restore and build the solution.

You are now ready to contribute to Marten.

See more in Contribution Guidelines.

Tooling

Build Commands

Description Windows Commandline PowerShell Linux Shell DotNet CLI
Run restore, build and test build.cmd build.ps1 build.sh dotnet build src\Marten.sln
Run all tests including mocha tests build.cmd test build.ps1 test build.sh test dotnet run --project build/build.csproj -- test
Run just mocha tests build.cmd mocha build.ps1 mocha build.sh mocha dotnet run --project build/build.csproj -- mocha
Run StoryTeller tests build.cmd storyteller build.ps1 storyteller build.sh storyteller dotnet run --project build/build.csproj -- storyteller
Open StoryTeller editor build.cmd open_st build.ps1 open_st build.sh open_st dotnet run --project build/build.csproj -- open_st
Run docs website locally build.cmd docs build.ps1 docs build.sh docs dotnet run --project build/build.csproj -- docs
Publish docs build.cmd publish-docs build.ps1 publish-docs build.sh publish-docs dotnet run --project build/build.csproj -- publish-docs
Run benchmarks build.cmd benchmarks build.ps1 benchmarks build.sh benchmarks dotnet run --project build/build.csproj -- benchmarks

Note: You should have a running Postgres instance while running unit tests or StoryTeller tests.

xUnit.Net Specs

The tests for the main library are now broken into three testing projects:

  1. CoreTests -- basic services like retries, schema management basics
  2. DocumentDbTests -- anything specific to the document database features of Marten
  3. EventSourcingTests -- anything specific to the event sourcing features of Marten

To aid in integration testing, Marten.Testing has a couple reusable base classes that can be use to make integration testing through Postgresql be more efficient and allow the xUnit.Net tests to run in parallel for better throughput.

  • IntegrationContext -- if most of the tests will use an out of the box configuration (i.e., no fluent interface configuration of any document types), use this base type. Warning though, this context type will not clean out the main public database schema between runs, but will delete any existing data
  • DestructiveIntegrationContext -- similar to IntegrationContext, but will wipe out any and all Postgresql schema objects in the public schema between tests. Use this sparingly please.
  • OneOffConfigurationsContext -- if a test suite will need to frequently re-configure the DocumentStore, this context is appropriate. You do not need to decorate any of these test classes with the [Collection] attribute. This fixture will use an isolated schema using the name of the test fixture type as the schema name
  • BugIntegrationContext -- the test harnesses for bugs tend to require custom DocumentStore configuration, and this context is a specialization of OneOffConfigurationsContext for the bugs schema.
  • StoreFixture and StoreContext are helpful if a series of tests use the same custom DocumentStore configuration. You'd need to write a subclass of StoreFixture, then use StoreContext<YourNewStoreFixture> as the base class to share the DocumentStore between test runs with xUnit.Net's shared context (IClassFixture<T>)

Mocha Specs

Refer to the build commands section to look up the commands to run Mocha tests. There is also npm run tdd to run the mocha specifications in a watched mode with growl turned on.

Note: remember to run npm install

Storyteller Specs

Refer to build commands section to look up the commands to open the StoryTeller editor or run the StoryTeller specs.

Current Build Matrix

CI .NET Postgres plv8 Serializer
GitHub Actions 6 12.8 โœ… STJ
GitHub Actions 6 15-alpine โŒ Newtonsoft
GitHub Actions 7 12.8 โœ… JSON.NET
GitHub Actions 7 latest โŒ STJ
Azure Pipelines 8 12.8 โœ… JSON.NET
Azure Pipelines 8 12.8 โœ… STJ
Azure Pipelines 8 15-alpine โŒ STJ
Azure Pipelines 8 latest โŒ Newtonsoft

Documentation

All the documentation is written in Markdown and the docs are published as a static site hosted in Netlify. v4.x and v3.x use different documentation tools hence are detailed below in separate sub-sections.

v4.x and above

VitePress is used as documentation tool. Along with this, MarkdownSnippets is used for adding code snippets to docs from source code and Algolia DocSearch is used for searching the docs via the search box.

The documentation content is the Markdown files in the /docs directory directly under the project root. To run the docs locally use npm run docs with auto-refresh on any changes.

To add code samples/snippets from the tests in docs, follow the steps below:

Use C# named regions to mark a code block as described in the sample below

#region sample_my_snippet
// code sample/snippet
// ...
#endregion

All code snippet identifier starts with sample_ as a convention to clearly identify that the region block corresponds to a sample code/snippet used in docs. Recommend to use snake case for the identifiers with words in lower case.

Use the below to include the code snippet in a docs page

<!-- snippet: sample_my_snippet -->
<!-- endSnippet -->

Note that when you run the docs locally, the above placeholder block in the Markdown file will get updated inline with the actual code snippet from the source code. Please commit the changes with the auto-generated inline code snippet as-is after you preview the docs page. This helps with easier change tracking when you send PR's.

Few gotchas:

  • Any changes to the code snippets will need to done in the source code. Do not edit/update any of the auto-generated inline code snippet directly in the Markdown files.
  • The latest snippet are always pulled into the docs while we publish the docs. Hence do not worry about the inline code snippet in Markdown file getting out of sync with the snippet in source code.

v3.x

stdocs is used as documentation tool. The documentation content is the markdown files in the /documentation directory directly under the project root. Any updates to v3.x docs will need to done in 3.14 branch. To run the documentation website locally with auto-refresh, refer to the build commands section above.

If you wish to insert code samples/snippet to a documentation page from the tests, wrap the code you wish to insert with // SAMPLE: name-of-sample and // ENDSAMPLE. Then to insert that code to the documentation, add <[sample:name-of-sample]>.

Note: content is published to the gh-pages branch of this repository. Refer to build commands section to lookup the command for publishing docs.

License

Copyright ยฉ Jeremy D. Miller, Babu Annamalai, Oskar Dudycz, Joona-Pekka Kokko and contributors.

Marten is provided as-is under the MIT license. For more information see LICENSE.

Code of Conduct

This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community.

marten's People

Contributors

adamralph avatar cocowalla avatar coreykaylor avatar danielmarbach avatar davehylde avatar ekulakov avatar elexisvenator avatar eouw0o83hf avatar erdtsieck avatar ericgreenmix avatar gfoidl avatar hawxy avatar jabellard avatar jenspettersson avatar jeremydmiller avatar jimgolfgti avatar jokokko avatar leh2 avatar mdissel avatar mysticmind avatar nieve avatar oakio avatar oskardudycz avatar phillip-haydon avatar scooletz avatar seankilleen avatar simoncropp avatar tim-cools avatar villehakli avatar wastaz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

marten's Issues

Document Projections

Just a placeholder for now. Have some kind of model that allows you to do transforms to read side views in the database. Might finally get into PLV8 code inside of postgres. I'm thinking that we could do this in lieu of supporting truly crazy LINQ queries of the real model.

Support Select() from the MartenQueryable() ????

If you do session.Query().Where(blah blah blah).Select(x => x.Something) it goes pear shaped. You have to do an intermediate ToArray() or ToList() in between.

Think about this and see if anyone really wants projections directly over the top of the document tables. Might temporarily have it fetch and deserialize, then do the projection in memory. Because you know that no one would kill their perf by using that naively.

Optimize the Document.SaveChanges()

Maybe use data adapters in ADO.Net to submit all the changes in one batch? May not be that crazy. Another alternative would be to submit a batch of JSON and have a plv8 sproc in the database do the work of pushing the data to the right places.

ProductionDocumentSchema

A production time version of IDocumentSchema that blows up when an unknown document type is encountered. Can't be building new database tables in production in most likelihood anyway.

Consider More Proven Document Stores

Don't take this as knocking postgres, which I like in other contexts...but it seems that using something slightly less new (and in use in more projects or companies) would be worthwhile. There's always redis.

Helpers to retrieve the raw JSON

For web app scenarios, when all you wanna do is spit the raw json directly over the outgoing HTTP response, just get the raw json data as a string and spit it out over the wire.

Synchronous and Async. Might investigate some kind of lightweight way to strip out the type information that comes in from Json.Net or whatever the serializer is gonna be.

Command line management tool?

I think we need some kind of facility to generate the database schema objects for document types. Think about this one more. Is it a console tool on top of the library support? Migrations?

Possible features would be:

  1. Dump the full DDL for the entire schema
  2. Generate patches for any places where the schema is different than the Marten configuration
  3. Bulk data loading?
  4. Wipe out Marten schema objects?
  5. Load the Javascript projections?

Support Single() or First() in the Linq support

I think this mostly means implementing DocumentSession's IQueryExecutor.ExecuteSingle<T>() method

  • Single hit only one
  • Single hit more than one
  • Single miss
  • First hit
  • First miss
  • SingleOrDefault hit only one
  • SingleOrDefault hit more than one
  • SingleOrDefault miss
  • FirstOrDefault hit
  • FirstOrDefault miss

Optionally use Sequence for numeric Id property (and/or high/low implementation)

Implement an attribute to identify a Id property as [Identity] with one or more implementations:

  • fetch next number from sequence when DocumentSession.Save(..) is called and assign to Id property
  • use high-low algorithm at domain level to assign next number from range or fetch next range from sequence. Set Id property when DocumentSession.Save(..) is called

Define indexes by document type in code

This is going to need a lot more thought. What I'd like is a way to define Marten index's in code for common queries that may:

  1. Create extra fields and matching db indexes in the document tables to duplicate information in the son payloads specifically to make querying more performant
  2. Somehow directs the Linq query support to use these extra fields
  3. Adds something to the DB to duplicate that data upon inserts or updates

Support more .Net primitive types

Need to add tests and postgres casts in MartenExpressionParser for:

  • bit [ (n) ] --->Byte[]
  • bit varying [ (n) ] --->Byte
  • bytea --->Byte[]
  • interval [ (p) ] --->TimeSpan

Support partial document updates?

I used this feature a tiny bit with Raven. It's actually possible w/ Postgresql 9.5 to do partial updates to JSON documents, but I think you'd have to express the change as JSON. Not sure it'd be worth doing a strong typed facade over that, but could be wrong. Maybe use .Net's Dynamic type

Compiled Queries

So I'm kind of freaked out now that I've seen how many intermediate steps there are between a Linq expression and actually getting the SQL command out of that.

For performance, what if we had a way to express a commonly used query as a Linq statement once and get the SQL string w/ the right parameters built upfront so you can avoid that parsing hit over and over again.

I've got some vague ideas about doing this, but nothing definite yet

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.