elfocrash / cosmonaut Goto Github PK

hi,
in my project I need to add some audit property (CreatedBy, CreatedAt, etc...), is possible to remove sealed from CosmosStore class and add virtual to its methods?
If you agree I can do a PR for these changes, otherwise I create a custom version in my project.
Thanks

Does this work with Azure Functions?

Is it possible to pass in a Dictionary<string, object> to the Query<T> method on the CosmonautClient. I am generating a dynamic SQL statement based on query parameters passed into a HTTP method. To avoid SQL Injection attacks I am using the SQL query params but in my case I am unable to use an anonymous type since I do not know the object property names at design time.

When I build the SQL I end up with something like $" c.{fieldName} {op} @_{index}" which will end up as " c.code = @_0" for example

The issue I have is I cannot create an anonymous object at runtime, such as:

queryParams = new {
    _0 = "some-code"
}

So I end up with a dictionary

queryParams = new Dictionary<string, object> 
{
    { "_0", "some-code" }
}

But when I pass this into the Query method I get an error of Parameter count mismatch

Is there a way in the current implementation that supports this?

Hi,

Sorry to bother you again. I'm very new to this.

I'm trying to inset data into my cosmosdb collection called AppUsers.
Basically for my social net app, a person is able to add friend using userId and friendsUserId.
So I want to insert into collection (friendUserId in Friends) where userId="AXYZ963".
I'm not able to figure out how to do that using cosmonaut and Linq.

Here is my collection:

2. Also is there anyway to return an Id of a newly inserted data?
   I did await _cosmosStore.AddAsync(newUser); can it return id as well?

Thanks
KL

Hi,

I have set the Property CosmosEntityName but AddAsync still persists with my classname pluralized.

Is this a known issue?

Thanks!

Hello team ,
Please change in this file.
Cosmonaut-develop\samples\Cosmonaut.Console\Program.cs

Change Following line:
var firstPage = await booksStore.Query().WithPagination(1, 10).OrderBy(x => x.Name).ToPagedListAsync();
To
var firstPage = await booksStore.Query("select * from c").WithPagination(1, 10).OrderBy(x => x.Name).ToPagedListAsync();

It Gives Error

System.ArgumentException
  HResult=0x80070057
  Message=Expression of type 'System.Linq.IQueryable`1[System.Object]' cannot be used for parameter of type 'System.Linq.IQueryable`1[Cosmonaut.Console.Book]' of method 'System.Linq.IOrderedQueryable`1[Cosmonaut.Console.Book] OrderBy[Book,String](System.Linq.IQueryable`1[Cosmonaut.Console.Book], System.Linq.Expressions.Expression`1[System.Func`2[Cosmonaut.Console.Book,System.String]])'
  Source=System.Linq.Expressions
  StackTrace:
   at System.Dynamic.Utils.ExpressionUtils.ValidateOneArgument(MethodBase method, ExpressionType nodeKind, Expression arguments, ParameterInfo pi, String methodParamName, String argumentParamName, Int32 index)
   at System.Linq.Expressions.Expression.Call(Expression instance, MethodInfo method, Expression arg0, Expression arg1)
   at System.Linq.Queryable.OrderBy[TSource,TKey](IQueryable`1 source, Expression`1 keySelector)
   at Cosmonaut.Console.Program.<Main>d__0.MoveNext() in E:\Cosmonaut-develop\samples\Cosmonaut.Console\Program.cs:line 132
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Cosmonaut.Console.Program.<Main>(String[] args)

Hi Nick,
Understand that Cosmonaut is an object mapper, but wondering if there is a simple way to query for and return dynamic documents... I have imported many JSON docs in bulk which vary in terms of structure, would be great to be able to return a dynamic.

FIX IT NICK

I use a shared collection (that is non-partitioned) for my implementation, and have added the SharedCosmosCollection attribute together with the SharedCollectionName property.

Persisting entities and query (all) works fine, but the ICosmosStore.FindAsync method generates the Exception:

Microsoft.Azure.Documents.DocumentClientException: Partition key provided either doesn't correspond to definition in the collection or doesn't match partition key field values specified in the document.

Any help would be highly appreciated.

The LINQ query builder can run into an issue when you use a class level decorator like:

[JsonObject(NamingStrategyType = typeof(CamelCaseNamingStrategy))]
public class SomeEntity

to manage field names in entities. They will not be picked up so if you build a query like:
store.Query().Where(entity => entity.FieldA == "some value").ToList(); it will build a query that looks like: SELECT * FROM c WHERE c.FieldA == "some value" instead of what the json decorater makes the field name: SELECT * FROM c WHERE c.fieldA == "some value"

as a workaround I stopped using that decorator and decorated each field with [JsonProperty("fieldA")] - these get picked up correctly and build the expected query.

Hi,

I'm trying to set up Cosmonaut with my .NetASPCore project. I followed you video tutorial but I'm getting an error. How can I solve this issue?
Attached screenshot.

Hi,

Got another error while following the video. Attached below:

My controller class looks like this:

Also is there any way you can provide me a download link of the example project you work with in your Getting started video?

Thanks
Kunal

Or Do i need to check that my self before I POST?

I am facing the following problems with Moq. I am mocking an ICosmosStore repository to write unit tests in the service layer.

var fakeDataRepo = new Mock<ICosmosStore<FakeData>>();
var fakeData = Factory.CreateFakeData90;

fakeDataRepo.Setup(p => 
						p.AddAsync(It.IsAny<FakeData>()))
						.Returns<FakeData>(a => 
								Task.FromResult(
									new CosmosResponse<FakeData>(fakeData, null)));

I get the error Error CS0854 An expression tree may not contain a call or invocation that uses optional arguments

The root cause is that AddAsync has two other optional parameters RequestOptions and CancellationToken which trips up Moq. When I add the other two parameters, it obviously fails because I didn't setup the method that takes a single parameter. The problem is not with Cosmonaut per se, but it'll really help with a very popular mocking framework to also provide a version of AddAsync in the interface without the two optional parameter that just delegates to the method with the two optional parameter not being used.

Hello,

We're trying to use Cosmonaut in our multi-tenant application.
Each tenant has it's own collections, and such we need to be able to, during runtime, specify the collection name. Not only during general HTTP requests, but also during background work (events, messages, ...).
I read in the documentation that CosmosStore should preferably be registered as a singleton per entity type.
In our case that's not possible because the collection name needs to be provided when the cosmosstore is created.

What would be the preferred way going forward? One approach I was thinking, was registering a singleton factory that caches cosmosstores for tenants, and creates the correct cosmosstore once requested...

Hey there....nice library (mostly!)

It seems that because Cosmonaut is doing some synchronous initialisation of the collection and database, deadlocking can occur under certain conditions.

I think it might be good to at least make this behaviour configurable, since in many situations, it would be better to fail (e.g. there might be numerous indexing and partitioning requirements for a collection and I would rather know if I'd forgotten to provision it properly rather than the client library blindly moving ahead) but also, blocking asynchronous code within the constructor seems a little dodgy.

We can replicate this problem specifically by registering the CosmosStore as an open generic and then attempting to resolve a class which has two ICosmosStore parameters within the constructor but I haven't been able to work out exactly why this would behave like this. Also, I've only been able to replicate this within an ASP.NET core web application. It seems fine within a console app (although a little slow to initialise).

Example registration code:

services.AddSingleton(typeof(ICosmosStore<>), typeof(CosmosStore<>));

Reference to blocking async calls:

First of all, I really like this project :).

How can we call a stored procedure with this library? I didn't see any interface in the docs or in ComsosStorage.

Firstly, great work on the library, it's been so helpful in getting my app up and running on Cosmos without breaking the bank.
I'm wondering if there is a way to speed up the initialisation of my API when I'm loading lots of CosmosStores. Mine is probably a bit of an edge case. I have about a dozen entity types and multiple regions so I've created a service for each region (which gets created as a singleton) and I'm creating the CosmosStores in each service. The ctor for the service does something like:
foo = new CosmosStore(settings);
bar = new CosmosStore(settings);

... and so on.
It works great but the startup is slow and when profiling I can see that most of the time is burnt calling GetDatabaseAsync for every CosmosStore init. Depending on which region I'm calling it may burn 3 seconds per call.
Is there any way I can get that once and share it for each CosmosStore I create in the same region?
Hopefully that all makes sense :)

PS: If you're wondering why I have different regions and I'm not using Cosmos' awesome global replication features it's because I have to comply with data sovereignty issues.

Hi!

We would love to have a property LastUpdatedDate being set on the entity on every Create and Update configured directly in the store.

Down the road, this type of configuration capability would also be helpful if we were to implement soft deletes etc.

Is this possible today without wrapping the store?

Many thanks for all the great work on the client!

Typically applications only require one TelemetryConfiguration instance with one set of ITelemetryInitializer but we have a few cases where we have multiple configurations running within a single application. This problem has popped up in testing scenarios but also in odd cases where a component has implemented specific telemetry with a specific channel (and other configuration) different to the host application.

In these cases, the AppInsightsTelemetryModule is problematic because once the Initialize method has been called then all other TelemetryClient instances are ignored.

We have a similar problem with one of our ITelemetryModule implementations and have got around this by keeping a reference to multiple instances of the client - i.e. if a client has a different Instrumentation key then we reference it.

Is this something that can be addressed?

Something worths noting is that because you will use this to share objects partitioning will be virtually impossible. For that reason the id will be used as a partition key by default as it is the only property that will be definately shared between all objects.

When using Collection Sharing you say that only id can be used. But this is a major limitation imho. You should allow for a PartitionKey to be set as a property of the shareable interface.

There may well be cases where different objects have the same field (in the same level of the object / json path) and therefore they can happily co-exist in the same collection and be partitioned.

Imagine this -

Device -
{"deviceId": "foo123",...……..}

Telemetry -
{"telemetryId":"1", "deviceId": "foo123", ….}
{"telemetryId":"2", "deviceId": "foo123", ….}
{"telemetryId":"3", "deviceId": "foo123", ….}
{"telemetryId":"4", "deviceId": "foo123", ….}

etc.

If I set PartitionKey as a property on the Shareable Interface then Device and Telemetry could co-exist in the same partitioned collection. No problem.

So not really an issue. More like a feature request. :)

I have a generic service using the CosmonautClient and using shared collections, this is a single service with a single POST method that I then use to post Queries or Commands to the endpoint and return the relevant entity.

Everything in this setup works with the exception of the DeleteDocumentAsync method call which returns null and the subsequent query agains the same collection yields results containing the "deleted" document

var deleted = await this._cosmonautClient.DeleteDocumentAsync(this._databaseName, collectionName, message.Id, new RequestOptions { PartitionKey = new PartitionKey("id") });

Can you advise if this is a problem please?

NOTE: I have also tried getting the SelfLink and using the DocumentClient to delete the document this way but I get an exception stating the resource does not exist

Hi,

Currently we're using our own fork of EF Core 2.2 preview 3, with dozens of fixes, as our ORM that communcates with Cosmos DB.
We always have an API service that handles individual gets, puts, posts etc. And then we have an OData service that handles an OData query, purely for reading, and outputs a list that is displayed in a listpage to the user.
Since EF Core does not support continuation tokens at all, and we're actually at the point of thinking of moving away from EF Core, due to the (at least for now) lack of more 'complex' cosmos queries, I'm investigating the effort of moving the whole infrastructure layer to Cosmonaut.
One of the things I'm investigating here is the possiblity to use OData with Cosmonaut.

Any experiences with this?

Can support be added for proxy authentication? Version 2.2.1 of Azure.DocumentDB allows for specifying an HttpClientHandler in the DocumentClient constructor , through which proxy settings can be set so that anyone behind a proxy can connect to Azure Cosmos DB?

I'm using it like this in my own implementation:
var documentClient = new DocumentClient(serviceEndpoint, authKey, handler: handler);

Hi,

We're using the Mediatr library together with the command/query pipeline within our system.
All queries (read) that are being executed, are going through this pipeline, and are implemented using a custom-built base QueryHandler. This QueryHandler is using Polly to do an exponential backoff when there's a high load (TooManyRequests) on the database.

From what I understood when reading the documentation, the docs say that Cosmonaut will throw an exception when the requestrate is too large:

Response Handling
Cosmonaut follows a different approach when it comes to error handling. The CosmosDB SDK is throwing exceptions for almost every type of error. Cosmonaut follows a different approach.
In Cosmonaut methods that return CosmosResponse or CosmosMultipleResponse won't throw exceptions for the following errors: ResourceNotFound, PreconditionFailed and Conflict. They will instead return a CosmosResponse with the IsSuccess flag to false, the CosmosOperationStatus enum explaining what the error was and the Exception object containing the exceptions that caused the request to fail.

But when going through the code, it looks as if you are, next to the aforementioned 3 statuses, you also catch the TooManyRequests exception.
This yields some problems in our queryhandler, as Polly would only act on the exception that is being thrown, and I'd prefer not going down the route of checking the result of a query in every handler manually, which potentially can be forgotten, to throw an exception when appropriate.

Is there either a possibility to configure which of the operation statuses should be thrown as an exception, and which can be handled by the CosmosResponse?

Thanks!

BTW, we completely moved away from EF Core. Not only is our application easily 4x faster in request handling, development is also much more speed up, thanks for the framework!

Hi,

Nice work on the library! Regarding setting up a schema and validating of documents off of that schema. Does something like this already exist in the library? If not, is it on the map?

This library is something C# and Cosmos need. So I'd like to help out.

Regards,
Ethan

Can we share a collection between multiple entities? The RUs are causing big spike in cost as I have quite de-normalized data structure, can different entities point to single collection and same can work seamlessly for CRUD?

Hi Nick,

Wondering if you perhaps support a way to count documents in a collection (multi-partition)?

Currently using SELECT VALUE COUNT(1) FROM C in the portal but it doesn't seem to work for collections with multiple partitions... or else it is taking forever to return a value as its been running for over 5m on approx 70k documents?

Thanks
Leon

Hi, noticing something strange when writing doubles as fields/properties in a document... they always end up being rounded when written into the database.

Can't seem to find a way to prevent this... eventually had to switch the property to a string type and store it that way...

Hi,

I am trying to use the continuation token approach to paginate... It works fine for a couple pages but then it fails with Request headers must contain only ASCII characters. -- in looking at the token, there are indeed non-ascii characters in the continuation token (people's names with accents).

Is there something I can do to transform the token to make it safe or is it something that can be fixed in Cosmonaut?

Thanks!

It would be nice to include the default TTL setting when creating collections.

The TTL field on a document is not acknowledged if the collection does not have TTL enabled

So I was following what you suggested that I create another collection to save relationships.

This is how I set it up:

1. Model

2. Initiation:

3. Post Method:

But I'm getting error:

Do I need to create another controller to handle this?

Is there any way through this library (or the SDK even) to adjust the scale of the new database level scaling feature (where you can have many collections share the same RU/s limit).

I did find this way of doing it in Powershell, and it works:
https://github.com/PlagueHO/CosmosDB/blob/82199fe9df0e8648ab8829adaa9c938cbde1de66/src/lib/offers/Set-CosmosDbOffer.ps1

Can you help me here? great library btw!

https://stackoverflow.com/questions/51468975/adding-a-cosmosdb-entity-unable-to-resolve-id-for-entity-of-type-tenant

I have a CosmosDB collection where I am storing ISO formatted dates as strings, for example, I have saved a document with a start value of 2018-07-26T09:00:00Z and when I look in the Data Explorer I can see the value saved correctly.

I am using Cosmonaut to access my CosmosDB which exposes the DocumentClient and I am using CreateDocumentQuery to access my document but when the document gets returned from CosmosDB the start value is converted to a DateTime string 07/26/2018 09:00:00

var collectionUri = UriFactory.CreateDocumentCollectionUri("assessment-center", "ac-template-timetables");
Document document = await _cosmonautClient.DocumentClient.CreateDocumentQuery(collectionUri, new SqlQuerySpec
{
    QueryText = "SELECT * FROM c WHERE c.id = '713d5115-83cb-4005-abaa-25756ebc86d1'"
}, new FeedOptions { EnableCrossPartitionQuery = true }).FirstOrDefaultAsync();

The above is my query, has anyone else encountered this and if so how did they resolve this problem, my expectation is that my string value should be returned as is and not interpreted as a DateTime which seems to be what is happening

It's been observed that the FindAsync method will ignore the entity type when used in a shared collection and it will return a value cast as the CosmosStore object type.

Awesome library!! However, when I use CountAsync I get a query that looks like this:

{"query":"SELECT VALUE root FROM root WHERE (root[\"AccountId\"] = XXXXX) "}

The regular Count() function will properly transform the query into SELECT VALUE COUNT(1) FROM...
The performance implications of pulling all of the data in memory to perform the count are obviously very bad.

My query is nothing fancy, it looks like this:

await ArtistStore.Query().Where(x => x.AccountId == accountId).CountAsync();

Hi,

Is it possible to rewrite so that AddAsync (and similiar) in ICosmosStore returns some form of interface instead of CosmosResponse. This would simplify mocking/testing a lot.

Many thanks!

I am looking to create a REST API with generic methods for CRUD across many collections and would like to know if it is possible to create a CosmosStore instance using a collection name string, or alternatively what would be the best way to accomplish this?

I followed the documentation as stated, and when I try to create a ICosmosStore to get information from CosmosDB, I got webapi timeouts, when I attached the debugger, I noticed the problem is not in the webapi, but apparently something in the cosmonaut library

My code is very simple>

[HttpGet]
        public async Task<List<SharepointTenant>> GetTenants()
        {
            //var tenantStore = CosmosStoreFactory.CreateForEntity<SharepointTenant>();

            var  _settings = new CosmosStoreSettings(ConfigurationManager.AppSettings["database"].ToString(),
                ConfigurationManager.AppSettings["endpoint"].ToString(),
                ConfigurationManager.AppSettings["authKey"].ToString());

            ICosmosStore<SharepointTenant> tenantStore = new CosmosStore<SharepointTenant>(_settings);

            return await tenantStore.Query().Where(x => x.TenantName != null).ToListAsync();

        }

I think that your implementation of FirstOrDefaultAsync is not cost-optimal.

In this line:

You are calling to GetSingleOrFirstFromQueryable. Without setting RequestContinuation there is a call to GetResultsFromQueryToList which performs the query and store the results in a list.
Even that stopOnAny is set to true, I suspect that you are getting from Cosmos more than 1 entry as a result.
You are using the FirstOrDefault after the query. Isn't it inefficient?

Don't you need to set FeedOptions.MaxItemCount = 1 in that case, in order to force Cosmos to return only 1 result and to reduce the amount of the transferred data (which also will reduce the cost)?

First, thank you for putting the work into this great library.

I see that you throw an exception if more than 1 property is decorated with the CosmosPartitionKey attribute. Is there a specific reason?

Instead of having client code populate a single property with a synthetic partition key value (per MS Create a synthetic partition key), could the CosmosPartitionKey be enhanced to allow multi-property decoration with an int position parameter to establish the assembly order? Then use PartitionKeyDefinition when creating the collection to include multiple paths.

A pattern I've used with Azure's SQL Server and Postgres PaaS offerings is to use a single instance to hold data for all our test environments (e.g. dev, test, qa), by using database schemas. For example, if I had a Widgets table, it would exist in all of the schemas:

dev.Widgets
test.Widgets
qa.Widgets

This reduces costs, as there is only 1 database instance instead of 3, and also means there are less Azure resources to manage.

I'd like to do the same with Cosmos DB, but AFAIK it doesn't have a schema/namespace concept, so the idea would be to prefix the name of each collection with the environment, e.g.:

dev__Widgets
test__Widgets
qa__Widgets

What I'm looking for is perhaps a CollectionPrefix property on CosmosStoreSettings.

Is there any way to achieve this at present? If not, are you open to discussing a mechanism for this?

Hi,
I'm seeing the following since updating to the latest version of Cosmonaut. Where/how would I best enable Cross partition query, can't seem to find it.
Many thanks, L

Cross partition query is required but disabled. Please set x-ms-documentdb-query-enablecrosspartition to true, specify x-ms-documentdb-partitionkey, or revise your query to avoid this exception.
ActivityId: dfb5cf0a-cdff-4b1b-9996-2c714b5690bd, Microsoft.Azure.Documents.Common/2.1.0.0, Darwin/10.14 documentdb-netcore-sdk/2.1.3

Hello,

I could not find this directly in the documentation, but, Cosmos DB, since recently, allows to define a shared throughput on DB level, instead of collection level.
As such, I want Cosmonaut to not set a fixed throughput on collection level when creating the collection. How can I achieve this?

I use multiple shared collections in my app. I want to make sure that I injection the collection name for groups of entities. How can I accomplish that with Cosmonaut?

Currently, the only way to use Shared Collection is through an attribute, which doesn't lend itself to injecting a collection name.

Command operations in shared collections throw an exception.

Hi, I've a test for NotFound resource by using wrong id.
DocumentClient throw this exception "Microsoft.Azure.Documents.DocumentClientException: Message: {"Errors":["Resource Not Found"]}" in Visual Studio Locals tab $exception variable, but CosmonautClient return Success and no exception catched.

try
{
       result = await cosmosService.EntityStore.UpdateAsync(Entity.Value);
}
catch (DocumentClientException ex)
{
       var error = ex; //Breakpoint here ignored
       throw;
}

return result.Exception is null ? // this is null
     (ActionResult)new OkObjectResult($"Success") : 
     new BadRequestObjectResult($"Error");

Hi, just wondering if there is any way one could limit the number of documents returned for a particular search query... ie something like a .Take(10)

Hey Guys,

I was investigating Cosmonaut for a multi-tenanted system. Basically I was looking to override the CosmosStore to use the visitor pattern to ensure that queries always include a tenant id. However I see that CosmosStore is sealed. Is there any particular reason for sealing it?