swisslife-oss / squadron Goto Github PK

Describe the bug
Neo4J container fails if it does not start within 5 secs

To Reproduce
Steps to reproduce the behavior:

1. Occurs when running multiple containers at once
2. The ready state is not configured properly which throws an error

We should also provide a convenient way to run azure functions

There should have a Provider creating an AzureStorageAccount in Azure. Should be implemented like the Provider for Servicebus

With the new restrictions of docker you can quickly run into the issue that you pull to many images from the server.

It is possible to configure to prefere the local image in ContainerResourceSettings.
At the moment it is set to false

        public bool PreferLocalImage {get; internal set;} = false;

I think we should change this to true

See #85

To test transactions with mongodb a replicateset is required.

Update SqlServer dependencies in order to remove .NET 4.6 support

At the moment, the MongoReplicaSetResource of Squadron creates one docker container with a MongoDB server which runs in the replica set mode.
Nice to have would be, if we could choose how many nodes we want to have for the replica set, then Squadron creates via docker compose the containers and on each container runs one MongoDB node of the replica set.

Is your feature request related to a problem? Please describe.
When debugging a test case and killing the debug process, the containers are not cleaned up, but live forever.

Describe the solution you'd like
Java testcontainers has found a solution to this by spinning up a container, that kills other containers, after the test process got killed. Maybe you can find some inspiration in their approach:

testcontainers/testcontainers-java#545

Hi @glucaci,

at our company we had a hackathon. One of the subjects were your project, squadron.

We set out to implement some features that we thought were missing and it resulted in three PRs.

• #95
• #96
• #98

Is this something your are intersted in your main branche?
If not, then we will start investigation a solution on our end, to setup an internal nuget feed, from where we can utilise the changes.

Squadron containers uses automatic port assignment from Docker. The ports range starts from 32768 to 61000 and increments with every container created. If the port is already taken the docker api doesn't throw an exception or takes next port. It's also unclear how to reset the current port counter that's assigned since it's persisted across pc restarts or even docker removal from pc.

A bug is created on Docker Cli.

Possible solution is to set other port range on the system itself like described here .

When updated docker desktop to v4.7.0 he library stopped working and throws exceptions:
Squadron.ContainerException : Error in ImageExists: mongo:latest ---- Docker.DotNet.DockerApiException : Docker API responded with status code=BadRequest, response=400 Bad Request

Steps to reproduce the behavior:

1. Update docker desktop to v4.7.0
2. Run tests using squadrom's mongo instance

Desktop (please complete the following information):

• OS: Windows 10
• Version 21H1 (19043.1466)

Because of the limitations on MacOS container communication should fallback to localhost:port. Communication over IP is not possible.

Collect test result and use them in build job in order to optimize the build time

Creation of Service Bus Namespace is a timeconsuming task in Azure, thus using an existing named Service Bus Namespace is preferable to achieve high speed integration testing. On the other hand it is preferable to avoid manual setup of Service Bus Namespaces in Azure.

The solution could consist of the following:

• When a named Service Bus Namespace exists within resource group it will be used as is
• When a named Service Bus Namespace does not exist within resource group it will be created

I'm trying to add support for OPA to Squadron.
My plan was to use the REST API OPA offers.
The problem I think I have is that the container immediately exists after it started. When IsReadyAsync of my status object is called, the container has already exited and the connection to the health endpoint results in Target machine actively refused.

On the command line I would run the container like so: docker run -it --rm -p 8181:8181 openpolicyagent/opa run --addr :8181

Can I pass the same arguments before calling base.InitializeAsync() somehow so that the initialization runs the container with these parameters?

Is your feature request related to a problem? Please describe.
Docker allows you to spin up a container and expose multiple port settings on it. As far as I can see, the ContainerResourceBuilder only allows for one port to be mapped.
Concrete example where this is useful:
VerneMq Mqtt exposes port 1883 for connections, but also 8888 for exposing health checks and metrics.

Describe the solution you'd like
Allow multiple port mappings per container by adapting the api of ContainerResourceBuilder ports to the same approach as cmds, volumes and env vars.

Microsoft provides a Docker image for the Azure Blob Storage. Support for this should be integrated.

Complete documentation for:

• Azure Blob and Queues
• RabittMQ
• Redis
• PostgresSQL
• Azure Service Bus

Because there is no ServiceBus container support from Microsoft this will be done by targeting a configurable ServiceBus subscription

Is your feature request related to a problem? Please describe.

Would like to see Neo4j added.

Describe the solution you'd like

Able to spin up and test Neo4j.

Hi

Is your feature request related to a problem? Please describe.
The mssql image mcr.microsoft.com/mssql/server:2019-latest which we wanted to start needs at least 2GB ram. Due to our build server environment (see chapter Additional context) we need to specify the --memory option for the container else it won't run. With Squadron we don't have any possibilities to define this option.

Describe the solution you'd like
I saw in your code that you create the HostConfig within the method CreateContainerAsync() in the class DockerContainerManager. Would be nice if we can provide our own HostConfig to specify the Memory property or that we at least can configure it through the ContainerResourceBuilder.

Additional context
Locally we use WSL2 as docker backend and on our build server we use Windows Server 2019 1809 with LCOW to run linux containers in Hyper-V isolation. The default memory limit is 1GB for linux container in Hyper-V isolation mode.

Thank you for your answer!

Hi

Is your feature request related to a problem? Please describe.
We use MongoDB with replica sets to support transactions. We wanted to define another image and prefer the local image instead of pulling it every time (due to Docker Hub's new pull rate limit). Therefore we have to implement an own options class to override the Configure method. But to use Mongo with replica sets and our options we have to create a custom class which also inherits from MongoResource but with the own options class as generic parameter and this looks the same as the MongoReplicaSetResource.

    public class MongoContainerOptions : MongoDefaultOptions
    {
        public MongoContainerOptions(string replicaSetName)
        {
            ReplicaSetName = replicaSetName;
        }

        public MongoContainerOptions()
            : this("rs0")
        {
        }

        internal string ReplicaSetName { get; }

        public override void Configure(ContainerResourceBuilder builder)
        {
            base.Configure(builder);

            builder.AddCmd("--replSet", ReplicaSetName)
                .Image("mongo:bionic")
                .PreferLocalImage();
        }
    }

    public class MongoContainerResource : MongoResource<MongoContainerOptions>
    {
        public override IMongoClient Client => new MongoClient(ConnectionString);

        public async override Task InitializeAsync()
        {
            await base.InitializeAsync();
            var client = new MongoClient(ConnectionString + "/?connect=direct");
            var rsConfig = CreateReplicaSetConfiguration();
            var command = new BsonDocumentCommand<BsonDocument>(new BsonDocument
            {
                { "replSetInitiate", rsConfig },
            });

            await client.GetDatabase("admin").RunCommandAsync(command);

            await Initializer.WaitAsync(new MongoReplicaSetStatus(ConnectionString));
        }

        private BsonDocument CreateReplicaSetConfiguration()
        {
            var membersDocument = new BsonArray
            {
                new BsonDocument
                {
                    { "_id", 0 },
                    { "host", $"{Manager.Instance.Address}:{Settings.InternalPort}" },
                },
            };

            var cfg = new BsonDocument
            {
                { "_id", ResourceOptions.ReplicaSetName },
                { "members", membersDocument },
            };

            return cfg;
        }
    }

Describe the solution you'd like
Is it possible also to provide a MongoReplicaSetResource class where we can inherit with a custom options class without implementing again the logic of the current MongoReplicaSetResource class (as you have done with MongoResource)?

    public class MongoContainerOptions : MongoReplicaSetDefaultOptions
    {
        public MongoContainerOptions(string replicaSetName)
            : base(replicaSetName)
        {
        }

        public MongoContainerOptions()
            : this("rs0")
        {
        }

        public override void Configure(ContainerResourceBuilder builder)
        {
            base.Configure(builder);

            builder.AddCmd("--replSet", ReplicaSetName)
                .Image("mongo:bionic")
                .PreferLocalImage();
        }
    }

    public class MongoContainerResource : MongoReplicaSetResource<MongoContainerOptions>
    {
    }

Additional context
Locally we use WSL2 as docker backend and on our build server we use Windows Server 2019 1809 with LCOW to run linux containers in Hyper-V isolation.

Thank you!

There is one testcase found here, which passes, but doesn't actually validate if the volume works. If you start the testcase and inspect the container, you will see, that the .AddVolume("/dev/shm:/dev/shm"); call of the builder will actually result in a volume, that maps from the docker directory onto the path "/dev/shm:/dev/shm" in the container.
See:

and
https://docs.docker.com/engine/api/v1.41/#operation/ContainerCreate

I'll try to fix this issue and provide a PR for fixing it.

At the moment we are using one container per class instance since in most cases we inject a resource as a IClassFixture. This can really drag down test performance. In many cases, for the test, it would be enough to create a new database within the container.

So the question is should we have some kind of a runtime state and reuse a container instance through multiple resources. I think the main issue why we did not do that was that we could not determine when the last test has be run and we can get rid of the container. Essentially a test run wide init.