Live hosted instance :) about traderx HOT 11 CLOSED

Yes, agreed on everything. Is there any data bootstrap process already in place? If so, we could start an ECS scheduled task that wipes-out / bootstraps data every X hours and we keep it simple for now; in the future we can delegate to AWS RDS.

The initialSchema.sql file contains both the schema and initial data load. I think the easiest thing would be to simply schedule to restart the entire instance every 24 hours.

from traderx.

Shall we track that work on a separate issue? Ultimately what we have now is a live view of what it is (for better or worse)

from traderx.

Question on this one - what hosting infra is available? Using the existing docker compose setup this should be pretty straight forward. Tag: @karlmoll FYI

from traderx.

thanks @rocketstack-matt - i will bring this conversation up internally as we get this meetup figured out. appreciate the tag!

from traderx.

Hi @rocketstack-matt - thanks for raising this, I was about to open the same conversation, as I quickly discussed with @mindthegab yesterday and had a follow up with @DovOps on my todo list.

I think that we can easily spin up the Docker Compose, and I'll discuss the technical details below, but first I'd like to better understand the purpose of this environment, its users and the data flow.

For example, we want to spin up a demo.traderx.finos.org frontend, load the system with mock data, which gets re-bootstrapped every 48 hours, with a user (credentials trader/trader) that can access and play a bit with the system; something like demo.waltz.finos.org . I'm not sure if this is what you had in mind, or you were thinking about something different.

Now for the technical part.

We usually run project infrastructure on AWS, but we're open to discuss other providers if you prefer; I must admit, I've never deployed a Docker compose in production, as I always considered it a development/staging tool; however, it seems that things have changed.

Searching for "Docker Compose on AWS" I found this, which seems to be a decent starting point (and uses ECS, which costs way less than EKS).

It would help if the docker compose definition defines cpu and memory reservations/limits, in order to get a better idea of the costs.

That's the bare minimum I guess; after that, we'll probably have to build a bunch of scripts/configurations around it to manage automated backups and restores, some AWS configuration, DNS configurations and that's it.

Eager to hear your thoughts!

from traderx.

Hey @maoo I guess the purpose I'll let @mindthegab and @DovOps opine on, but your description seems inline with my assumptions of what they'd be after.

I would agree with you regarding Docker Compose in a real prod environment, but think it would probably do the job for this demo instance. I can look to add in the compute requirements, but I know from running this up in Codespaces that it requires an environment with at least 32GB for everything to run up successfully.

Do you think we would even need backups? My assumption is we'd run this purely as a demo and any data loss would be acceptable. If we want something more than that then we may want to look at hooking it up to a DB service which could handle the backup, but I guess we're back to your first question of what is the actual use case.

FYI I had flagged this for @karlmoll as we were discussing a meetup he's planning on the new year which TraderX will be at least a large part of.

from traderx.

Hey @maoo I guess the purpose I'll let @mindthegab and @DovOps opine on, but your description seems inline with my assumptions of what they'd be after.

I would agree with you regarding Docker Compose in a real prod environment, but think it would probably do the job for this demo instance. I can look to add in the compute requirements, but I know from running this up in Codespaces that it requires an environment with at least 32GB for everything to run up successfully.

Ok, that's already a good indication. I'll try to find some time tomorrow to start playing with it.

Do you think we would even need backups? My assumption is we'd run this purely as a demo and any data loss would be acceptable. If we want something more than that then we may want to look at hooking it up to a DB service which could handle the backup, but I guess we're back to your first question of what is the actual use case.

Yes, agreed on everything. Is there any data bootstrap process already in place? If so, we could start an ECS scheduled task that wipes-out / bootstraps data every X hours and we keep it simple for now; in the future we can delegate to AWS RDS.

FYI I had flagged this for @karlmoll as we were discussing a meetup he's planning on the new year which TraderX will be at least a large part of.

Thanks for that! It's always good to have another pair of eyes 😄.

from traderx.

Hey @rocketstack-matt - I did a first run of tests, and it didn't go really well to be honest:

• The Docker Compose ECS integration project was archived last month - https://github.com/docker/compose-ecs ; still works, but I'm not sure how long it will be shipped in the Docker CLI
• I had to publish images to ECR, in order to get it to work; I guess this is code that will come in handy if/when we want to run it on AWS, maybe as GitHub Actions ; see gist below.
• I run a docker-compose with just the database and had to strip off the volumes part (not supported by the ECS integration), so we'd have to figure out how to deal with that

This is the gist - https://gist.github.com/maoo/f86eef1e99563a5cfaa33df5371b6804

Before moving forward with this, I'd consider the Helm option; it's reusable, production-ready, works across providers, and probably fairly easy to write, starting from the docker-compose we already have.

I guess we can make the Docker ECS integration work, but with some tweaking and the uncertainty that new docker cli versions will break it. Maybe there are also alternatives - that I couldn't find - on how to deploy Docker Compose on AWS.

Eager to hear your thoughts. Thanks!

from traderx.

Quick update on this. We decided to start spinning up an EC2 instance that runs docker-compose, while - in parallel - build support for cloud-native deployments.

We'll update this issue as soon as we have a running endpoint of TraderX on FINOS.

from traderx.

Hey folks, quick update; after some iterations, we now have https://demo.traderx.finos.org/ up and running.

We COULD probably close this issue, although we believe that in order to have a "live demo" we should probably do some basic UI/UX work.

from traderx.

Shall we track that work on a separate issue? Ultimately what we have now is a live view of what it is (for better or worse)

Opened #125 , closing this one. Thanks everyone!

from traderx.