Comments (7)
Hi @Kaash, there's nothing about server that relies on running inside docker specifically (as far as I know), but we also don't provide a nice way to run it outside docker. Prefect Server consists of several services - the easiest way we could provide users tooling for setting this up was through a dockerized setup. There might be some changes in our deployment story for Server in the coming months, but in the near future I don't believe this is something that we'll work on on our side of things. That said, if you can standup all the services in this docker-compose file, things should work the same as they would normally in docker.
Alternatively, if you'd be comfortable using Prefect Cloud instead, you can use Prefect Cloud to externally manage the things Prefect Server would, and use a local agent
to execute flow runs without docker.
from server.
Hi,
I've created a set of Terraform modules to run Prefect Server in Kubernetes.
https://github.com/mingfang/terraform-k8s-modules/tree/master/examples/prefect
It runs all the services include Prefect server, scheduler, apollo, ui, agent, hasura and postegres.
I used nginx as a trick to get the UI to talk to apollo.
Please feel free to PM me to discuss.
--ming
from server.
Hi @Kaash!
While there's nothing that requires Prefect Server to be run in Docker, spinning it up via docker-compose is our currently supported method of deploying Prefect Server. If you run into any hard-coded localhost
references you'd like to template out, we'd definitely welcome a PR back to the repo! And if your issues are specifically with the UI's localhost references, this script may be useful!
As for whether you're trying to use Prefect Server in a manner it wasn't designed for, I can't immediately say. You'll likely be able to spin it up without Docker, but it'll require customization outside of our standard deploy strategy and we may not be able to offer support. If you're open to using Cloud, I'd strongly recommend it. Aside from the fact that it side-steps the Docker issue, Cloud also avoids some of the networking and infrastructure issues folks encounter while deploying Server to a remote environment. We even have a free developer tier, so you can assess whether Prefect works well within your organization's restrictions.
from server.
@jcrist thank you very much for your response. We've been taking a look at getting Prefect Server running in my company using Kubernetes as standalone services and have made relatively good progress.
We do however have a few questions:
Services required by the Prefect server are set-up to run on localhost. In my company, we have a distributed network of Virtual machines across various data centres so we would need to ensure services are isolated and can independently run with their own IP address. Is this possible? We've modified references to localhost in the docker-compose file before running through "Kompose" to allow us to run in Kubernetes but references to localhost exist in the UI JS code too.
Will running the services via docker resolve the above problem? As mentioned previously, we have a limitation of using
Docker (which we are trying to figure out if there is a way around) and so for completeness it would be good to understand if this gets around our issue?
Finally, are we trying to use the Prefect server in a way it wasn't designed? If so, is Prefect Cloud the way forward?
Thanks, Kaash
from server.
Hi @Kaash, I'll second everything @zdhughes said.
Also - we had a livestream last week on deploying server (recording here). Note that there was a brief discussion on why docker is used currently and our feelings about deploying via other methods here in the video.
from server.
Hey guys, thank you so much for your advice and sharing the video - it proved helpful!
A bit of a status update where i'm at with deploying Prefect with Kubernetes:
- Converted docker-compose to deployment/service files via Kompose. Created separate pods for the images.
- Created ingress files to expose end-points (UI, Hasura, GraphQL and PostGres)
- The UI and Hasura services are up and running - I can access the end-points.
I am however running into a few issues - notably around the theme of networking - and was hoping you maybe able to help? Alternatively, is there any guide for setting up Prefect in Kubernetes that I can follow?
- The Prefect UI -> API Status displays http://localhost:4200/graphql however I've modified /.prefect/config.toml by changing server "endpoint" as well as "graphql_url" in server.ui to the new Graphql URL i've exposed (via Ingress) however I notice the UI does not update. Is there anything else I need to update ?
- This script is great but I note that we need sudo access which we lack in work environment. Is there a way to copy the code to a different location?
- I'm struggling to get Apollo running running in my new environment. Again I think this is network related (connection to GraphQL fails) so completely understand if you feel it's for me to figure out. At this keen for any feedback!
As always, really appreciate your help with this so far.
Many thanks,
Kaash
from server.
Hi @Kaash,
I am currently on the same path as you, looking for an automation tool that we can use in my team. Globally I hit the same wall as you with a small nuance, I can use Docker. But I would like to deploy to Kubernetes and go from the docker-compose to a Deployment. I would also like to use a pre-existing PG database and handle the localhost references.
Is there a chance you can share your work on the K8S part ?
from server.
Related Issues (20)
- Can't use special characters in `--postgres-url` connection string (user name) HOT 3
- Helm default values for env are of incorrect type HOT 1
- prefect 0.15.12 - zombie processes from hasura container HOT 10
- The latest Server release fails to start on M1 even when allocating 8GB RAM HOT 3
- Registering parameterized flows fails on 2022.01.12 HOT 1
- default value for agent.resources.limits is invalid HOT 1
- Maximum size for state payloads is limited to 1 mb HOT 4
- Update Postgres Helm chart template notes to include postgres prefix when setting password value HOT 3
- White flow run bars in the Cloud UI - flow runs created with Version 0 and with no state
- [Helm] Add RBAC support for KubeCluster (dask-kubernetes) HOT 6
- [Helm] Updating helm chart to 2022.03.29 fails HOT 1
- Cloud hooks not working with Microsoft Teams HOT 1
- helm chart (prefect-server) shouldn't contain a dash HOT 2
- Latest Helm chart does not pin versions HOT 1
- Upgrade from Ubuntu 16 CircleCI issues by May 31, 2022
- Backend Server is unable to start properly HOT 4
- Helm dep up failed because Postgresql version
- Malformed imagePullSecrets in Kubernetes agent
- Prefect Server on raspberry pi4 with Ubuntu 20.04 HOT 1
- Duplicated Helm chart
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from server.