Consider adding Docker-based TFX workshop tutorial about tfx HOT 18 CLOSED

from tfx.

from tfx.

Any update on this? I'm building a similar image right now and could use this.

I believe that this idea has become obsolete as reported in that pr. If that is the situation maybe we could consider closing this issue.

from tfx.

While I liked this idea and have mentioned it from times to times, we need to determine how much we will keep a docker based tutorial similar to the step by step workshop . Specifically, if we want to preserve the step oriented progression, how will the workflow looks like for the user?

@rcrowe-google

from tfx.

Also to think about how the development flow will look like if user started the tutorial w/ a simplified docker based tutorial, but decided to proceed to use TFX to solve their own problem (on their own data and code). We want to make sure that the tutorial serves as a step stone towards that.

from tfx.

I'm interested in working on this tutorial. Is anybody already working on it?

I believe that to solve the issues mentioned by the @zhitaoli would need to separate the installation section from the usage example. Thinking as a developer start using the application he could look for installation guide that would suit him best and then move on to the get started page. That way we could maintain the installation and get started documentation separately.

from tfx.

Searching in dockerhub I find that image (that I believe it's official). The image is prepared to run the run executor script. Since that the image is not referenced by the official documentation should I use it as a base ? I mean, that is a official image ?

If that is not recognized as a official, it's a goal of the project run the system in a docker image ?

from tfx.

Hi @Bumbleblo,

Yes this image is the official TFX image. It currently runs the run_executor script and will later be converted to a (WIP) run_component.py which runs driver/executor/

I think it's the right thing to extend this image (through "FROM tensorflow/tfx" docker file clause), install apache-airflow/mysql/etc necessary to run the demo, and create another image (i.e, "tensorflow/tfx:latest-airflow-demo"). I can help once we have this image built.

from tfx.

I already create a initial version for that image (that version is using sqlite). Now I'm trying to understand what is important in default setup script of the workshop.

Is there any other step that was not in that script?

from tfx.

from tfx.

Great! Can you post the Dockerfile for the initial version? IMHI, high level things we need from the default setup script should be: - install jupyter notebook; - install notebook related dependencies; - start notebook on a different port and expose that port to browser; - make sure pipeline data and sqlite DB file is on a volume (which can be mounted to the container); The part of copying python files around should be more or less optional as long as you can get Airflow running.
…
On Fri, Jun 28, 2019 at 7:32 AM Felipe Borges @.***> wrote: I already create a initial version for that image https://hub.docker.com/r/bumbleblo/tfx (that version is using sqlite). Now I'm trying to understand what is important in default setup script of the workshop https://github.com/tensorflow/tfx/blob/master/tfx/examples/workshop/setup/setup_demo.sh . With that complete I believe I can start writing the tutorial. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51?email_source=notifications&email_token=AAY6AZVJWEBVH3G4WGFVPCDP4YOG7A5CNFSM4HH3TOU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY2HU6Q#issuecomment-506755706>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAY6AZUR54DRGM7HH3RQGDLP4YOG7ANCNFSM4HH3TOUQ .
-- Cheers, Zhitao Li

Yes, that is the Initial Dockerfile

FROM tensorflow/tfx

# write that in a file later
RUN pip install --no-cache \
    tensorflow==1.13.1 \
    tfx==0.13.0 \
    flask==1.0.3 \
    apache-airflow 

# removing base script
RUN rm -rf * 

WORKDIR /home

RUN airflow initdb

COPY setup.sh . 

ENTRYPOINT bash -c "./setup.sh"

setup.sh script:

#!/bin/bash
airflow webserver -p 8080  &
airflow scheduler  

from tfx.

Recently a new release of Flask broke the Dockerfile above. I already did an update on the post with the repair

from tfx.

Great! Can you post the Dockerfile for the initial version? IMHI, high level things we need from the default setup script should be: - install jupyter notebook; - install notebook related dependencies; - start notebook on a different port and expose that port to browser; - make sure pipeline data and sqlite DB file is on a volume (which can be mounted to the container); The part of copying python files around should be more or less optional as long as you can get Airflow running.
…
On Fri, Jun 28, 2019 at 7:32 AM Felipe Borges @.***> wrote: I already create a initial version for that image https://hub.docker.com/r/bumbleblo/tfx (that version is using sqlite). Now I'm trying to understand what is important in default setup script of the workshop https://github.com/tensorflow/tfx/blob/master/tfx/examples/workshop/setup/setup_demo.sh . With that complete I believe I can start writing the tutorial. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51?email_source=notifications&email_token=AAY6AZVJWEBVH3G4WGFVPCDP4YOG7A5CNFSM4HH3TOU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY2HU6Q#issuecomment-506755706>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAY6AZUR54DRGM7HH3RQGDLP4YOG7ANCNFSM4HH3TOUQ .
-- Cheers, Zhitao Li

I pushed a new version of the image with that volumes (one for user data and other with airflow data). Also I already update the dockerhub page description with a new usage command. All files used for build that image it's here.

from tfx.

@Bumbleblo Thanks. I'll try the image out later today and provide feedback. Since I cannot comment on your branch, consider send a PR so I can comment?

@rcrowe-google

from tfx.

For sure, I'll open a pull request for that docker files and link here.

from tfx.

Hey @zhitaoli, can you send me your feedback about the latest version of tutorial image ? I made some changes to python 3 and that new version is based in the Dockerfile on master branch.

from tfx.

Any update on this? I'm building a similar image right now and could use this.

from tfx.

Closing this as per the comment trace above.Please feel free to reopen if required.Thanks.

from tfx.