pip install does not work on Python 3 about transform HOT 37 CLOSED

I won't be giving updates on the Apache Beam timeline, please refer to the Apache Beam project. Apart from Beam dependencies, we have starting running our tests against Python 2 and 3, so we are on track to support Python 3 soon after Apache Beam does.

from transform.

@zoyahav, @KesterTong Should we close this now?

from transform.

@luke-zhu did some excellent work on Python 3 type inference for BEAM (see apache/beam#4183 ) We still have more steps (see https://issues.apache.org/jira/browse/BEAM-2784 ), I have an open PR for a somewhat hacky solution to the proto issue ( apache/beam#4077 ) & more folks are always welcome to join in :)

from transform.

Is this still an issue ?

Yes it is and should stay until https://issues.apache.org/jira/browse/BEAM-1251 is fixed.

from transform.

The situation is unchanged: we are blocked on https://issues.apache.org/jira/browse/BEAM-1373. Once this is resolved there are no major blockers to Python 3 support in tf.Transform

from transform.

@rafiqhasan, Python 3 support is a priority for tf.Transform, but we are blocked by Beam (please see Beam for updates on their Python 3 status). To re-iterate our strategy from my comment above:

we have starting running our tests against Python 2 and 3, so we are on track to support Python 3 soon after Apache Beam does.

from transform.

TFT 0.13 has support for Python 3.5 [1].

Other version of Python (like 3.6 and 3.7) will follow in future releases (after Beam officially supports them).

[1] https://github.com/tensorflow/transform/releases/tag/v0.13.0

from transform.

One of our dependencies, google-cloud-dataflow, does not currently support Python 3. There is a issue in their repo tracking this. As soon as that issue is resolved, we should be able to add Python 3 support for tf.Transform very quickly.

from transform.

ETA?

from transform.

Don't have an ETA at this time, but the Beam developers are actively investigating this issue. In the meantime we will ensure our code is Python 3 ready so that the only remaining issues are the dependencies.

from transform.

Any update on this?

from transform.

We are still blocked on Beam Python, see https://issues.apache.org/jira/browse/BEAM-1373 for more details

from transform.

Any ETA? It would be helpful if the error message stated this explicitly. The error message that I got was:

Collecting tensorflow-transform
Could not find a version that satisfies the requirement tensorflow-transform (from versions: )
No matching distribution found for tensorflow-transform

from transform.

At this point we can't give an ETA for this as it's blocked by https://issues.apache.org/jira/browse/BEAM-1373. I know that issue is being discussed but don't have a timeline for it.

Regarding the error message, I will look into this.

from transform.

An update: thanks to @cclauss we have #24, #25, #26, #27, #28 which fixed some Python 3 issues in the tf.Transform codebase, though the Beam dependency issue remains. @cclauss does that fix all the Python 3 issues in tf.Transform code that your'e aware of?

from transform.

@KesterTong My technique ($ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics) allows me to find and propose fixes to Python 3 porting issues. We have fixed all the ones that I have found in the tf.Transform codebase but there may be other issues that are not found using my technique.

from transform.

As of Tensorflow Transform Release 0.3.1 we offer a Source distribution as well (as opposed to just a WHL).

Going forward (ie for anything >= 0.4), we plan to offer only a Source distribution (and no longer offer a WHL) since Tensorflow Transform is a pure python package.

However as @KesterTong pointed out we still depend on Python 3 support from the Beam SDK (captured by https://issues.apache.org/jira/browse/BEAM-1373).

Beyond the Beam SDK, for folks that want to run Tensorflow Transform on Google Cloud Dataflow, I believe resolution of https://issues.apache.org/jira/browse/BEAM-1251 would also be needed.

/cc @aaltay (in case he can confirm and in case he has any noteworthy updates on the above Jiras).

from transform.

The previous comment is accurate. @holdenk is making steady progress on the former issue.

from transform.

any update on this ?

from transform.

• 1 Python 3 Syntax Error in tensorflow/transform
• 14 Python 3 Syntax Errors in apache/beam
  Additional work is probably required beyond fixing these Syntax Errors.

flake8 testing of https://github.com/tensorflow/transform on Python 3.6.4

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./tensorflow_transform/beam/impl_test.py:1916:37: E999 SyntaxError: invalid syntax
                         cmp=lambda (xi, xv), (yi, yv): cmp(xv, yv))
                                    ^

flake8 testing of https://github.com/apache/beam on Python 3.6.4

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

    [ ... ]

14    E999 SyntaxError: invalid syntax
115   F821 undefined name 'unicode'
129

from transform.

Hey @holdenk, is https://issues.apache.org/jira/browse/BEAM-1251 still the best issue to follow for Python 3 support?

Our interest is using Dataflow via Apache Airflow. I'm trying to find a workaround for that in the short term, like interacting with Dataflow inside a dedicated Py 2 virtual env or Docker container.

from transform.

I'd say https://issues.apache.org/jira/browse/BEAM-1251 is probably still the right place.

from transform.

What the progress and remaining blockers for tft on python3?

from transform.

Or possibly https://issues.apache.org/jira/browse/BEAM-1251

from transform.

Any updates on this?

from transform.

You might add one more line to your test cases...

flake8 testing of https://github.com/tensorflow/transform on Python 3.7.0

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./tensorflow_transform/beam/impl_test.py:2497:14: F821 undefined name 'cmp'
      return cmp(a[1], b[1])
             ^
1     F821 undefined name 'cmp'
1

The builtin cmp() was removed in Python 3. See #84

from transform.

Strange, we are in 2018 with Python 3.x and this still doesn't seem to be a priority. The issue has been open since 2017 and we are soon approaching Python 2.7 end of support.

from transform.

@KesterTong , highly appreciate your quick comment on it. We are building ML pipelines on Tensorflow and currently all transformations are being managed via our custom transformers that we have attached as functions in the model and values have been saved as function params.

I am pretty sure that most of the real world ML TF projects are already on Python 3+ and involve feature engineering transformations, so how is this unavailability of Py 3 support being tackled in those projects ? Is no one else using TFT in production with Py 3 ? Are there any alternatives ?

from transform.

@rafiqhasan One example of users who are able to user TFT in production, is existing users of Apache Beam or Google Cloud Dataflow, since they would already by using Python 2. We will be adding Python 3 support once https://issues.apache.org/jira/browse/BEAM-1251 is fixed.

from transform.

Is this still an issue ?

from transform.

is this still an issue?

from transform.

Yes. https://pypi.org/project/tensorflow-transform/#data still says that tf-transform only support legacy Python with just one year left until it’s end-of-life... http://pythonclock.org

from transform.

Google will probably have to get involved a lot more closely in the Beam project to address this. Google-specific issues with DataflowPythonSDK, gcp-variant-transforms and datalab also look to be affected, motivating a need for closer involvement.

from transform.

There’s been some good progress, see https://issues.apache.org/jira/browse/BEAM-1251 , but it does seem we are still not there yet.

from transform.

too long time ~

from transform.

#101 seems to be related.

from transform.

This library currently supports python 2.7, 3.5-3.7 (since tf.transform 0.14).
If there are any issues with tensorflow_transform>=0.14 and python 3 please open a new issue, I think it's safe to close this one.

from transform.