Comments (37)
I won't be giving updates on the Apache Beam timeline, please refer to the Apache Beam project. Apart from Beam dependencies, we have starting running our tests against Python 2 and 3, so we are on track to support Python 3 soon after Apache Beam does.
from transform.
@zoyahav, @KesterTong Should we close this now?
from transform.
@luke-zhu did some excellent work on Python 3 type inference for BEAM (see apache/beam#4183 ) We still have more steps (see https://issues.apache.org/jira/browse/BEAM-2784 ), I have an open PR for a somewhat hacky solution to the proto issue ( apache/beam#4077 ) & more folks are always welcome to join in :)
from transform.
Is this still an issue ?
Yes it is and should stay until https://issues.apache.org/jira/browse/BEAM-1251 is fixed.
from transform.
The situation is unchanged: we are blocked on https://issues.apache.org/jira/browse/BEAM-1373. Once this is resolved there are no major blockers to Python 3 support in tf.Transform
from transform.
@rafiqhasan, Python 3 support is a priority for tf.Transform, but we are blocked by Beam (please see Beam for updates on their Python 3 status). To re-iterate our strategy from my comment above:
we have starting running our tests against Python 2 and 3, so we are on track to support Python 3 soon after Apache Beam does.
from transform.
TFT 0.13 has support for Python 3.5 [1].
Other version of Python (like 3.6 and 3.7) will follow in future releases (after Beam officially supports them).
[1] https://github.com/tensorflow/transform/releases/tag/v0.13.0
from transform.
One of our dependencies, google-cloud-dataflow
, does not currently support Python 3. There is a issue in their repo tracking this. As soon as that issue is resolved, we should be able to add Python 3 support for tf.Transform very quickly.
from transform.
ETA?
from transform.
Don't have an ETA at this time, but the Beam developers are actively investigating this issue. In the meantime we will ensure our code is Python 3 ready so that the only remaining issues are the dependencies.
from transform.
Any update on this?
from transform.
We are still blocked on Beam Python, see https://issues.apache.org/jira/browse/BEAM-1373 for more details
from transform.
Any ETA? It would be helpful if the error message stated this explicitly. The error message that I got was:
Collecting tensorflow-transform
Could not find a version that satisfies the requirement tensorflow-transform (from versions: )
No matching distribution found for tensorflow-transform
from transform.
At this point we can't give an ETA for this as it's blocked by https://issues.apache.org/jira/browse/BEAM-1373. I know that issue is being discussed but don't have a timeline for it.
Regarding the error message, I will look into this.
from transform.
An update: thanks to @cclauss we have #24, #25, #26, #27, #28 which fixed some Python 3 issues in the tf.Transform codebase, though the Beam dependency issue remains. @cclauss does that fix all the Python 3 issues in tf.Transform code that your'e aware of?
from transform.
@KesterTong My technique ($ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics) allows me to find and propose fixes to Python 3 porting issues. We have fixed all the ones that I have found in the tf.Transform codebase but there may be other issues that are not found using my technique.
from transform.
As of Tensorflow Transform Release 0.3.1 we offer a Source distribution as well (as opposed to just a WHL).
Going forward (ie for anything >= 0.4), we plan to offer only a Source distribution (and no longer offer a WHL) since Tensorflow Transform is a pure python package.
However as @KesterTong pointed out we still depend on Python 3 support from the Beam SDK (captured by https://issues.apache.org/jira/browse/BEAM-1373).
Beyond the Beam SDK, for folks that want to run Tensorflow Transform on Google Cloud Dataflow, I believe resolution of https://issues.apache.org/jira/browse/BEAM-1251 would also be needed.
/cc @aaltay (in case he can confirm and in case he has any noteworthy updates on the above Jiras).
from transform.
The previous comment is accurate. @holdenk is making steady progress on the former issue.
from transform.
any update on this ?
from transform.
- 1 Python 3 Syntax Error in tensorflow/transform
- 14 Python 3 Syntax Errors in apache/beam
Additional work is probably required beyond fixing these Syntax Errors.
flake8 testing of https://github.com/tensorflow/transform on Python 3.6.4
$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
./tensorflow_transform/beam/impl_test.py:1916:37: E999 SyntaxError: invalid syntax
cmp=lambda (xi, xv), (yi, yv): cmp(xv, yv))
^
flake8 testing of https://github.com/apache/beam on Python 3.6.4
$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
[ ... ]
14 E999 SyntaxError: invalid syntax
115 F821 undefined name 'unicode'
129
from transform.
Hey @holdenk, is https://issues.apache.org/jira/browse/BEAM-1251 still the best issue to follow for Python 3 support?
Our interest is using Dataflow via Apache Airflow. I'm trying to find a workaround for that in the short term, like interacting with Dataflow inside a dedicated Py 2 virtual env or Docker container.
from transform.
I'd say https://issues.apache.org/jira/browse/BEAM-1251 is probably still the right place.
from transform.
What the progress and remaining blockers for tft on python3?
from transform.
Or possibly https://issues.apache.org/jira/browse/BEAM-1251
from transform.
Any updates on this?
from transform.
You might add one more line to your test cases...
flake8 testing of https://github.com/tensorflow/transform on Python 3.7.0
$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
./tensorflow_transform/beam/impl_test.py:2497:14: F821 undefined name 'cmp'
return cmp(a[1], b[1])
^
1 F821 undefined name 'cmp'
1
The builtin cmp() was removed in Python 3. See #84
from transform.
Strange, we are in 2018 with Python 3.x and this still doesn't seem to be a priority. The issue has been open since 2017 and we are soon approaching Python 2.7 end of support.
from transform.
@KesterTong , highly appreciate your quick comment on it. We are building ML pipelines on Tensorflow and currently all transformations are being managed via our custom transformers that we have attached as functions in the model and values have been saved as function params.
I am pretty sure that most of the real world ML TF projects are already on Python 3+ and involve feature engineering transformations, so how is this unavailability of Py 3 support being tackled in those projects ? Is no one else using TFT in production with Py 3 ? Are there any alternatives ?
from transform.
@rafiqhasan One example of users who are able to user TFT in production, is existing users of Apache Beam or Google Cloud Dataflow, since they would already by using Python 2. We will be adding Python 3 support once https://issues.apache.org/jira/browse/BEAM-1251 is fixed.
from transform.
Is this still an issue ?
from transform.
is this still an issue?
from transform.
Yes. https://pypi.org/project/tensorflow-transform/#data still says that tf-transform only support legacy Python with just one year left until it’s end-of-life... http://pythonclock.org
from transform.
Google will probably have to get involved a lot more closely in the Beam project to address this. Google-specific issues with DataflowPythonSDK, gcp-variant-transforms and datalab also look to be affected, motivating a need for closer involvement.
from transform.
There’s been some good progress, see https://issues.apache.org/jira/browse/BEAM-1251 , but it does seem we are still not there yet.
from transform.
too long time ~
from transform.
#101 seems to be related.
from transform.
This library currently supports python 2.7, 3.5-3.7 (since tf.transform 0.14).
If there are any issues with tensorflow_transform>=0.14 and python 3 please open a new issue, I think it's safe to close this one.
from transform.
Related Issues (20)
- Expose type information HOT 1
- Segmentation Fault on tft.compute_and_apply_vocabulary HOT 8
- Release new version that's compatible with `tfx[kfp]==1.10.0` HOT 7
- Can't install due to dependency on numpy HOT 5
- Could tft support embedding transform function? HOT 11
- Python 3.10 support HOT 19
- TypeError: an integer is required [while running 'AnalyzeAndTransformDataset/TransformDataset/InstrumentInputBytes[Transform] HOT 6
- Allow Identity "Transformation" HOT 4
- Beam AnalyzeAndTransformDataset runs expensive transformation _InstanceDictInputToTFXIOInput Twice HOT 3
- Install fails on Apple M2 Ventura HOT 3
- Apple Silicon support for tensorflow-transform not available as tfx-bsl does not support it? HOT 3
- Raise pyarrow upper bound? HOT 12
- apart from the batch dimension, all dimensions must have known size [while running 'AnalyzeAndTransformDataset/AnalyzeDataset/CreateSavedModel[tf_v2_only]/CreateSavedModel'] HOT 2
- Transform component and tf.function HOT 4
- Transform graph returns an empty dictionary at serve_tf_examples_fn function HOT 4
- Graph error when using TFHub Universal-sentence-encoder model HOT 2
- Wheel missing py.typed HOT 2
- universal sentence encoder batch pipeline failing HOT 3
- tfx.components.Transform returns invalid results HOT 1
- TFX Transform layer returns dict with missing keys HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transform.