Addding Support for Image Dataset about tfx HOT 7 CLOSED

Hi,

We are creating a import example gen for directly import external tfrecord files (with tf example in it) and create train and eval split for downstream component

a custom component is also doable, we are working on creating an example for people to follow

from tfx.

TFRecords + TFDV @ https://github.com/aaronlelevier/apache_beam_examples/blob/3afa0e411676c22734e7a2189602eda611938597/notebooks/MNIST-all-data-and-partition-fn.ipynb

from tfx.

@1025KB I am eagerly waiting for the example. The preprocessing_fn reads the content of the tfrecord files as sparse tensors. I am unable to convert them to fixedlen features and reshape the image data.

from tfx.

preprocessing_fn is for Transform, so you are able to custom ExampleGen to convert your input source to tf.Examples?

from tfx.

if you are talking about custom ExampleGen for reading data file format that we don't support yet, check this

you can also do your own conversion before TFX, and use ImportExampleGen to directly import the tf.Examples, check this

if your problem is Transform related, let me know and I can find you a Transform expert

from tfx.

@thak123 There are some pending work in the Schemagen to enable our support for fixedlen features. We will keep this thread updated.

from tfx.

preprocessing_fn is for Transform, so you are able to custom ExampleGen to convert your input source to tf.Examples?

Thanks for showing me the right direction. I am able to train the model now.

from tfx.