tf.data.experimental.OptimizationOptions

TensorFlow 1 version

View source on GitHub

A parallel version of the Dataset.interleave() transformation. (deprecated)

View aliases

Compat aliases for migration

See Migration guide for more details.

tf.compat.v1.data.experimental.parallel_interleave

tf.data.experimental.parallel_interleave(
    map_func, cycle_length, block_length=1, sloppy=False,
    buffer_output_elements=None, prefetch_input_elements=None
)

parallel_interleave() maps map_func across its input to produce nested datasets, and outputs their elements interleaved. Unlike tf.data.Dataset.interleave, it gets elements from cycle_length nested datasets in parallel, which increases the throughput, especially in the presence of stragglers. Furthermore, the sloppy argument can be used to improve performance, by relaxing the requirement that the outputs are produced in a deterministic order, and allowing the implementation to skip over nested datasets whose elements are not readily available when requested.

Example usage:

# Preprocess 4 files concurrently.
filenames = tf.data.Dataset.list_files("/path/to/data/train*.tfrecords")
dataset = filenames.apply(
    tf.data.experimental.parallel_interleave(
        lambda filename: tf.data.TFRecordDataset(filename),
        cycle_length=4))

Args
`map_func`	A function mapping a nested structure of tensors to a `Dataset`.
`cycle_length`	The number of input `Dataset`s to interleave from in parallel.
`block_length`	The number of consecutive elements to pull from an input `Dataset` before advancing to the next input `Dataset`.
`sloppy`	A boolean controlling whether determinism should be traded for performance by allowing elements to be produced out of order. If `sloppy` is `None`, the `tf.data.Options.experimental_deterministic` dataset option (`True` by default) is used to decide whether to enforce a deterministic order.
`buffer_output_elements`	The number of elements each iterator being interleaved should buffer (similar to the `.prefetch()` transformation for each interleaved iterator).
`prefetch_input_elements`	The number of input elements to transform to iterators before they are needed for interleaving.

Returns
A `Dataset` transformation function, which can be passed to `tf.data.Dataset.apply`.

TensorFlow

tf

tf.audio

tf.autograph

tf.bitwise

tf.compat

tf.config

tf.data

tf.debugging

tf.distribute

tf.dtypes

tf.errors

tf.estimator

tf.experimental

tf.feature_column

tf.graph_util

tf.image

tf.initializers

tf.io

tf.keras

tf.linalg

tf.lite

tf.lookup

tf.losses

tf.math

tf.metrics

tf.nest

tf.nn

tf.optimizers

tf.quantization

tf.queue

tf.ragged

tf.random

tf.raw_ops

tf.saved_model

tf.sets

tf.signal

tf.sparse

tf.strings

tf.summary

tf.sysconfig

tf.test

tf.tpu

tf.train

tf.version

tf.xla

tf.data / experimental / experimental.OptimizationOptions

View aliases

Example usage:

Args

Returns