Tensorflow is so immensely frustrating.
I've been trying to build a Dataset from a generator. This fails completely:
def get_random_tensors(count):
for _ in range(count):
yield tf.random_uniform((30, 1))
dset = tf.data.Dataset.from_generator(lambda: get_random_tensors(5000), tf.float32)
but this works correctly:
def get_random_tensors(count):
for _ in range(count):
yield np.random.uniform(-1.0, 1.0, (30,1))
dset = tf.data.Dataset.from_generator(get_random_tensors(5000), tf.float32)
I should have known I need to produce an actual numpy value immediately rather than a tensor, I guess. Though why I have to wrap my generator expression in a lambda, I have no goddamned idea. But my bigger complaint is, those two errors where super hard to figure out, because the error reports I got look like this:
Traceback (most recent call last):
File "<ipython-input-49-1b94387c5bca>", line 1, in <module>
runfile('/home/ /tensorflow_learning/dset race.py', wdir='/home/ /tensorflow_learning')
File "/home/ /tensorflow_learning/tflowv/lib64/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 705, in runfile
execfile(filename, namespace)
File "/home/ /tensorflow_learning/tflowv/lib64/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "/home/ /tensorflow_learning/dset race.py", line 32, in <module>
sess.run(x_dset)
File "/home/ /tensorflow_learning/tflowv/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/ /tensorflow_learning/tflowv/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ /tensorflow_learning/tflowv/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/ /tensorflow_learning/tflowv/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
InvalidArgumentError: ValueError: setting an array element with a sequence.
Traceback (most recent call last):
File "/home/ /tensorflow_learning/tflowv/lib64/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 157, in __call__
ret = func(*args)
File "/home/ /tensorflow_learning/tflowv/lib64/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 391, in generator_py_func
nest.flatten_up_to(output_types, values), flattened_types)
File "/home/ /tensorflow_learning/tflowv/lib64/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 390, in <listcomp>
for ret, dtype in zip(
File "/home/ /tensorflow_learning/tflowv/lib64/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 124, in _convert
result = np.asarray(value, dtype=dtype, order="C")
File "/home/ /tensorflow_learning/tflowv/lib64/python3.6/site-packages/numpy/core/numeric.py", line 492, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
[[Node: PyFunc = PyFunc[Tin=[DT_INT64], Tout=[DT_FLOAT], token="pyfunc_73"](arg0)]]
[[Node: IteratorGetNext_41 = IteratorGetNext[output_shapes=[<unknown>], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](Iterator_37)]]
I've learned a lesson about building software from Tesorflow today: test your inputs and throw your errors early, so you can give the user useful information about what they did, instead of
shitting up like this.