Sped up the creation and filling of datasets #30
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sped up the creation and filling of datasets (in Dataset.py), making the RSG over an order of magnitude faster for large datasets.
This requires the datavault server to have a setting "shape" which returns the shape of the dataset.
Instead of initializing an empty numpy array and appending successive data grabs from the datavault, we now use the datavault's shape setting to preallocate a numpy array and fill it with data.
Additionally, the size of each data grab has been increased from 100 to 1000 since most of the overhead arises from filling the dataset.