Converting a large TDMS file (> 100 GB) to HDF5

[rgklager]

Hello,
Can I convert a large TDMS file (> 100 GB) to an HDF5 file in a streaming way: reading the TDMS file chunk by chunk and writing the HDF5 file chunk by chunk, to prevent insufficient memory errors?

[adamreeve]

Hi @rgklager, how are you currently converting TDMS to HDF5. Have you tried this and you're getting errors?

Assuming your TDMS file is split into multiple chunks, then something like this code will read the data one chunk at a time:

with nptdms.TdmsFile.open(file_path) as tdms_file:
    hdf_file = tdms_file.as_hdf(hdf_path)

I'm not very familiar with h5py though. We write to slices of a dataset at a time, and my understanding is this should write to disk without needing to keep the full array in memory:

npTDMS/nptdms/export/hdf_export.py

Line 77 in d2cddeb

datasets[channel.path][offset:end] = _hdf_array(channel, channel_chunk[:])

[rgklager]

Thanks Adam, using open instead of read solved my issue. And yes, you are right, h5py indeed writes to file without the need to keep it in memory.
After some digging I found out that the chunk size in my TDMS file of about 100 GB was only 1 sample. That caused a near infinite processing time for the HDF5 conversion, which clouded my analysis. After adapting my LabVIEW VI such that the chunk size of the TDMS file is 5 MS, the conversion from TDMS to HDF5 with npTDMS works very smooth.