Error

[alexiokay]

I'm trying to run import_bag.py and I keep getting this;

[==========] 100% 1.1s | XML total: 350,465 | per second: 330,633
2024-11-23 12:34:35.576936 ready: parse XML OpenbareRuimte | 1.1s | XML total: 350,465
2024-11-23 12:34:35.600466 start: parse Nummeraanduiding
2024-11-23 12:34:35.600573 unzip temp/9999NUM08112024.zip
2024-11-23 12:35:39.934371 convert XML files to SQLite
[ ] 0% 18s | XML total: 10,000 | per second: 535 concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/root/miniconda3/envs/fat_script/lib/python3.13/concurrent/futures/process.py", line 253, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/root/development/metaqdata script/bag_temp2/bag/bag_parser.py", line 135, in parse_xml_file
save_function(db_row)
~~~~~~~~~~~~~^^^^^^^^
File "/root/development/metaqdata script/bag_temp2/database_sqlite/database_sqlite.py", line 111, in save_nummer
self.connection.execute(
~~~~~~~~~~~~~~~~~~~~~~~^
"""REPLACE INTO nummers (id, postcode, huisnummer, huisletter, toevoeging, woonplaats_id,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<5 lines>...
data["einddatum_geldigheid"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
sqlite3.OperationalError: database is locked
"""

The above exception was the direct cause of the following exception:

[digitaldutch]

Looks like the database is open in another app. Or a hanging Python script. Try again after rebooting your pc

[alexiokay]

I tried in ubuntu, I tried on windows, tried rebooting or killing processes. Nothing worked. Fighting 2 day with it.

[alexiokay]

Also.. You recommend python 3.13 to be used, but on windows psycopg2 is not updated yet to work with python 3.13. Would be nice to include it in readme.

[alexiokay]

I used python3.12 on windows and it started converting XML for some time and the same happened. It won't even start in windows wsl.

[digitaldutch]

I just run it on my Windows 11 PC without any problems. I will try it in another environment when I find time. Could be that some locking of the SQLite when writing data is needed as the parsing is heavily using parallel processing.

You recommend python 3.13 to be used, but on windows psycopg2 is not updated yet to work with python 3.13.

The project does not use psycopg2 so i don't fully understand this remark

[robinfaber]

I'm having the same issue when turning the multiprocessing on. Because of multiprocessing, multiple processes are trying to write to the sqlite db at the same time. However, sqlite doesn't allow concurrent writing so it's locked for the other processes running at the same time.

Edit: great work though!! 🥇