404 response when using API key

[ZaneEme]

When I add my NIST API key to ~./cvexplore/.env as described in the documentation I get the following errors when populating my database:

(env) testVM@cvesearch:~/cvexplore/cve-search$ ./sbin/db_updater.py -f -c
2025-01-03 09:42:48,205 - CveXplore.main - INFO     - Using mongodb as datasource, connection details: None
2025-01-03 09:42:48,215 - CveXplore.main - INFO     - Failed to load Celery TaskHandler; continuing without the optional Celery backend.
2025-01-03 09:42:48,216 - CveXplore.main - INFO     - Initialized CveXplore version: 0.3.35
2025-01-03 09:42:48,216 - DBUpdater - INFO     - ==========================
2025-01-03 09:42:48,217 - DBUpdater - INFO     - Repopulate
2025-01-03 09:42:48,217 - DBUpdater - INFO     - Fri 03 January 2025 16:42
2025-01-03 09:42:48,217 - DBUpdater - INFO     - ==========================
2025-01-03 09:42:48,217 - DBUpdater - INFO     - Dropping metadata: cpeother
2025-01-03 09:42:48,218 - DBUpdater - INFO     - Dropping metadata: mgmt_whitelist
2025-01-03 09:42:48,219 - DBUpdater - INFO     - Dropping metadata: mgmt_blacklist
2025-01-03 09:42:48,220 - DBUpdater - INFO     - Dropping metadata: info
2025-01-03 09:42:48,224 - DBUpdater - INFO     - Dropping metadata: schema
2025-01-03 09:42:48,225 - DBUpdater - INFO     - Starting initial import...
2025-01-03 09:42:48,225 - CveXplore.core.database_maintenance.main_updater - INFO     - Starting Database initialization....
2025-01-03 09:42:48,235 - CveXplore.core.nvd_nist.nvd_nist_api - INFO     - NVD NIST API Key found!
2025-01-03 09:42:48,236 - CveXplore.core.database_maintenance.sources_process - INFO     - CPE Database population started
2025-01-03 09:42:48,239 - CveXplore.core.database_maintenance.sources_process - INFO     - Starting download...
2025-01-03 09:42:48,239 - CveXplore.core.nvd_nist.nvd_nist_api - INFO     - Getting count for datasource: 2
2025-01-03 09:42:48,600 - CveXplore.core.nvd_nist.nvd_nist_api - ERROR    - Unexpected response type: <Response [404]>
2025-01-03 09:42:48,600 - CveXplore.core.nvd_nist.nvd_nist_api - ERROR    - Data retrieval error: {'resultsPerPage': 1}
Traceback (most recent call last):
  File "/home/testVM/cvexplore/cve-search/./sbin/db_updater.py", line 296, in <module>
    main(args)
  File "/home/testVM/cvexplore/cve-search/./sbin/db_updater.py", line 74, in main
    cvex.database.initialize()
  File "/home/testVM/cvexplore/env/lib/python3.12/site-packages/CveXplore/core/database_maintenance/main_updater.py", line 198, in initialize
    cpe_pop.populate()
  File "/home/testVM/cvexplore/env/lib/python3.12/site-packages/CveXplore/core/database_maintenance/sources_process.py", line 270, in populate
    self.process_downloads()
  File "/home/testVM/cvexplore/env/lib/python3.12/site-packages/CveXplore/core/database_maintenance/sources_process.py", line 120, in process_downloads
    total_results = self.api_handler.get_count(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/testVM/cvexplore/env/lib/python3.12/site-packages/CveXplore/core/nvd_nist/nvd_nist_api.py", line 280, in get_count
    raise ApiDataRetrievalFailed(resource)
CveXplore.errors.apis.ApiDataRetrievalFailed: {'resultsPerPage': 1}

I've tried this using two separate API keys I've generated and gotten the same issue. If I comment out the API key line, as is default, the database initialization seems to start fine, and records are downloaded as expected. I'm not sure if the request format for the API key has changed recently? Or I may just be doing something incorrectly upon setup.

[oh2fih]

The error CveXplore.errors.apis.ApiDataRetrievalFailed: {'resultsPerPage': 1} is a bit confusing as it simply contains a hard-coded dictionary (in CveXplore/core/nvd_nist/nvd_nist_api.py):

261:        resource = {"resultsPerPage": 1}

275:            if not isinstance(ret_data, Response):
276:                self.logger.debug(f"API response data: {ret_data}")
277:                return ret_data["totalResults"]
278:            else:
279:                self.logger.error(f"Unexpected response type: {ret_data}")
280:                raise ApiDataRetrievalFailed(resource)

287:         except ApiDataRetrievalFailed as data_err:
288:            self.logger.error(f"Data retrieval error: {data_err}")

The log does not show which URL it tried to GET, but the meaningful error is that you got a 404 Not Found response:

CveXplore.core.nvd_nist.nvd_nist_api - ERROR    - Unexpected response type: <Response [404]>

I have performed a full repopulation test on January 5th with an API key without getting this error. Could you try again? If it still fails, you could raise the logging level to DEBUG and post some more detailed information.