Code

Importing v3#482

Code

Merged

northdpole merged 91 commits intomainOWASP/OpenCRE:mainfrom importing-v3OWASP/OpenCRE:importing-v3Copy head branch name to clipboardJul 6, 2024

+7,168-25,512

Commits

Commits on Jun 27, 2024

• logging improvement

  northdpole

  committed
  0cb4fd7

  Copy full SHA for 0cb4fd7

• add design doc

  northdpole

  committed
  1d9ed6f

  Copy full SHA for 1d9ed6f

• spreadsheet parser refactored

  northdpole

  committed
  eb3102f

  Copy full SHA for eb3102f

• parallelize main

  northdpole

  committed
  db3b273

  Copy full SHA for db3b273

• parallelise importing of standards from a spreadsheet and generation of related data

  northdpole

  committed
  d1ddecb

  Copy full SHA for d1ddecb

• add progress bars

  northdpole

  committed
  cc1458c

  Copy full SHA for cc1458c

• return list of imported resources

  northdpole

  committed
  423daa4

  Copy full SHA for 423daa4

• initial base parser implementation

  northdpole

  committed
  a67c417

  Copy full SHA for a67c417

• move parsers to own dir and migrate cwe, dsomm and iso27k to the new parser interface

  northdpole

  committed
  75d4bda

  Copy full SHA for 75d4bda

• migrate parsers

  northdpole

  committed
  9e9774f

  Copy full SHA for 9e9774f

• make main call parser

  northdpole

  committed
  8d3afbb

  Copy full SHA for 8d3afbb

• add embeddings in the document object allow prasers to generate and fill embeddings of each document, decouple parsers from main methods to avoid circular imports, mode embeddings database storage …

  Show description for 187a013
  northdpole

  committed
  187a013

  Copy full SHA for 187a013

• add default value for embeddings object in dataclass

  northdpole

  committed
  e94bd17

  Copy full SHA for e94bd17

• sort out imports

  northdpole

  committed
  ddd4008

  Copy full SHA for ddd4008

• cwe parser test

  northdpole

  committed
  630c69a

  Copy full SHA for 630c69a

• dsomm parser tests

  northdpole

  committed
  9ad3627

  Copy full SHA for 9ad3627

• secure headers + tests

  northdpole

  committed
  0639649

  Copy full SHA for 0639649

• cloud native security controls parser +tests

  northdpole

  committed
  c1bfc64

  Copy full SHA for c1bfc64

• ccmv4

  northdpole

  committed
  1e7e396

  Copy full SHA for 1e7e396

• rename method

  northdpole

  committed
  e1fe452

  Copy full SHA for e1fe452

• juiceshop + tests

  northdpole

  committed
  68ac798

  Copy full SHA for 68ac798

• mega lint

  northdpole

  committed
  d1f5be2

  Copy full SHA for d1f5be2

• rm commented code

  northdpole

  committed
  d38645e

  Copy full SHA for d38645e

• fix docs equality test

  northdpole

  committed
  0102356

  Copy full SHA for 0102356

• change external project parsers to return dict of 'resourcename':<resource entries>

  northdpole

  committed
  4f0bf55

  Copy full SHA for 4f0bf55

• fix web main test related to CSV returning inconsistency

  northdpole

  committed
  e31ea15

  Copy full SHA for e31ea15

• fix nit on gap analysis enqueue job

  northdpole

  committed
  debbe50

  Copy full SHA for debbe50

• operational changes to make mass importing easier

  northdpole

  committed
  6c6172e

  Copy full SHA for 6c6172e

• fix spreadsheet importing bugs, add validation to dataclasses

  northdpole

  committed
  0c29d19

  Copy full SHA for 0c29d19

• makefile improvements

  northdpole

  committed
  4dc85ce

  Copy full SHA for 4dc85ce

• add validation and fix tests

  northdpole

  committed
  2b731bb

  Copy full SHA for 2b731bb

• drop support for OSIB

  northdpole

  committed
  d358508

  Copy full SHA for d358508

• fix more tests

  northdpole

  committed
  5917fdd

  Copy full SHA for 5917fdd

• pin black to same version as superlinter and lint everything

  northdpole

  committed
  e0d86ff

  Copy full SHA for e0d86ff

• change array hash to array key so that its legible, introduce ids for nodes which is a combination of their values, partially fix gap analysis

  northdpole

  committed
  6d92982

  Copy full SHA for 6d92982

• add tests for bug where standards would only link to one cre

  northdpole

  committed
  2b20c6b

  Copy full SHA for 2b20c6b

• change neo4j standards for regular postgres standards

  northdpole

  committed
  be25860

  Copy full SHA for be25860

• adjust main to not require redis when getting standards

  northdpole

  committed
  de8e479

  Copy full SHA for de8e479

• cache key to str

  northdpole

  committed
  03602be

  Copy full SHA for 03602be

• add ability to import only the projects

  northdpole

  committed
  82326da

  Copy full SHA for 82326da

• delete all traces of node and gap analysis of node, used when reimporting standards after either structural or informational changes

  northdpole

  committed
  eb5626a

  Copy full SHA for eb5626a

• nit: rearrange argument handling on main

  northdpole

  committed
  3778930

  Copy full SHA for 3778930

• fix previously introduced cre hierarchy bug

  northdpole

  committed
  1fa2544

  Copy full SHA for 1fa2544

• move commands for regenerating DB to a new 'import-all' script

  northdpole

  committed
  bf182b1

  Copy full SHA for bf182b1

• fix embedding gen

  northdpole

  committed
  0ea58bb

  Copy full SHA for 0ea58bb

• disable iso, set port to 5001

  northdpole

  committed
  e4943ff

  Copy full SHA for e4943ff

• add 'automatically linked to' linktype and use it for low confidence mappings

  northdpole

  committed
  b10f6f5

  Copy full SHA for b10f6f5

• fix scripts, make import-projects use scripts

  northdpole

  committed
  ceef928

  Copy full SHA for ceef928

• rm cres, too large to keep around

  northdpole

  committed
  eccd29f

  Copy full SHA for eccd29f

• add migration

  northdpole

  committed
  b78ba02

  Copy full SHA for b78ba02

• since we removed cres dir, also remove export functionality

  northdpole

  committed
  90eb188

  Copy full SHA for 90eb188

• fix importing script

  northdpole

  committed
  f0bcac4

  Copy full SHA for f0bcac4

• add new link status to db

  northdpole

  committed
  1b123b8

  Copy full SHA for 1b123b8

• add import-only for external parsers and remove export and review functionality

  northdpole

  committed
  e979c79

  Copy full SHA for e979c79

• fix cwe typo

  northdpole

  committed
  229a6cf

  Copy full SHA for 229a6cf

• generate embeddings for guaranteed non-none name

  northdpole

  committed
  5140392

  Copy full SHA for 5140392

• add message on waiting jobs

  northdpole

  committed
  10f570c

  Copy full SHA for 10f570c

• improve gap analysis logging

  northdpole

  committed
  ce1af06

  Copy full SHA for ce1af06

• add ability to skip reimporting if something already exists

  northdpole

  committed
  1f0e314

  Copy full SHA for 1f0e314

• logging nit

  northdpole

  committed
  fb74978

  Copy full SHA for fb74978

• make the base parser not load the in-memory graph by default and fix the

  Show description for d93df17
  northdpole

  committed
  d93df17

  Copy full SHA for d93df17

• make loading the graph in memory optional

  Show description for e279081
  northdpole

  committed
  e279081

  Copy full SHA for e279081

• improve logging for gap analysis jobs

  northdpole

  committed
  1317af3

  Copy full SHA for 1317af3

• fix endless loop when importing and gap analysis exists

  northdpole

  committed
  a6cb654

  Copy full SHA for a6cb654

• add gap analysis relationship 'automaticallyt linked to' and throw exception when cycle detection gets called with no in memory graph

  northdpole

  committed
  b763af9

  Copy full SHA for b763af9

• paginate graph retrieval init (#491)

  Show description for 5bf0cb5
  northdpole

  committed
  5bf0cb5

  Copy full SHA for 5bf0cb5

• Macos support (#496)

  Show description for c156a87
  northdpole

  committed
  c156a87

  Copy full SHA for c156a87

• backend runs on 5001 (#495)

  Show description for f478561
  northdpole

  committed
  f478561

  Copy full SHA for f478561

• Revert "backend runs on 5001 (#495)" (#498)

  Show description for 7cac06a
  northdpole

  committed
  7cac06a

  Copy full SHA for 7cac06a

• rm version from cwe and make ccm and iso disabled parsers commented out so it does not affect coverage

  northdpole

  committed
  4d65092

  Copy full SHA for 4d65092

• add dev environment variables that do only graph importing to be used for debugging

  northdpole

  committed
  fce0d2b

  Copy full SHA for fce0d2b

• import external projects individually

  northdpole

  committed
  dc388be

  Copy full SHA for dc388be

• print less when calculating ga

  northdpole

  committed
  e5c1767

  Copy full SHA for e5c1767

• [ticket-508] Ensure autolinks appear on the CRE page (#509)

  Show description for 603cfc0
  

  dlicheva

  authored and

  northdpole

  committed
  603cfc0

  Copy full SHA for 603cfc0

• fix broken rebase

  northdpole

  committed
  1bd363d

  Copy full SHA for 1bd363d

• add ability to run cre as a container and sync local cre with upstream

  northdpole

  committed
  41b6e84

  Copy full SHA for 41b6e84

• cleanup unused spreadsheet parser methods

  northdpole

  committed
  056cd77

  Copy full SHA for 056cd77

• fix e2e tests

  northdpole

  committed
  c442f4f

  Copy full SHA for c442f4f

• pin black to same version as superlinter

  northdpole

  committed
  5f6951d

  Copy full SHA for 5f6951d

• pin node version in github e2e

  northdpole

  committed
  e2ee9b9

  Copy full SHA for e2ee9b9

• upgrade actions node

  northdpole

  committed
  85bf4e9

  Copy full SHA for 85bf4e9

• ga query test

  northdpole

  committed
  5b801f0

  Copy full SHA for 5b801f0

• add ability to external project parsers to skip gap analysis and embedding calculation

  northdpole

  committed
  ed27b55

  Copy full SHA for ed27b55

• nit: logging

  northdpole

  committed
  9388f84

  Copy full SHA for 9388f84

• fix gap analysis bug where we wouldn't remove calculated ga from waiting list

  northdpole

  committed
  404ac9b

  Copy full SHA for 404ac9b

• move ga preloading into script

  northdpole

  committed
  702b336

  Copy full SHA for 702b336

• fix error where standards would get preloaded twice

  northdpole

  committed
  85c2fd0

  Copy full SHA for 85c2fd0

• add explorer to header in staging

  northdpole

  committed
  864fedd

  Copy full SHA for 864fedd

Commits on Jul 6, 2024

• Merge branch 'main' into importing-v3

  northdpole

  authored
  addc315

  Copy full SHA for addc315

• in pyyaml, try to fix incompatibility with cython

  northdpole

  committed
  d6394fb

  Copy full SHA for d6394fb

• init fix e2e tests

  northdpole

  committed
  3ab53c0

  Copy full SHA for 3ab53c0