Docs or guidance on writing additional third party fan out integrations

[garethr]

You mention Third Party integration fan out in the docs. I'm wondering if there are any documentation, or even just examples, of how to write additional integrations to send that information to other backends?

[migmartri]

Hi @garethr!

The short answer is no :( we don't have documentation yet and in fact there is probably some work to do first to make sure the abstractions are properly set or to create an actual plugins architecture.

I can try to in the meantime explain what I believe would be the steps involved

1 - Implement the actual integration code

A integration needs to implement this interface.

chainloop/app/controlplane/internal/integrations/integration.go

Lines 20 to 29 in e770fae

	type Checker interface {
	// Validate that the integration can be registered for future calls
	Validate(ctx context.Context) error
	}
	type Doer interface {
	Do(ctx context.Context) error
	// Validate that the integration can be performed with the parameters provided
	Validate(ctx context.Context) error
	}

and you can see the only integration we have right now (Dependency-Track for SBOMs) that implements this interface here

NOTE: OCI registries although they are depicted in our diagrams as fan-out integrations, it's implementation doesn't follow this approach for historical reasons, but it will fix with this #28

2 - Registration logic

2.1 - Add use-case business logic

This means that an user can register their instance of the integration in their control-plane org. Basically this step

To do that you'd need to add an useCase to register the new integration in the biz layer. Similar to

chainloop/app/controlplane/internal/biz/integration.go

Line 94 in e770fae

func (uc *IntegrationUseCase) AddDependencyTrack(ctx context.Context, orgID, host, apiKey string, enableProjectCreation bool) (*Integration, error) {

that code will validate and store any credentials in the secrets storage, i.e dependency-track API key, and store an entry in the DB.

2.2 - Add API endpoint

Next, you'll need to add an API endpoint to add your new integration.

Similar to this one

chainloop/app/controlplane/api/controlplane/v1/integrations.proto

Lines 25 to 27 in e770fae

	service IntegrationsService {
	// ORG related CRUD
	rpc AddDependencyTrack (AddDependencyTrackRequest) returns (AddDependencyTrackResponse);

3 - Enable attachment to workflows

At this point you'll have a mechanism for your users to add this new integration to their organization, so they'll be able to get smth like

$ chainloop integration ls
┌──────────────────────────────────────┬──────────────────┬──────────────────────────────────────────────┬─────────────────────┐
│ ID                                   │ KIND             │ CONFIG                                       │ CREATED AT          │
├──────────────────────────────────────┼──────────────────┼──────────────────────────────────────────────┼─────────────────────┤
│ c48f3041-5745-4773-aed9-ccf2f288b2e4 │ Dependency-Track │ host: https://dependency-track.chainloop.dev │ 21 Dec 22 16:29 UTC │
│                                      │                  │ allowAutoCreate: true                        │                     │
└──────────────────────────────────────┴──────────────────┴──────────────────────────────────────────────┴─────────────────────┘

Next, you'll need to implement how to allow your users to attach this new integration to their workflow, so it gets executed once an attestation is received.

to do that

3.1 - Extend API with custom configuration

If the attachment command requires additional, specific configurations, you'll need to extend this oneof and it's handling

chainloop/app/controlplane/api/controlplane/v1/integrations.proto

Lines 96 to 99 in e770fae

	message IntegrationConfig {
	oneof config {
	DependencyTrack dependency_track = 1;
	}

3.2 - Handle attach to workflow logic

chainloop/app/controlplane/internal/biz/integration.go

Line 269 in e770fae

func validateAttachment(ctx context.Context, integration *Integration, credsR credentials.Reader, ic *v1.IntegrationConfig, ac *v1.IntegrationAttachmentConfig) error {

4 - Implement the actual firing mechanism

Now our new integration is ready to 1) be registered by any organization and 2) be attached to as many workflows as needed.

Next, we need to implement the actual actuation of the integration code you implemented in step 1.

And this is by far the part that needs more work. I have pending adding an issue about adding an event bus (nats.io or similar) to handle persistence, retries, etc.

but for now the code is executed in the service implementation when the attestation is received, like for example

chainloop/app/controlplane/internal/service/attestation.go

Line 223 in e770fae

go func() {

I know that this is a lot to digest and it's current architecture is not great. As I mentioned before, it might make sense to try to encapsulate most of these changes as part of a new registration/attachment interface, so the custom code related to adding a new integration will be mostly encapsulated in the first step.

I am going to take a look at that as part of #28 but if you have any thoughts, want to chat or help please let me know!!

Thank you!

[garethr]

I love "The short answer is no" and then the longer answer being all the details that would be needed, along with all of the context 🥇

I'll definitely take a look if/when I have time, and will track the issue mentioned.

Thanks for the detailed response.

[migmartri]

Sounds good! and yes, I error on the verbosity side as long as a TLDR is in place haha :)

For the record, and to elaborate on my previous statement...

... there is probably some work to do first to make sure the abstractions are properly set or to create an actual plugins architecture.

These are some of the additional work streams around third-party integrations that I have on top of my priority list.

• [fan-out integration] new SDK #38
• [fan-out integration] add resiliency to firing mechanism #39

In any case @garethr, if you have in mind any specific integration you'd like to explore or have added to Chainloop, feel free to let us know.

Having an integration candidate will also help with the effort on defining the right abstractions and documentation.

Thank you!

[migmartri]

Hi @garethr. We just wanted to let you know that we've put in place a new plugins sdk and documentation. This allows anybody to add a fan-out integration in an encapsulated manner. Note: plugins are not fully decoupled from there core yet, we will get there :) #195

We also expect to have a build a plugin example #204 published sometime in the future, but in the meantime, is there any integration endpoint/use-case that you could think it would be interesting to implement? We'd happy to give it a crack.