Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ppantt 210 feat introduce healthcheck job #2640

Open
wants to merge 74 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
335cdd4
[PPANTT-137] feat: updated modules and workload identity for aks-leon…
alessio-cialini Oct 14, 2024
c9ef711
[PPANTT-137] feat: updated modules and workload identity for aks-leon…
alessio-cialini Oct 14, 2024
1c9537f
[PPANTT-168] feat: Introducing infra domain for gpd ingestion
alessio-cialini Oct 21, 2024
f204df5
Merge branch 'PPANTT-137-feat-update-aks-and-identity-on-payopts' int…
alessio-cialini Oct 24, 2024
0e41c0a
[PPANTT-168] feat: Introducing debezium connector, updated module ver…
alessio-cialini Oct 24, 2024
9c9ce98
[PPANTT-168] feat: updated eventhub config and kafka-connect.yaml
alessio-cialini Oct 25, 2024
53d946b
Merge branch 'main' into PPANTT-168-feat-introduce-ingestion-infra
alessio-cialini Oct 28, 2024
b966aa5
fix
pasqualespica Oct 28, 2024
c50f0dc
fix
pasqualespica Oct 28, 2024
923b306
fix
pasqualespica Oct 28, 2024
948d295
fix
pasqualespica Oct 28, 2024
f7b8593
fix
pasqualespica Oct 29, 2024
1c12976
[PPANTT-168] feat: updated terraform vars
alessio-cialini Oct 30, 2024
1381777
fix
pasqualespica Oct 30, 2024
6594ea6
fix
pasqualespica Oct 30, 2024
ecb91b1
[PPANTT-168] feat: updated 05_debezium_connect.tf, debezium-secrets.y…
alessio-cialini Oct 31, 2024
310a520
fix
pasqualespica Oct 31, 2024
6328de1
[PPANTT-168] feat: updated 05_debezium_connect.tf, debezium-secrets.y…
alessio-cialini Oct 31, 2024
5ccc3b9
Merge remote-tracking branch 'origin/PPANTT-168-feat-introduce-ingest…
alessio-cialini Oct 31, 2024
5623b13
fix
pasqualespica Oct 31, 2024
24c922c
fix
pasqualespica Nov 4, 2024
3ab638c
fix wait chk resources
pasqualespica Nov 4, 2024
1a18c4c
Merge branch 'main' into PPANTT-168-feat-introduce-ingestion-infra
pasqualespica Nov 4, 2024
87f516d
fix
pasqualespica Nov 4, 2024
a78935e
fix
pasqualespica Nov 4, 2024
e8e94b6
fix
pasqualespica Nov 4, 2024
47429c9
fix
pasqualespica Nov 4, 2024
8172cb9
fix
pasqualespica Nov 4, 2024
5beb9e5
fix
pasqualespica Nov 4, 2024
9536951
fix
pasqualespica Nov 4, 2024
443005c
fix
pasqualespica Nov 4, 2024
6efa8db
align config
pasqualespica Nov 4, 2024
64e2c0c
change evh to public
pasqualespica Nov 4, 2024
44271bc
fix
pasqualespica Nov 4, 2024
c3dd08e
add mng to cdc-gpd-connection-string
pasqualespica Nov 4, 2024
c4d60f7
merge
pasqualespica Nov 7, 2024
9d89d65
fix
pasqualespica Nov 12, 2024
02d6a6e
fix
pasqualespica Nov 13, 2024
0eb5749
fi
pasqualespica Nov 13, 2024
045c102
deploy UAT
pasqualespica Nov 14, 2024
a74cc7b
Merge branch 'main' into PPANTT-168-feat-introduce-ingestion-infra
alessio-cialini Nov 14, 2024
d4605e3
[PPANTT-171] feat: introduced ingestion manager to the status page
alessio-cialini Nov 14, 2024
f532fe8
[PPANTT-186] feat: introduced alerts for ingestion manager functions
alessio-cialini Nov 15, 2024
95e9eaa
Merge branch 'main' into PPANTT-168-feat-introduce-ingestion-infra
pasqualespica Nov 15, 2024
a8a6c3e
fix
pasqualespica Nov 15, 2024
333a70e
[PPANTT-186] feat: introduced alerts for ingestion manager functions
alessio-cialini Nov 15, 2024
cf0d630
Merge branch 'main' into PPANTT-168-feat-introduce-ingestion-infra
pasqualespica Nov 18, 2024
8c1d635
fix
pasqualespica Nov 18, 2024
d9907eb
[PPANTT-185] feat: introduced logging
alessio-cialini Nov 18, 2024
1a64374
[PPANTT-168] feat: Updated postgres flex config
alessio-cialini Nov 18, 2024
3e5a398
Merge branch 'PPANTT-185-feat-add-logging' into PPANTT-168-feat-intro…
alessio-cialini Nov 18, 2024
7cb3259
fix new name gpd db
pasqualespica Nov 19, 2024
f4c7310
fix new name gpd db
pasqualespica Nov 19, 2024
a45f996
fix
pasqualespica Nov 19, 2024
f10ca80
fix
pasqualespica Nov 19, 2024
a4694f7
fix
pasqualespica Nov 21, 2024
5ef75e5
fix
pasqualespica Nov 21, 2024
60007c1
Merge branch 'main' into PPANTT-168-feat-introduce-ingestion-infra
pasqualespica Nov 25, 2024
b0dbf87
fix
pasqualespica Nov 25, 2024
7bd972f
Merge branch 'main' into PPANTT-168-feat-introduce-ingestion-infra
pasqualespica Nov 27, 2024
e89dab9
fix
pasqualespica Nov 27, 2024
6cec4af
Merge branch 'main' into PPANTT-168-feat-introduce-ingestion-infra
pasqualespica Nov 27, 2024
b3dfd27
fix
pasqualespica Nov 28, 2024
76a8bcd
fix
pasqualespica Nov 28, 2024
b28fc94
merge main
pasqualespica Dec 4, 2024
d7f3d71
fix
pasqualespica Dec 4, 2024
38fc2d2
[PPANTT-210] feat: Introducing cronjob script to restart if connector…
alessio-cialini Dec 5, 2024
21a8d24
[PPANTT-210] feat: Introducing cronjob script to restart if connector…
alessio-cialini Dec 5, 2024
6e5f08f
[PPANTT-210] feat: introducing policies for networking
alessio-cialini Dec 6, 2024
ded4f7f
[PPANTT-210] feat: introducing updated script and network policies
alessio-cialini Dec 6, 2024
86f03e4
Merge branch 'main' into PPANTT-210-feat-introduce-healthcheck-job
alessio-cialini Dec 9, 2024
9187e28
[PPANTT-210] feat: Debezium GPD API
alessio-cialini Dec 9, 2024
5c2ba09
Merge remote-tracking branch 'origin/PPANTT-210-feat-introduce-health…
alessio-cialini Dec 9, 2024
c66b69d
merge main
pasqualespica Dec 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions src/domains/gps-app/04_apim_gpd_debezium.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
####################
## Local variables #
####################

locals {
apim_gpd_debezium_api = {
published = false
subscription_required = true
approval_required = false
subscriptions_limit = 1000
service_url = format("https://%s/debezium-gpd", local.gps_hostname)
}
}

##############
## Products ##
##############

module "apim_gpd_debezium_product" {
source = "./.terraform/modules/__v3__/api_management_product"

product_id = "product-gpd-debezium"
display_name = "GPD Debezium API pagoPA"
description = "Prodotto GPD Debezium API"

api_management_name = local.pagopa_apim_name
resource_group_name = local.pagopa_apim_rg

published = local.apim_gpd_debezium_api.published
subscription_required = local.apim_gpd_debezium_api.subscription_required
approval_required = local.apim_gpd_debezium_api.approval_required
subscriptions_limit = local.apim_gpd_debezium_api.subscriptions_limit

policy_xml = file("./api_product/debezium-api/_base_policy.xml")
}

##############
## API ##
##############

resource "azurerm_api_management_api_version_set" "api_gpd_debezium_api" {

name = format("%s-api-gpd-debezium-api", var.env_short)
api_management_name = local.pagopa_apim_name
resource_group_name = local.pagopa_apim_rg
display_name = "GPD Debezium API"
versioning_scheme = "Segment"
}


module "apim_api_gpd_debezium_api" {
source = "./.terraform/modules/__v3__/api_management_api"

name = format("%s-api-gpd-debezium-api", var.env_short)
api_management_name = local.pagopa_apim_name
resource_group_name = local.pagopa_apim_rg
product_ids = [module.apim_gpd_debezium_product.product_id, module.apim_gpd_debezium_product.product_id]
subscription_required = local.apim_gpd_debezium_api.subscription_required
api_version = "v1"
version_set_id = azurerm_api_management_api_version_set.api_gpd_debezium_api.id
service_url = format("https://%s", module.reporting_analysis_function.default_hostname)

description = "Api GPD Debezium"
display_name = "GPDDebezium API pagoPA"
path = "gpd-debezium/api"
protocols = ["https"]

content_format = "openapi"
content_value = templatefile("./api/debezium-api/v1/_openapi.json.tpl", {
host = local.apim_hostname
})

xml_content = templatefile("./api/debezium-api/v1/_base_policy.xml", {
origin = format("https://%s.%s.%s", var.cname_record_name, var.apim_dns_zone_prefix, var.external_domain)
})
}
51 changes: 51 additions & 0 deletions src/domains/gps-app/05_debezium_connect.tf
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,24 @@ locals {
max_threads = var.max_threads
})

healthchecker_config_yaml = templatefile("${path.module}/yaml/healthchecker-config-map.yaml", {
namespace = "gps" # kubernetes_namespace.namespace.metadata[0].name
})

debezium_health_checker_cron_yaml = templatefile("${path.module}/yaml/debezium-health-checker-cron.yaml", {
namespace = "gps" # kubernetes_namespace.namespace.metadata[0].name
})

debezium_network_policy_yaml = templatefile("${path.module}/yaml/debezium-network-policy.yaml", {
namespace = "gps" # kubernetes_namespace.namespace.metadata[0].name
})

debezium_ingress_yaml = templatefile("${path.module}/yaml/debezium-ingress.yaml", {
namespace = "gps" # kubernetes_namespace.namespace.metadata[0].name
host = "${var.location_short}${var.env_short}.gps.internal.${var.env_short}.platform.pagopa.it"
secret = "${var.location_short}${var.env_short}-gps-internal-${var.env_short}-platform-pagopa-it"
})

}

resource "kubectl_manifest" "debezium_role" {
Expand Down Expand Up @@ -178,3 +196,36 @@ resource "null_resource" "wait_postgres_connector" {
interpreter = ["/bin/bash", "-c"]
}
}

resource "kubectl_manifest" "healthchecker-config-map" {
depends_on = [
helm_release.strimzi-kafka-operator
]
force_conflicts = true
yaml_body = local.healthchecker_config_yaml
}

resource "kubectl_manifest" "healthchecker-cron" {
depends_on = [
helm_release.strimzi-kafka-operator, kubectl_manifest.healthchecker-config-map
]
force_conflicts = true
yaml_body = local.debezium_health_checker_cron_yaml
}

resource "kubectl_manifest" "debezium-ingress" {
depends_on = [
kubectl_manifest.kafka_connect
]
force_conflicts = true
yaml_body = local.debezium_ingress_yaml
}

resource "kubectl_manifest" "debezium-network-policy" {
depends_on = [
kubectl_manifest.kafka_connect
]
force_conflicts = true
yaml_body = local.debezium_network_policy_yaml
}

28 changes: 28 additions & 0 deletions src/domains/gps-app/api/debezium-api/v1/_base_policy.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
<!--
IMPORTANT:
- Policy elements can appear only within the <inbound>, <outbound>, <backend> section elements.
- To apply a policy to the incoming request (before it is forwarded to the backend service), place a corresponding policy element within the <inbound> section element.
- To apply a policy to the outgoing response (before it is sent back to the caller), place a corresponding policy element within the <outbound> section element.
- To add a policy, place the cursor at the desired insertion point and select a policy from the sidebar.
- To remove a policy, delete the corresponding policy statement from the policy document.
- Position the <base> element within a section element to inherit all policies from the corresponding section element in the enclosing scope.
- Remove the <base> element to prevent inheriting policies from the corresponding section element in the enclosing scope.
- Policies are applied in the order of their appearance, from the top down.
- Comments within policy elements are not supported and may disappear. Place your comments between policy elements or at a higher level scope.
-->
<policies>
<inbound>
<base />
<!-- rate limit by subscription key -->
<rate-limit calls="300" renewal-period="10" remaining-calls-variable-name="remainingCallsPerSubscription"/>
</inbound>
<backend>
<base />
</backend>
<outbound>
<base />
</outbound>
<on-error>
<base />
</on-error>
</policies>
156 changes: 156 additions & 0 deletions src/domains/gps-app/api/debezium-api/v1/_openapi.json.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
{
"openapi": "3.0.0",
"info": {
"title": "Debezium API - GPD",
"version": "1.0.0"
},
"servers": [
{
"url": "${host}"
}
],
"paths": {
"/connectors": {
"get": {
"tags": [
"Get Connectors List"
],
"summary": "getConnectors",
"parameters": [],
"responses": {
"200": {
"description": "Successful response",
"content": {
"application/json": {
}
}
},
"400": {
"description": "Error response",
"content": {
"application/json": {}
}
}
}
}
},
"/connectors/{connectorId}/status": {
"get": {
"tags": [
"Get Detail on Connector Status"
],
"summary": "getConnectorStatus",
"parameters": [
{
"name": "connectorId",
"in": "path",
"schema": {
"type": "string"
},
"required": true,
"example": "debezium-connector-postgres"
}
],
"responses": {
"200": {
"description": "Successful response",
"content": {
"application/json": {}
}
},
"404": {
"description": "Not Found",
"content": {
"application/json": {}
}
}
}
}
},
"/connectors/{connectorId}/restart": {
"post": {
"tags": [
"Restart Connector"
],
"summary": "restartConnector",
"parameters": [
{
"name": "connectorId",
"in": "path",
"schema": {
"type": "string"
},
"required": true,
"example": "debezium-connector-postgres"
}
],
"responses": {
"204": {
"description": "Successful response",
"content": {
"application/json": {}
}
},
"404": {
"description": "Not Found",
"content": {
"application/json": {}
}
}
}
}
},
"/connectors/{connectorId}/tasks/{taskId}/restart": {
"post": {
"tags": [
"Restart Task "
],
"summary": "restartTask",
"parameters": [
{
"name": "connectorId",
"in": "path",
"schema": {
"type": "string"
},
"required": true,
"example": "debezium-connector-postgres"
},
{
"name": "taskId",
"in": "path",
"schema": {
"type": "string"
},
"required": true,
"example": "0"
}
],
"responses": {
"204": {
"description": "Successful response",
"content": {
"application/json": {}
}
},
"404": {
"description": "Not Found",
"content": {
"application/json": {}
}
}
}
}
}
},
"components": {
"securitySchemes": {
"ApiKey": {
"type": "apiKey",
"description": "The API key to access this function app.",
"name": "Ocp-Apim-Subscription-Key",
"in": "header"
}
}
}
}
26 changes: 26 additions & 0 deletions src/domains/gps-app/api_product/debezium-api/_base_policy.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
<!--
IMPORTANT:
- Policy elements can appear only within the <inbound>, <outbound>, <backend> section elements.
- To apply a policy to the incoming request (before it is forwarded to the backend service), place a corresponding policy element within the <inbound> section element.
- To apply a policy to the outgoing response (before it is sent back to the caller), place a corresponding policy element within the <outbound> section element.
- To add a policy, place the cursor at the desired insertion point and select a policy from the sidebar.
- To remove a policy, delete the corresponding policy statement from the policy document.
- Position the <base> element within a section element to inherit all policies from the corresponding section element in the enclosing scope.
- Remove the <base> element to prevent inheriting policies from the corresponding section element in the enclosing scope.
- Policies are applied in the order of their appearance, from the top down.
- Comments within policy elements are not supported and may disappear. Place your comments between policy elements or at a higher level scope.
-->
<policies>
<inbound>
<base />
</inbound>
<backend>
<base />
</backend>
<outbound>
<base />
</outbound>
<on-error>
<base />
</on-error>
</policies>
26 changes: 26 additions & 0 deletions src/domains/gps-app/yaml/debezium-health-checker-cron.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: debezium-health-checker-cron
namespace: ${namespace}
spec:
schedule: "*/5 * * * *" # Runs every 5 minutes
successfulJobsHistoryLimit: 0
failedJobsHistoryLimit: 1
jobTemplate:
spec:
ttlSecondsAfterFinished: 100
template:
spec:
containers:
- name: debezium-health-checker
image: bitnami/kubectl:latest
command: ["/bin/bash", "/scripts/check_kafka_connect.sh"]
volumeMounts:
- name: script-volume
mountPath: /scripts
restartPolicy: OnFailure
volumes:
- name: script-volume
configMap:
name: health-checker-script
Loading
Loading