Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Add connection pool to configure connection lifecycle. #289

Merged
merged 16 commits into from
Dec 13, 2024

Conversation

tristanvuong2021
Copy link
Contributor

The pool will close connections that are idle for too long.

@wfa-reviewable
Copy link

This change is Reviewable

Copy link
Member

@SanjayVas SanjayVas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 4 of 5 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @tristanvuong2021)


src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresConnectionFactories.kt line 38 at r2 (raw file):

          .option(ConnectionFactoryOptions.DATABASE, flags.database)
          .option(ConnectionFactoryOptions.HOST, flags.cloudSqlInstance)
          .option(ConnectionFactoryOptions.STATEMENT_TIMEOUT, Duration.ofSeconds(120))

Have all of these values come from flags.

Copy link
Contributor Author

@tristanvuong2021 tristanvuong2021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 4 of 6 files reviewed, 1 unresolved discussion (waiting on @SanjayVas)


src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresConnectionFactories.kt line 38 at r2 (raw file):

Previously, SanjayVas (Sanjay Vasandani) wrote…

Have all of these values come from flags.

Done.

Copy link
Member

@SanjayVas SanjayVas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 2 files at r3, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @tristanvuong2021)


src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresFlags.kt line 50 at r3 (raw file):

    required = false,
  )
  var statementTimeoutSeconds: Long = 120

Use Duration for these

Suggestion:

    names = ["--statement-timeout"],
    description = ["Statement timeout"],
    required = false,
    defaultValue = "120s",
  )
  lateinit var statementTimeout: Duration

Copy link
Contributor Author

@tristanvuong2021 tristanvuong2021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 4 of 6 files reviewed, 1 unresolved discussion (waiting on @SanjayVas)


src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresFlags.kt line 50 at r3 (raw file):

Previously, SanjayVas (Sanjay Vasandani) wrote…

Use Duration for these

Done.

@tristanvuong2021 tristanvuong2021 force-pushed the tristanvuong-add-connection-pool branch from fee87f1 to e24fa72 Compare December 9, 2024 16:22
Copy link
Member

@SanjayVas SanjayVas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 4 of 4 files at r4, all commit messages.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @tristanvuong2021)


src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresFlags.kt line 51 at r4 (raw file):

    required = false,
  )
  var statementTimeout: Duration = Duration.ofSeconds(120)

Don't assign the default in Kotlin. Make these lateinit vars and use defaultValue from the Option annotation

Code quote:

= Duration.ofSeconds(120)

src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresFlags.kt line 55 at r4 (raw file):

  @CommandLine.Option(
    names = ["--max-idle-time-minutes"],

Suggestion:

--max-idle-time

Copy link
Contributor Author

@tristanvuong2021 tristanvuong2021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 5 of 6 files reviewed, 2 unresolved discussions (waiting on @SanjayVas)


src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresFlags.kt line 51 at r4 (raw file):

Previously, SanjayVas (Sanjay Vasandani) wrote…

Don't assign the default in Kotlin. Make these lateinit vars and use defaultValue from the Option annotation

Done.

Copy link
Contributor Author

@tristanvuong2021 tristanvuong2021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 5 of 6 files reviewed, 2 unresolved discussions (waiting on @SanjayVas)


src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresFlags.kt line 55 at r4 (raw file):

  @CommandLine.Option(
    names = ["--max-idle-time-minutes"],

Done.

Copy link
Member

@SanjayVas SanjayVas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all commit messages.
Reviewable status: 5 of 6 files reviewed, 1 unresolved discussion (waiting on @tristanvuong2021)


src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresFlags.kt line 50 at r5 (raw file):

    description = ["Statement timeout duration."],
    required = false,
    defaultValue = "PT120S",

This should be using human readable durations. I've stated in a previous PR that org.wfanet.measurement.common.DurationFormat shouldn't exist and it should always use the human-readable format, i.e. that #174 should never have been merged.

Suggestion:

120s

Copy link
Contributor Author

@tristanvuong2021 tristanvuong2021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 5 of 6 files reviewed, 1 unresolved discussion (waiting on @SanjayVas)


src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresFlags.kt line 50 at r5 (raw file):

Previously, SanjayVas (Sanjay Vasandani) wrote…

This should be using human readable durations. I've stated in a previous PR that org.wfanet.measurement.common.DurationFormat shouldn't exist and it should always use the human-readable format, i.e. that #174 should never have been merged.

Done.

Copy link
Member

@SanjayVas SanjayVas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 1 files at r6, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @tristanvuong2021)


src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresFlags.kt line 69 at r6 (raw file):

    required = false,
  )
  var maxPoolSize: Int = 16

Same here re: not setting defaults in Kotlin when they can be set using defaultValue

Suggestion:

var maxPoolSize by Delegates.notNull<Int>()

Copy link
Contributor Author

@tristanvuong2021 tristanvuong2021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 5 of 6 files reviewed, 1 unresolved discussion (waiting on @SanjayVas)


src/main/kotlin/org/wfanet/measurement/gcloud/postgres/PostgresFlags.kt line 69 at r6 (raw file):

Previously, SanjayVas (Sanjay Vasandani) wrote…

Same here re: not setting defaults in Kotlin when they can be set using defaultValue

Done.

Copy link
Member

@SanjayVas SanjayVas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 1 files at r7, all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @stevenwarejones)

Copy link
Contributor

@stevenwarejones stevenwarejones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 5 files at r1, 3 of 4 files at r4, 1 of 1 files at r7, all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @tristanvuong2021)

@tristanvuong2021 tristanvuong2021 merged commit bfc08ff into main Dec 13, 2024
3 checks passed
@tristanvuong2021 tristanvuong2021 deleted the tristanvuong-add-connection-pool branch December 13, 2024 22:23
SanjayVas added a commit that referenced this pull request Dec 18, 2024
tristanvuong2021 added a commit that referenced this pull request Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Number of open connections locks keep increasing on reporting server postgresql instance
4 participants