Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance improvements: Add Brainstorming Label to idea + broadcast lane update #507

Merged
merged 4 commits into from
Dec 23, 2024

Conversation

PragTob
Copy link
Collaborator

@PragTob PragTob commented Nov 30, 2024

This was mostly me looking at SQL emitted when adding a new label.

This PR reduces the total amount of SQL queries issued for adding a label from 20 to 14

Log for add label before optimization
[debug] HANDLE EVENT "add_idea_label_to_idea" in MindwendelWeb.BrainstormingLive.Show
  Component: MindwendelWeb.IdeaLive.CardComponent
  Parameters: %{"idea-id" => "7c9dca19-94df-4653-b2db-1c2cc49edacc", "idea-label-id" => "84aedeef-eb27-4cb4-8804-05eca963b6e0"}
[debug] QUERY OK source="ideas" db=0.3ms idle=484.8ms
SELECT i0."id", i0."body", i0."position_order", i0."username", i0."comments_count", i0."user_id", i0."brainstorming_id", i0."lane_id", i0."inserted_at", i0."updated_at" FROM "ideas" AS i0 WHERE (i0."id" = $1) ["7c9dca19-94df-4653-b2db-1c2cc49edacc"]
[debug] QUERY OK source="idea_files" db=0.2ms idle=485.4ms
SELECT i0."id", i0."name", i0."path", i0."file_type", i0."idea_id", i0."inserted_at", i0."updated_at", i0."idea_id" FROM "idea_files" AS i0 WHERE (i0."idea_id" = $1) ORDER BY i0."idea_id" ["7c9dca19-94df-4653-b2db-1c2cc49edacc"]
[debug] QUERY OK source="idea_comments" db=0.2ms idle=485.5ms
SELECT i0."id", i0."idea_id", i0."user_id", i0."body", i0."username", i0."inserted_at", i0."updated_at", i0."idea_id" FROM "idea_comments" AS i0 WHERE (i0."idea_id" = $1) ORDER BY i0."idea_id", i0."inserted_at" DESC ["7c9dca19-94df-4653-b2db-1c2cc49edacc"]
[debug] QUERY OK source="links" db=0.3ms idle=485.6ms
SELECT l0."id", l0."url", l0."title", l0."description", l0."img_preview_url", l0."idea_id", l0."inserted_at", l0."updated_at", l0."idea_id" FROM "links" AS l0 WHERE (l0."idea_id" = $1) ["7c9dca19-94df-4653-b2db-1c2cc49edacc"]
[debug] QUERY OK source="idea_labels" db=1.1ms idle=485.4ms
SELECT i0."id", i0."name", i0."color", i0."position_order", i0."brainstorming_id", i0."inserted_at", i0."updated_at", i1."idea_id"::uuid FROM "idea_labels" AS i0 INNER JOIN "idea_idea_labels" AS i1 ON i0."id" = i1."idea_label_id" WHERE (i1."idea_id" = ANY($1)) ORDER BY i1."idea_id"::uuid [["7c9dca19-94df-4653-b2db-1c2cc49edacc"]]
[debug] QUERY OK source="idea_labels" db=0.4ms idle=486.7ms
SELECT i0."id", i0."name", i0."color", i0."position_order", i0."brainstorming_id", i0."inserted_at", i0."updated_at" FROM "idea_labels" AS i0 WHERE (i0."id" = $1) ["84aedeef-eb27-4cb4-8804-05eca963b6e0"]
[debug] QUERY OK db=0.1ms idle=487.3ms
begin []
[debug] QUERY OK source="idea_idea_labels" db=0.4ms
INSERT INTO "idea_idea_labels" ("idea_label_id","idea_id","inserted_at","updated_at") VALUES ($1,$2,$3,$4) ["84aedeef-eb27-4cb4-8804-05eca963b6e0", "7c9dca19-94df-4653-b2db-1c2cc49edacc", ~N[2024-11-30 18:37:32], ~N[2024-11-30 18:37:32]]
[debug] QUERY OK db=7.6ms
commit []
[debug] QUERY OK source="brainstormings" db=0.1ms idle=495.8ms
SELECT b0."id", b0."name", b0."option_allow_manual_ordering", b0."admin_url_id", b0."last_accessed_at", b0."filter_labels_ids", b0."creating_user_id", b0."inserted_at", b0."updated_at" FROM "brainstormings" AS b0 WHERE (b0."id" = $1) ["25a35202-c68d-4278-af69-382849134847"]
[debug] QUERY OK source="idea_labels" db=0.2ms idle=11.1ms
SELECT i0."id", i0."name", i0."color", i0."position_order", i0."brainstorming_id", i0."inserted_at", i0."updated_at", i0."brainstorming_id" FROM "idea_labels" AS i0 WHERE (i0."brainstorming_id" = $1) ORDER BY i0."brainstorming_id", i0."position_order" ["25a35202-c68d-4278-af69-382849134847"]
[debug] QUERY OK source="users" db=0.4ms idle=496.1ms
SELECT u0."id", u0."username", u0."inserted_at", u0."updated_at", b1."brainstorming_id"::uuid FROM "users" AS u0 INNER JOIN "brainstorming_moderating_users" AS b1 ON u0."id" = b1."user_id" WHERE (b1."brainstorming_id" = ANY($1)) ORDER BY b1."brainstorming_id"::uuid [["25a35202-c68d-4278-af69-382849134847"]]
[debug] QUERY OK source="users" db=0.4ms idle=373.5ms
SELECT u0."id", u0."username", u0."inserted_at", u0."updated_at", b1."brainstorming_id"::uuid FROM "users" AS u0 INNER JOIN "brainstorming_users" AS b1 ON u0."id" = b1."user_id" WHERE (b1."brainstorming_id" = ANY($1)) ORDER BY b1."brainstorming_id"::uuid [["25a35202-c68d-4278-af69-382849134847"]]
[debug] QUERY OK source="brainstormings" db=1.0ms idle=11.1ms
UPDATE "brainstormings" SET "last_accessed_at" = $1, "updated_at" = $2 WHERE "id" = $3 [~U[2024-11-30 18:37:32Z], ~N[2024-11-30 18:37:32], "25a35202-c68d-4278-af69-382849134847"]
[debug] QUERY OK source="lanes" db=0.2ms idle=12.3ms
SELECT l0."id", l0."name", l0."position_order", l0."brainstorming_id", l0."inserted_at", l0."updated_at" FROM "lanes" AS l0 WHERE (l0."brainstorming_id" = $1) ORDER BY l0."position_order", l0."inserted_at" ["25a35202-c68d-4278-af69-382849134847"]
[debug] QUERY OK source="ideas" db=0.2ms idle=12.6ms
SELECT i0."id", i0."body", i0."position_order", i0."username", i0."comments_count", i0."user_id", i0."brainstorming_id", i0."lane_id", i0."inserted_at", i0."updated_at", i0."lane_id" FROM "ideas" AS i0 WHERE (i0."lane_id" = ANY($1)) ORDER BY i0."lane_id", i0."position_order", i0."inserted_at" [["ef8dbd09-f1be-410b-8e35-298ffaaf1a09", "69f02828-b1f7-4b73-a8a7-6177090fda22"]]
[debug] QUERY OK source="idea_files" db=0.1ms idle=3.3ms
SELECT i0."id", i0."name", i0."path", i0."file_type", i0."idea_id", i0."inserted_at", i0."updated_at", i0."idea_id" FROM "idea_files" AS i0 WHERE (i0."idea_id" = ANY($1)) ORDER BY i0."idea_id" [["2dfd9635-f0e0-44e3-8acf-f080c681b07f", "f8b75443-8214-465a-af87-f0f87f1d3512", "30ca05a6-71e0-4fe1-9acf-9a99beaf50f8", "7c9dca19-94df-4653-b2db-1c2cc49edacc", "7bff94f2-e3e3-440d-8346-4c68a2ea00ab"]]
[debug] QUERY OK source="links" db=0.1ms idle=3.1ms
SELECT l0."id", l0."url", l0."title", l0."description", l0."img_preview_url", l0."idea_id", l0."inserted_at", l0."updated_at", l0."idea_id" FROM "links" AS l0 WHERE (l0."idea_id" = ANY($1)) [["2dfd9635-f0e0-44e3-8acf-f080c681b07f", "f8b75443-8214-465a-af87-f0f87f1d3512", "30ca05a6-71e0-4fe1-9acf-9a99beaf50f8", "7c9dca19-94df-4653-b2db-1c2cc49edacc", "7bff94f2-e3e3-440d-8346-4c68a2ea00ab"]]
[debug] QUERY OK source="idea_labels" db=0.4ms idle=12.4ms
SELECT i0."id", i0."name", i0."color", i0."position_order", i0."brainstorming_id", i0."inserted_at", i0."updated_at", i1."idea_id"::uuid FROM "idea_labels" AS i0 INNER JOIN "idea_idea_labels" AS i1 ON i0."id" = i1."idea_label_id" WHERE (i1."idea_id" = ANY($1)) ORDER BY i1."idea_id"::uuid [["2dfd9635-f0e0-44e3-8acf-f080c681b07f", "f8b75443-8214-465a-af87-f0f87f1d3512", "30ca05a6-71e0-4fe1-9acf-9a99beaf50f8", "7c9dca19-94df-4653-b2db-1c2cc49edacc", "7bff94f2-e3e3-440d-8346-4c68a2ea00ab"]]
[debug] QUERY OK source="likes" db=0.4ms idle=11.9ms
SELECT l0."id", l0."idea_id", l0."user_id", l0."inserted_at", l0."updated_at", l0."idea_id" FROM "likes" AS l0 WHERE (l0."idea_id" = ANY($1)) ORDER BY l0."idea_id" [["2dfd9635-f0e0-44e3-8acf-f080c681b07f", "f8b75443-8214-465a-af87-f0f87f1d3512", "30ca05a6-71e0-4fe1-9acf-9a99beaf50f8", "7c9dca19-94df-4653-b2db-1c2cc49edacc", "7bff94f2-e3e3-440d-8346-4c68a2ea00ab"]]
[debug] Replied in 15ms
Log for add label after optimization
[debug] HANDLE EVENT "add_idea_label_to_idea" in MindwendelWeb.BrainstormingLive.Show
  Component: MindwendelWeb.IdeaLive.CardComponent
  Parameters: %{"idea-id" => "7c9dca19-94df-4653-b2db-1c2cc49edacc", "idea-label-id" => "84aedeef-eb27-4cb4-8804-05eca963b6e0"}
[debug] QUERY OK source="ideas" db=0.5ms idle=1617.1ms
SELECT i0."id", i0."body", i0."position_order", i0."username", i0."comments_count", i0."user_id", i0."brainstorming_id", i0."lane_id", i0."inserted_at", i0."updated_at" FROM "ideas" AS i0 WHERE (i0."id" = $1) ["7c9dca19-94df-4653-b2db-1c2cc49edacc"]
[debug] QUERY OK source="idea_labels" db=0.3ms idle=624.0ms
SELECT i0."id", i0."name", i0."color", i0."position_order", i0."brainstorming_id", i0."inserted_at", i0."updated_at", i1."idea_id"::uuid FROM "idea_labels" AS i0 INNER JOIN "idea_idea_labels" AS i1 ON i0."id" = i1."idea_label_id" WHERE (i1."idea_id" = ANY($1)) ORDER BY i1."idea_id"::uuid [["7c9dca19-94df-4653-b2db-1c2cc49edacc"]]
[debug] QUERY OK source="idea_files" db=0.2ms idle=617.9ms
SELECT i0."id", i0."name", i0."path", i0."file_type", i0."idea_id", i0."inserted_at", i0."updated_at", i0."idea_id" FROM "idea_files" AS i0 WHERE (i0."idea_id" = $1) ORDER BY i0."idea_id" ["7c9dca19-94df-4653-b2db-1c2cc49edacc"]
[debug] QUERY OK source="idea_comments" db=0.3ms idle=618.0ms
SELECT i0."id", i0."idea_id", i0."user_id", i0."body", i0."username", i0."inserted_at", i0."updated_at", i0."idea_id" FROM "idea_comments" AS i0 WHERE (i0."idea_id" = $1) ORDER BY i0."idea_id", i0."inserted_at" DESC ["7c9dca19-94df-4653-b2db-1c2cc49edacc"]
[debug] QUERY OK source="links" db=0.2ms idle=618.1ms
SELECT l0."id", l0."url", l0."title", l0."description", l0."img_preview_url", l0."idea_id", l0."inserted_at", l0."updated_at", l0."idea_id" FROM "links" AS l0 WHERE (l0."idea_id" = $1) ["7c9dca19-94df-4653-b2db-1c2cc49edacc"]
[debug] QUERY OK source="idea_labels" db=0.2ms idle=618.7ms
SELECT i0."id", i0."name", i0."color", i0."position_order", i0."brainstorming_id", i0."inserted_at", i0."updated_at" FROM "idea_labels" AS i0 WHERE (i0."id" = $1) ["84aedeef-eb27-4cb4-8804-05eca963b6e0"]
[debug] QUERY OK source="idea_idea_labels" db=7.9ms queue=0.1ms idle=619.1ms
INSERT INTO "idea_idea_labels" ("idea_label_id","idea_id","inserted_at","updated_at") VALUES ($1,$2,$3,$4) ["84aedeef-eb27-4cb4-8804-05eca963b6e0", "7c9dca19-94df-4653-b2db-1c2cc49edacc", ~N[2024-11-30 19:37:49], ~N[2024-11-30 19:37:49]]
[debug] QUERY OK source="brainstormings" db=0.1ms idle=627.4ms
SELECT b0."id", b0."name", b0."option_allow_manual_ordering", b0."admin_url_id", b0."last_accessed_at", b0."filter_labels_ids", b0."creating_user_id", b0."inserted_at", b0."updated_at" FROM "brainstormings" AS b0 WHERE (b0."id" = $1) ["25a35202-c68d-4278-af69-382849134847"]
[debug] QUERY OK source="lanes" db=0.1ms idle=627.7ms
SELECT l0."id", l0."name", l0."position_order", l0."brainstorming_id", l0."inserted_at", l0."updated_at" FROM "lanes" AS l0 WHERE (l0."brainstorming_id" = $1) ORDER BY l0."position_order", l0."inserted_at" ["25a35202-c68d-4278-af69-382849134847"]
[debug] QUERY OK source="ideas" db=0.2ms idle=628.0ms
SELECT i0."id", i0."body", i0."position_order", i0."username", i0."comments_count", i0."user_id", i0."brainstorming_id", i0."lane_id", i0."inserted_at", i0."updated_at", i0."lane_id" FROM "ideas" AS i0 WHERE (i0."lane_id" = ANY($1)) ORDER BY i0."lane_id", i0."position_order", i0."inserted_at" [["ef8dbd09-f1be-410b-8e35-298ffaaf1a09", "69f02828-b1f7-4b73-a8a7-6177090fda22"]]
[debug] QUERY OK source="idea_labels" db=0.3ms idle=11.1ms
SELECT i0."id", i0."name", i0."color", i0."position_order", i0."brainstorming_id", i0."inserted_at", i0."updated_at", i1."idea_id"::uuid FROM "idea_labels" AS i0 INNER JOIN "idea_idea_labels" AS i1 ON i0."id" = i1."idea_label_id" WHERE (i1."idea_id" = ANY($1)) ORDER BY i1."idea_id"::uuid [["2dfd9635-f0e0-44e3-8acf-f080c681b07f", "f8b75443-8214-465a-af87-f0f87f1d3512", "30ca05a6-71e0-4fe1-9acf-9a99beaf50f8", "7c9dca19-94df-4653-b2db-1c2cc49edacc", "7bff94f2-e3e3-440d-8346-4c68a2ea00ab"]]
[debug] QUERY OK source="likes" db=0.3ms idle=10.5ms
SELECT l0."id", l0."idea_id", l0."user_id", l0."inserted_at", l0."updated_at", l0."idea_id" FROM "likes" AS l0 WHERE (l0."idea_id" = ANY($1)) ORDER BY l0."idea_id" [["2dfd9635-f0e0-44e3-8acf-f080c681b07f", "f8b75443-8214-465a-af87-f0f87f1d3512", "30ca05a6-71e0-4fe1-9acf-9a99beaf50f8", "7c9dca19-94df-4653-b2db-1c2cc49edacc", "7bff94f2-e3e3-440d-8346-4c68a2ea00ab"]]
[debug] QUERY OK source="idea_files" db=0.2ms idle=10.7ms
SELECT i0."id", i0."name", i0."path", i0."file_type", i0."idea_id", i0."inserted_at", i0."updated_at", i0."idea_id" FROM "idea_files" AS i0 WHERE (i0."idea_id" = ANY($1)) ORDER BY i0."idea_id" [["2dfd9635-f0e0-44e3-8acf-f080c681b07f", "f8b75443-8214-465a-af87-f0f87f1d3512", "30ca05a6-71e0-4fe1-9acf-9a99beaf50f8", "7c9dca19-94df-4653-b2db-1c2cc49edacc", "7bff94f2-e3e3-440d-8346-4c68a2ea00ab"]]
[debug] QUERY OK source="links" db=0.2ms idle=10.8ms
SELECT l0."id", l0."url", l0."title", l0."description", l0."img_preview_url", l0."idea_id", l0."inserted_at", l0."updated_at", l0."idea_id" FROM "links" AS l0 WHERE (l0."idea_id" = ANY($1)) [["2dfd9635-f0e0-44e3-8acf-f080c681b07f", "f8b75443-8214-465a-af87-f0f87f1d3512", "30ca05a6-71e0-4fe1-9acf-9a99beaf50f8", "7c9dca19-94df-4653-b2db-1c2cc49edacc", "7bff94f2-e3e3-440d-8346-4c68a2ea00ab"]]
[debug] Replied in 12ms
That number is still relatively high (imo) but this is due to the decision of broadcasting a full "Lane change" instead of the incremental equivalent update (which I didn't wanna touch here/change without discussion).

The PR mainly does 2 things:

  • limit what we do to add the label, as we discard the idea struct afterwards anyhow
  • on the broadcast event, limit what brainstorming loads as all its preloads don't matter

There is some further potential here, I'll highlight some and will check out some more one of these days for similar patterns.

@PragTob PragTob force-pushed the performance-improvements branch from bc36a24 to 966eed0 Compare November 30, 2024 20:03
@PragTob
Copy link
Collaborator Author

PragTob commented Nov 30, 2024

I think we can easily remove 2 more database queries:

  def handle_event(
        "add_idea_label_to_idea",
        %{
          "idea-id" => idea_id,
          "idea-label-id" => idea_label_id
        },
        socket
      ) do
    idea = Ideas.get_idea!(idea_id)
    idea_label = IdeaLabels.get_idea_label(idea_label_id)

    IdeaLabels.add_idea_label_to_idea(idea, idea_label)
    {:noreply, socket}
  end

We don't need to load the idea or idea_label - we only need the ids. However, later IdeaLabels.add_idea_label_to_idea needs the brainstorming_id of the idea to broadcast the changes. But I guess it's in the socket as well (I haven't checked yet) so we could pass that on explicitly and remove the 2 queries here as well. That'd bring the query count down to 12 :)

@JannikStreek
Copy link
Member

JannikStreek commented Nov 30, 2024

Thanks, thats a good start of cleaning up! All the broadcasts can be improved a lot, we often load e.g. an idea or lane but in the end we just need an id. I think that happens actually in more places.

but this is due to the decision of broadcasting a full "Lane change" instead of the incremental equivalent update

This is an interesting one, where I am open for feedback. The reason why we often broadcast a whole lane or all lanes is, that if you drag an idea, you change the complete ordering of all ideas. So it's not enough to just update idea currently. This could be done just for content changes. But if the position is changed, we need to update at least the lane where the idea is on and maybe even more lanes, if the idea was moved from lane A to B. Thats what makes it complicated and this is a very easy but improvable solution.

One solution here would be to do the ordering on the socket / client side and not to rely on ordering them before the broadcast happens. Then I guess (?) just updating one idea would be sufficient. Then the question is, if you have 10 clients, if it's more efficient to order 10x instead of a single db query. But I think you gave me already the answer to this :D Feely free to have a look at this.

@JannikStreek
Copy link
Member

Lanes.get_lanes_for_brainstorming_with_labels_filtered also worth to look at as its required an id but often the brainstorming is already loaded before, so we are doing the same query 2 times.

@JannikStreek
Copy link
Member

Also interesting question in general: Passing ids vs passing objects (refs). Passing the id, you always need to refetch the object in the designated method. On the other hand, always passing the full object might lead to strange errors as for an update method you need the most recent object usually (just happened to me).

1. IdeaIdeaLabel: Only create a new join table entry vs. processing lots
We don't end up using the idea in the code that calls
`IdeaLabels.add_idea_label_to_ideas` - we broadcast and load
all the brainstormings and lanes again. So, there's no need
to adjust the `Idea` struct to be fine. This saves us some
queries (I think) but at the very least some code and some
CPU cycles.

2.Do not trigger preloads for brainstorming when unneeded
@PragTob PragTob force-pushed the performance-improvements branch from 966eed0 to ce34983 Compare December 23, 2024 11:38
@PragTob
Copy link
Collaborator Author

PragTob commented Dec 23, 2024

Yeah it's all a big game of tradeoffs.

  • lanes and position updates - the emitted event could just be "this card moved from this position to this position" and each client just does a quick manipulation of their in memory (or client side) data structure. It's a case that often happens and should be doable. It should be less load on the machines and also far less data to broadcast.
    However:
    • You always have the danger of things running out of sync with incremental errors - fully reloading gets rid of that/creates a self-healing system
    • it's a special implementation every time around, if the performance isn't an issue, which I'm not sure it is right now - all the effort and potential bugs maybe/probably aren't worth it - updating everything everywhere is easier

In short, complexy and increased chances for bugs vs. lower load on the system and quicker response times. For the kind of project this is, I'd assume that going with the simpler "safer" implementation is fine. That said, it feels a bit "wasteful" in my mind.

  • ids vs. objects - it's an interesting/case by case basis. My default is "do I need the whole object? No? Then let's pass an id?" that said, passing the object is a safe default/assumption and only switching to the ID when needed. Here I just noticed there were a bunch of queries and so I went searching/hunting :)
    Passing the object can also be preferable since you avoid double fetching (i.e. 2 funtions get passed the id, but load their own record) and you can do a preload data for all needed associations (that said, if an association is already present it won't be refetched by default by Repo.preload).

Some functions are also built to handle both through a pattern match in the function head to deal with both being passed an id or being passed the full record.

* idea is already part of the state of the component (and we only
need id and brainstorming_id)
* idea_label we only need the id anyhow
@PragTob
Copy link
Collaborator Author

PragTob commented Dec 23, 2024

@JannikStreek should be ready!

@JannikStreek JannikStreek merged commit fc8729a into master Dec 23, 2024
8 checks passed
@JannikStreek JannikStreek deleted the performance-improvements branch December 23, 2024 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants