diff --git a/docs/reference.md b/docs/reference.md index 3b974f574..465b20975 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -10,3 +10,4 @@ The reference documentation are available for this module: * [Operations](./reference/Operations.md) +* [Rolling Update](./reference/RollingUpdate.md) diff --git a/docs/reference/RollingUpdate.md b/docs/reference/RollingUpdate.md new file mode 100644 index 000000000..6ff30e368 --- /dev/null +++ b/docs/reference/RollingUpdate.md @@ -0,0 +1,172 @@ +# Rolling Update + +When the scheduler is updated, minor of major, the switch version operation will +simply change the new active version in the database. Thus, who is responsible for +actually performing runtime changes to enforce that all existing Game Rooms are from +the active schedulers is the _health_controller_ operation. + +This has some benefits attached to it: + +1. Consolidates all game room operations in the Add and Remove operations +2. If the update fails, rollback is more smooth. Previously, in the +_switch_version_ opeartion, if any creation/deletion of GRs failed, we would +sequentially delete all created game rooms. This can hog the worker in this +operation (no upscale during this time) and also the delete was not taking into +consideration ready target, thus it could heavily offend it +3. Each _health_controller_ loop is able to adjust how much game rooms it asks for +creation and process operations in between, avoiding the worker to be hogged in a +single operation + +## Add Rooms Limit Impacts + +Keep in mind that the _add_rooms_ operation has its own limit of rooms that can create +per operation, thus if more rooms than the limit is requested Maestro will cap to this +limit and on the next cycle a new set of rooms will be requested. + +For example, if on the first cycle the autoscale asks for 1000 rooms and limit is set +to 150, then only 150 are created and in the next cycle 850 rooms (assuming the +autoscale compute is the same) will be requested - until the desired number of rooms is +reached. + +## Old behavior + +1. A new scheduler version is created or a call to switch the active version is made +2. _new_version_ operation starts by the worker +3. A new validation room is created to validate the scheduler config. If this room +becomes ready, the new version is marked as safe to be deployed +4. _switch_version_ operation starts by the worker, other operations will wait until +this one finishes (no autoscaling) +5. Maestro spawns a `maxSurge` amount of goroutines reading from a channel. Each +goroutine creates a game room with the new config and deletes a game room from +previous config +6. When all goroutines finish processing, meaning that there aren't any more GRs +from the old version to be deleted (channel closed), the operation changes the +active version in the database +7. Operation _switch_version_ finishes and other operations can run + +If at any point there is an error in one of the goroutines or other part of the +process, worker starts to rollback. The rollback mechanism will delete all newly +created game rooms sequentially, even if this means that the system will have no +available game rooms by the end of it. + +## New behavior + +The new behavior is heavily inspired by how Kubernetes perform rolling updates in +deployments working gracefully alongside HPA (Horizontal Pod Autoscaler). Thus, two +parameters will guide this process: + +* `maxUnavailable`: how many ready pods below the desired can exist at a time in the +deployment. In other words, how many game rooms below `readyTarget` policy we can +delete. For now, Maestro will use `maxUnavailable: 0`, which means we never go below +the `readyTarget` at any given cycle of the update. This can make the process a bit +slower, but it ensures higher availability. +* `maxSurge`: how many Game Rooms can be created at once (per cycle) and how many +Game Rooms can exist above the desired. The later concept is important to understand +that we should go above the desired, otherwise we can not spawn a new Game Room +since it would offend the autoscaler/HPA that would try to delete it + +1. A new scheduler version is created or a call to switch the active version is made +2. _new_version_ operation starts by the worker +3. A new validation room is created to validate the scheduler config. If this room +becomes ready, the new version is marked as safe to be deployed +4. _switch_version_ operation starts by the worker, simply changing the active +version in the database +5. _health_controller_ operation runs and check if there is any existing game room +that is not from the active scheduler version +6. If it does not have, run normal autoscale. If it has, it is performing a rolling +update, proceed with the update +7. The update checks how many rooms it can spawn by computing the below +``` +maxSurge * totalAvailableRooms (pending + unready + ready + occupied) +``` +8. Enqueues a priority _add_room_ operation to create the surge amount +9. Check how many old rooms it can delete by computing +``` +currentNumberOfReadyRooms - desiredAmountOfReadyRooms (autoscale) +``` +10. If it can delete, enqueue a _delete_room_ operation. The above is valid for +`maxUnavailable: 0` so we never offend the `readyTarget`. Rooms are deleted by ID +since Maestro must delete only the rooms that are not from the active scheduler +version. Also, the occupied rooms will be the last one deleted from the list. +11. One _health_controller_ cycle finishes running the rolling update +12. _health_controller_ runs as many cycles as needed creating and deleting until +there are no more rooms from non-active scheduler versions to be deleted. When this +happens, rolling update finishes and _health_controller_ performs normal autoscale + +# Scenarios + +Below you will find how rolling update will perform in different scenarios when updating the schduler. Use as a reference to observe the behavior and tune parameters accordingly. + +The scenarios assume that all rooms created in the surge will transition to ready +in the next loop, which usually runs each 30s to 1min (depends on the +configuration). In reality, depending on the number of rooms to surge, runtime might +take longer to provision a node or game code might take a while to initialize +and room actually becoming ready. + +Also, the number of occupied rooms will remain the same, which means that when an +occupied Game Room from a previous version is deleted, a new one that was ready +transitions to occupied just for the sake of simplicity in the computation of +numbers of the scenario. In reality, the number of occupied rooms will vary +throughout the cycle and rolling update will adjust to that as well. + +## Few Amount of Game Room + +* readyTarget: 0.5 +* maxSurge: 25% + +### Downscale + +| **loop** | **ready** | **occupied** | **available** | **desired** | **desiredReady** | **toSurge** | **toBeDeleted** | +|----------|-----------|--------------|---------------|-------------|------------------|-------------|-----------------| +| **1** | 20 | 5 | 25 (0 new) | 10 | 5 | 7 | 15 | +| **2** | 12 | 5 | 17 (7 new) | 10 | 5 | 4 | 7 | +| **3** | 9 | 5 | 14 (11 new) | 10 | 5 | 3 | 3 (4 actually) | +| **4** | 9 | 5 | 14 (14 new) | 10 | 5 | - | 4 by autoscale | +| **5** | 5 | 5 | 10 (10 new) | 10 | 5 | - | - | + +### Upscale + +| **loop** | **ready** | **occupied** | **available** | **desired** | **desiredReady** | **toSurge** | **toBeDeleted** | +|----------|-----------|--------------|---------------|-------------|------------------|-------------|-----------------| +| **1** | 5 | 20 | 25 (0 new) | 40 | 20 | 7 | 0 | +| **2** | 12 | 20 | 32 (7 new) | 40 | 20 | 8 | 0 | +| **3** | 20 | 20 | 40 (15 new) | 40 | 20 | 10 | 0 | +| **4** | 30 | 20 | 50 (25 new) | 40 | 20 | 13 | 10 | +| **5** | 33 | 20 | 53 (38 new) | 40 | 20 | 14 | 13 | +| **6** | 34 | 20 | 54 (52 new) | 40 | 20 | 14 | 2 (actual 14) | +| **7** | 46 | 20 | 66 (66 new) | 40 | 20 | - | 26 by autoscale | +| **8** | 20 | 20 | 40 (40 new) | 40 | 20 | - | - | + +## Big Amount of Game Rooms + +### Upscale + +* readyTarget: 0.4 -> 0.7 +* maxSurge: 25% + +| **loop** | **ready** | **occupied** | **available** | **desired** | **desiredReady** | **toSurge** | **toBeDeleted** | +|----------|-----------|--------------|-----------------|-------------|------------------|-------------|------------------| +| **1** | 209 | 458 | 667 (0 new) | 1526 | 1068 | 167 | 0 | +| **2** | 376 | 458 | 834 (167 new) | 1526 | 1068 | 209 | 0 | +| **3** | 585 | 458 | 1043 (376 new) | 1526 | 1068 | 261 | 0 | +| **4** | 846 | 458 | 1304 (637 new) | 1526 | 1068 | 326 | 0 | +| **5** | 1172 | 458 | 1630 (963 new) | 1526 | 1068 | 408 | 104 | +| **6** | 1476 | 458 | 1934 (1371 new) | 1526 | 1068 | 484 | 408 | +| **7** | 1552 | 458 | 2010 (1855 new) | 1526 | 1068 | 503 | 155 | +| **8** | 2055 | 458 | 2513 (2513 new) | 1526 | 1068 | - | 987 by autoscale | +| **9** | 1068 | 458 | 1526 (1526 new) | 1526 | 1068 | - | - | + +### Downscale + +* readyTarget: 0.7 -> 0.4 +* maxSurge: 25% + +| **loop** | **ready** | **occupied** | **available** | **desired** | **desiredReady** | **toSurge** | **toBeDeleted** | +|----------|-----------|--------------|-----------------|-------------|------------------|-------------|-------------------| +| **1** | 940 | 1040 | 1980 (0 new) | 1733 | 693 | 495 | 247 | +| **2** | 1188 | 1040 | 2228 (495 new) | 1733 | 693 | 557 | 495 | +| **3** | 1250 | 1040 | 2290 (1052 new) | 1733 | 693 | 573 | 557 | +| **4** | 1266 | 1040 | 2306 (1625 new) | 1733 | 693 | 577 | 573 | +| **5** | 1270 | 1040 | 2310 (2202 new) | 1733 | 693 | 578 | 108 (577) | +| **6** | 1740 | 1040 | 2780 (2780 new) | 1733 | 693 | - | 1047 by autoscale | +| **7** | 693 | 1040 | 1733 (1733 new) | 1733 | 693 | - | - | diff --git a/internal/core/operations/healthcontroller/executor.go b/internal/core/operations/healthcontroller/executor.go index 043db00c9..d33509a46 100644 --- a/internal/core/operations/healthcontroller/executor.go +++ b/internal/core/operations/healthcontroller/executor.go @@ -51,6 +51,7 @@ type Config struct { type Executor struct { autoscaler ports.Autoscaler roomStorage ports.RoomStorage + roomManager ports.RoomManager instanceStorage ports.GameRoomInstanceStorage schedulerStorage ports.SchedulerStorage operationManager ports.OperationManager @@ -60,10 +61,20 @@ type Executor struct { var _ operations.Executor = (*Executor)(nil) // NewExecutor creates a new instance of Executor. -func NewExecutor(roomStorage ports.RoomStorage, instanceStorage ports.GameRoomInstanceStorage, schedulerStorage ports.SchedulerStorage, operationManager ports.OperationManager, autoscaler ports.Autoscaler, config Config) *Executor { +func NewExecutor( + roomStorage ports.RoomStorage, + roomManager ports.RoomManager, + instanceStorage ports.GameRoomInstanceStorage, + schedulerStorage ports.SchedulerStorage, + operationManager ports.OperationManager, + autoscaler ports.Autoscaler, + config Config, +) *Executor { return &Executor{ - autoscaler: autoscaler, + autoscaler: autoscaler, + // TODO: replace roomStorage operations with roomManager roomStorage: roomStorage, + roomManager: roomManager, instanceStorage: instanceStorage, schedulerStorage: schedulerStorage, operationManager: operationManager, @@ -99,7 +110,7 @@ func (ex *Executor) Execute(ctx context.Context, op *operation.Operation, defini if len(expiredRooms) > 0 { logger.Sugar().Infof("found %v expired rooms to be deleted", len(expiredRooms)) - err = ex.enqueueRemoveExpiredRooms(ctx, op, logger, expiredRooms) + err = ex.enqueueRemoveRooms(ctx, op, logger, expiredRooms) if err != nil { logger.Error("could not enqueue operation to delete expired rooms", zap.Error(err)) } @@ -113,6 +124,11 @@ func (ex *Executor) Execute(ctx context.Context, op *operation.Operation, defini } reportDesiredNumberOfRooms(scheduler.Game, scheduler.Name, desiredNumberOfRooms) + // Check if the system is in a rollingUpdate by listing rooms that are not the current scheduler version + roomsPreviousSchedulerVersion, isRollingUpdate := ex.checkRollingUpdate(ctx, logger, scheduler, availableRooms) + if isRollingUpdate { + return ex.performRollingUpdate(ctx, op, def, logger, scheduler, desiredNumberOfRooms, availableRooms, roomsPreviousSchedulerVersion) + } err = ex.ensureDesiredAmountOfInstances(ctx, op, def, scheduler, logger, len(availableRooms), desiredNumberOfRooms) if err != nil { logger.Error("cannot ensure desired amount of instances", zap.Error(err)) @@ -280,16 +296,16 @@ func (ex *Executor) isRoomStatus(room *game_room.GameRoom, status game_room.Game return room.Status == status } -func (ex *Executor) enqueueRemoveExpiredRooms(ctx context.Context, op *operation.Operation, logger *zap.Logger, expiredRoomsIDs []string) error { +func (ex *Executor) enqueueRemoveRooms(ctx context.Context, op *operation.Operation, logger *zap.Logger, roomsIDs []string) error { removeOperation, err := ex.operationManager.CreatePriorityOperation(ctx, op.SchedulerName, &remove.Definition{ - RoomsIDs: expiredRoomsIDs, + RoomsIDs: roomsIDs, Reason: remove.Expired, }) if err != nil { return err } - msgToAppend := fmt.Sprintf("created operation (id: %s) to remove %v expired rooms.", removeOperation.ID, len(expiredRoomsIDs)) + msgToAppend := fmt.Sprintf("created operation (id: %s) to remove %v rooms.", removeOperation.ID, len(roomsIDs)) logger.Info(msgToAppend) ex.operationManager.AppendOperationEventToExecutionHistory(ctx, op, msgToAppend) @@ -371,3 +387,137 @@ func (ex *Executor) canPerformDownscale(ctx context.Context, scheduler *entities return can && !waitingCooldown, "ok" } + +func (ex *Executor) checkRollingUpdate( + ctx context.Context, + logger *zap.Logger, + scheduler *entities.Scheduler, + availableRoomsIDs []string, +) ([]string, bool) { + logger.Debug("checking if system is in the middle of a rolling update of scheduler") + var roomsPreviousScheduler, occupiedRoomsPreviousScheduler []string + // TODO: build this struct during findAvailableAndExpiredRooms() call + for _, roomID := range availableRoomsIDs { + room, err := ex.roomStorage.GetRoom(ctx, scheduler.Name, roomID) + // if err != nil we will miss the room, the system can still recover itself in + // the next health_controller operation + if err == nil && room.Version != scheduler.Spec.Version { + if room.Status == game_room.GameStatusOccupied { + occupiedRoomsPreviousScheduler = append(occupiedRoomsPreviousScheduler, roomID) + } else { + roomsPreviousScheduler = append(roomsPreviousScheduler, roomID) + } + } + } + // Append occupied to the end so when deleting we prioritize non-occupied rooms + roomsPreviousScheduler = append(roomsPreviousScheduler, occupiedRoomsPreviousScheduler...) + logger.Info("rooms that did not match current scheduler versions", zap.Int("rooms", len(roomsPreviousScheduler))) + return roomsPreviousScheduler, len(roomsPreviousScheduler) != 0 +} + +func (ex *Executor) performRollingUpdate( + ctx context.Context, + op *operation.Operation, + def *Definition, + logger *zap.Logger, + scheduler *entities.Scheduler, + desiredNumberOfRooms int, + availableRoomsIDs []string, + roomsWithPreviousSchedulerVersion []string, +) error { + logger.Info("performing rolling update", zap.String("scheduler.Version", scheduler.Spec.Version)) + maxSurgeAmount, err := ex.roomManager.SchedulerMaxSurge(ctx, scheduler) + if err != nil { + logger.Error("failed to perform rolling update while getting max surge amount of rooms", zap.Error(err)) + return err + } + if len(roomsWithPreviousSchedulerVersion) < maxSurgeAmount { + maxSurgeAmount = len(roomsWithPreviousSchedulerVersion) + } + if maxSurgeAmount <= 0 { + maxSurgeAmount = 1 + } + logger.Info( + "upscaling new rooms", + zap.Int("desired", desiredNumberOfRooms), + zap.Int("maxSurgeAmount", maxSurgeAmount), + zap.Int("available", len(availableRoomsIDs)), + zap.Int("oldRooms", len(roomsWithPreviousSchedulerVersion)), + ) + addOp, err := ex.operationManager.CreatePriorityOperation(ctx, op.SchedulerName, &add.Definition{ + Amount: int32(maxSurgeAmount), + }) + if err != nil { + logger.Error("failed to enqueue add operation for rolling update", zap.Error(err)) + return err + } + msgToAppend := fmt.Sprintf("created operation (id: %s) to surge %v rooms.", addOp.ID, maxSurgeAmount) + ex.operationManager.AppendOperationEventToExecutionHistory(ctx, op, msgToAppend) + ex.setTookAction(def, true) + + roomsMarkedForDeletion, err := ex.markPreviousSchedulerRoomsForDeletion( + ctx, + logger, + scheduler, + roomsWithPreviousSchedulerVersion, + desiredNumberOfRooms, + ) + if err != nil { + logger.Error("could not delete rooms with previous scheduler version", zap.Error(err)) + return err + } + if len(roomsMarkedForDeletion) <= 0 { + logger.Info("no rooms marked for deletion", zap.Int("roomsMarkedForDeletion", len(roomsMarkedForDeletion))) + return nil + } + removeOp, err := ex.operationManager.CreateOperation(ctx, op.SchedulerName, &remove.Definition{ + RoomsIDs: roomsMarkedForDeletion, + Reason: remove.RollingUpdateReplace, + }) + if err != nil { + logger.Error("failed to enqueue remove operation for rolling update", zap.Error(err)) + return err + } + msgToAppend = fmt.Sprintf("created operation (id: %s) to remove rooms with previous scheduler version.", removeOp.ID) + ex.operationManager.AppendOperationEventToExecutionHistory(ctx, op, msgToAppend) + ex.setTookAction(def, true) + + return nil +} + +func (ex *Executor) markPreviousSchedulerRoomsForDeletion( + ctx context.Context, + logger *zap.Logger, + scheduler *entities.Scheduler, + roomsWithPreviousSchedulerVersion []string, + desiredNumberOfTotalRooms int, +) ([]string, error) { + curReadyRooms, err := ex.roomStorage.GetRoomIDsByStatus(ctx, scheduler.Name, game_room.GameStatusReady) + if err != nil { + return []string{}, fmt.Errorf("failed to list scheduler rooms on ready status: %w", err) + } + curOccupiedRooms, err := ex.roomStorage.GetRoomIDsByStatus(ctx, scheduler.Name, game_room.GameStatusOccupied) + if err != nil { + return []string{}, fmt.Errorf("failed to list scheduler rooms on occupied status: %w", err) + } + desiredNumberOfReadyRooms := desiredNumberOfTotalRooms - len(curOccupiedRooms) + bufferRoomsToBeRemoved := len(curReadyRooms) - desiredNumberOfReadyRooms + logger = logger.With( + zap.Int("roomsWithPreviousSchedulerVersion", len(roomsWithPreviousSchedulerVersion)), + zap.Int("currentOccupiedRooms", len(curOccupiedRooms)), + zap.Int("currentReadyRooms", len(curReadyRooms)), + zap.Int("desiredNumberOfReadyRooms", desiredNumberOfReadyRooms), + zap.Int("desiredNumberOfTotalRooms", desiredNumberOfTotalRooms), + zap.Int("bufferRoomsToBeRemoved", bufferRoomsToBeRemoved), + ) + if bufferRoomsToBeRemoved < 0 { + logger.Info("can not delete old rooms without offending maxUnavailable: 0") + return []string{}, nil + } + if bufferRoomsToBeRemoved > len(roomsWithPreviousSchedulerVersion) { + logger.Info("less rooms on previous scheduler version than the amount to delete, capping it") + return roomsWithPreviousSchedulerVersion, nil + } + logger.Sugar().Infof("successfully marked %d rooms for deletion", bufferRoomsToBeRemoved) + return roomsWithPreviousSchedulerVersion[:bufferRoomsToBeRemoved], nil +} diff --git a/internal/core/operations/healthcontroller/executor_test.go b/internal/core/operations/healthcontroller/executor_test.go index 1241c722e..5f661bfab 100644 --- a/internal/core/operations/healthcontroller/executor_test.go +++ b/internal/core/operations/healthcontroller/executor_test.go @@ -51,6 +51,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { type executionPlan struct { planMocks func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -93,6 +94,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: false, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -113,6 +115,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: false, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -145,6 +148,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -154,10 +158,14 @@ func TestSchedulerHealthController_Execute(t *testing.T) { Status: game_room.GameStatusPending, LastPingAt: time.Now(), CreatedAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(gameRoomPending, nil) genericSchedulerNoAutoscaling.RoomsReplicas = 2 + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(gameRoomPending, nil) }, }, }, @@ -168,6 +176,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -198,6 +207,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -208,6 +218,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { Status: game_room.GameStatusPending, LastPingAt: time.Now(), CreatedAt: expiredCreatedAt, + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(gameRoomPending, nil) @@ -219,6 +230,8 @@ func TestSchedulerHealthController_Execute(t *testing.T) { operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &add.Definition{Amount: 1}).Return(op, nil) genericSchedulerNoAutoscaling.RoomsReplicas = 2 + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -229,6 +242,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -259,6 +273,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -269,6 +284,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { Status: game_room.GameStatusUnready, LastPingAt: time.Now(), CreatedAt: expiredCreatedAt, + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(gameRoomUnready, nil) @@ -280,6 +296,8 @@ func TestSchedulerHealthController_Execute(t *testing.T) { operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &add.Definition{Amount: 1}).Return(op, nil) genericSchedulerNoAutoscaling.RoomsReplicas = 2 + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -290,6 +308,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: false, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -322,6 +341,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { Status: game_room.GameStatusReady, LastPingAt: time.Now(), CreatedAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -331,10 +351,14 @@ func TestSchedulerHealthController_Execute(t *testing.T) { Status: game_room.GameStatusUnready, LastPingAt: time.Now(), CreatedAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(gameRoomUnready, nil) genericSchedulerNoAutoscaling.RoomsReplicas = 2 + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(gameRoomUnready, nil) }, }, }, @@ -345,6 +369,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: false, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -371,12 +396,15 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) genericSchedulerNoAutoscaling.RoomsReplicas = 1 roomStorage.EXPECT().DeleteRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(nil) instanceStorage.EXPECT().DeleteInstance(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(nil) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -387,6 +415,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: false, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -413,6 +442,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -421,6 +451,8 @@ func TestSchedulerHealthController_Execute(t *testing.T) { roomStorage.EXPECT().DeleteRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[2]).Return(nil) instanceStorage.EXPECT().DeleteInstance(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(errors.New("error")) instanceStorage.EXPECT().DeleteInstance(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[2]).Return(nil) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -431,6 +463,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -463,6 +496,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -472,6 +506,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now().Add(-time.Minute * 60), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(expiredGameRoom, nil) @@ -479,6 +514,8 @@ func TestSchedulerHealthController_Execute(t *testing.T) { op := operation.New(genericSchedulerNoAutoscaling.Name, definition.Name(), nil) operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()) operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &remove.Definition{RoomsIDs: []string{gameRoomIDs[1]}, Reason: remove.Expired}).Return(op, nil) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -489,6 +526,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: false, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -503,12 +541,23 @@ func TestSchedulerHealthController_Execute(t *testing.T) { }, }, } + gameRoom := &game_room.GameRoom{ + ID: gameRoomIDs[0], + SchedulerID: genericSchedulerAutoscalingEnabled.Name, + Status: game_room.GameStatusReady, + LastPingAt: time.Now(), + Version: genericSchedulerAutoscalingEnabled.Spec.Version, + } // load roomStorage.EXPECT().GetAllRoomIDs(gomock.Any(), gomock.Any()).Return(gameRoomIDs, nil) instanceStorage.EXPECT().GetAllInstances(gomock.Any(), gomock.Any()).Return(instances, nil) - schedulerStorage.EXPECT().GetScheduler(gomock.Any(), gomock.Any()).Return(genericSchedulerNoAutoscaling, nil) + schedulerStorage.EXPECT().GetScheduler(gomock.Any(), gomock.Any()).Return(genericSchedulerAutoscalingEnabled, nil) - operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), "current amount of rooms is equal to desired amount, no changes needed") + // Ensure current scheduler version + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerAutoscalingEnabled.Name, gameRoomIDs[0]).Return(gameRoom, nil) + + autoscaler.EXPECT().CalculateDesiredNumberOfRooms(gomock.Any(), genericSchedulerAutoscalingEnabled).Return(1, nil) + operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()) }, }, }, @@ -519,6 +568,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -550,6 +600,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -559,6 +610,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now().Add(-time.Minute * 60), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(expiredGameRoom, nil) @@ -568,6 +620,8 @@ func TestSchedulerHealthController_Execute(t *testing.T) { op := operation.New(genericSchedulerNoAutoscaling.Name, definition.Name(), nil) operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()) operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &add.Definition{Amount: 1}).Return(op, nil) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -578,6 +632,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -594,6 +649,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { op := operation.New(genericSchedulerNoAutoscaling.Name, definition.Name(), nil) operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()) operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &add.Definition{Amount: 2}).Return(op, nil) + }, }, }, @@ -604,6 +660,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -630,6 +687,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -656,6 +714,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -670,6 +729,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { genericSchedulerNoAutoscaling.RoomsReplicas = 2 operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &add.Definition{Amount: 2}).Return(nil, errors.New("error")) + }, shouldFail: true, }, @@ -681,6 +741,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -708,6 +769,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -715,6 +777,8 @@ func TestSchedulerHealthController_Execute(t *testing.T) { op := operation.New(genericSchedulerNoAutoscaling.Name, definition.Name(), nil) operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()) operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &remove.Definition{Amount: 1, Reason: remove.ScaleDown}).Return(op, nil) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -725,6 +789,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -752,6 +817,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerAutoscalingDisabled.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerAutoscalingDisabled.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerAutoscalingDisabled.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -759,6 +825,8 @@ func TestSchedulerHealthController_Execute(t *testing.T) { op := operation.New(genericSchedulerAutoscalingDisabled.Name, definition.Name(), nil) operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()) operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerAutoscalingDisabled.Name, &remove.Definition{Amount: 1, Reason: remove.ScaleDown}).Return(op, nil) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerAutoscalingDisabled.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -769,6 +837,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -797,12 +866,15 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerAutoscalingEnabled.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerAutoscalingEnabled.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerAutoscalingEnabled.Name, gameRoomIDs[0]).Return(gameRoom, nil) op := operation.New(genericSchedulerAutoscalingEnabled.Name, definition.Name(), nil) operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()) operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerAutoscalingEnabled.Name, &remove.Definition{Amount: 1, Reason: remove.ScaleDown}).Return(op, nil) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerAutoscalingEnabled.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -813,6 +885,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: false, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -840,9 +913,12 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerAutoscalingEnabled.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerAutoscalingEnabled.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerAutoscalingEnabled.Name, gameRoomIDs[0]).Return(gameRoom, nil) operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerAutoscalingEnabled.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -853,6 +929,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -880,11 +957,14 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) genericSchedulerNoAutoscaling.RoomsReplicas = 0 operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &remove.Definition{Amount: 1, Reason: remove.ScaleDown}).Return(nil, errors.New("error")) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, shouldFail: true, }, @@ -896,6 +976,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -913,6 +994,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -931,6 +1013,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -950,6 +1033,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -976,6 +1060,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerAutoscalingEnabled.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerAutoscalingEnabled.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerAutoscalingEnabled.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, @@ -989,6 +1074,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -1019,6 +1105,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(nil, errors.New("error")) @@ -1027,6 +1114,8 @@ func TestSchedulerHealthController_Execute(t *testing.T) { op := operation.New(genericSchedulerNoAutoscaling.Name, definition.Name(), nil) operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()) operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &add.Definition{Amount: 1}).Return(op, nil) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -1037,6 +1126,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -1067,6 +1157,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -1075,6 +1166,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusError, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(gameRoomError, nil) @@ -1082,6 +1174,8 @@ func TestSchedulerHealthController_Execute(t *testing.T) { op := operation.New(genericSchedulerNoAutoscaling.Name, definition.Name(), nil) operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()) operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &add.Definition{Amount: 1}).Return(op, nil) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -1092,6 +1186,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -1122,6 +1217,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -1130,6 +1226,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusTerminating, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(gameRoomTerminating, nil) @@ -1137,6 +1234,8 @@ func TestSchedulerHealthController_Execute(t *testing.T) { op := operation.New(genericSchedulerNoAutoscaling.Name, definition.Name(), nil) operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()) operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &add.Definition{Amount: 1}).Return(op, nil) + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) }, }, }, @@ -1147,6 +1246,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { tookAction: true, planMocks: func( roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, instanceStorage *mockports.MockGameRoomInstanceStorage, schedulerStorage *mockports.MockSchedulerStorage, operationManager *mockports.MockOperationManager, @@ -1177,6 +1277,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { SchedulerID: genericSchedulerNoAutoscaling.Name, Status: game_room.GameStatusReady, LastPingAt: time.Now(), + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) @@ -1187,6 +1288,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { Status: game_room.GameStatusTerminating, LastPingAt: time.Now().Add(5 * -time.Minute), CreatedAt: expiredCreatedAt, + Version: genericSchedulerNoAutoscaling.Spec.Version, } roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[1]).Return(gameRoomTerminating, nil) @@ -1198,6 +1300,85 @@ func TestSchedulerHealthController_Execute(t *testing.T) { operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), genericSchedulerNoAutoscaling.Name, &add.Definition{Amount: 1}).Return(op, nil) genericSchedulerNoAutoscaling.RoomsReplicas = 2 + + roomStorage.EXPECT().GetRoom(gomock.Any(), genericSchedulerNoAutoscaling.Name, gameRoomIDs[0]).Return(gameRoom, nil) + }, + }, + }, + { + title: "room scheduler version do not match current scheduler, start rolling update and not autoscale", + definition: &healthcontroller.Definition{}, + executionPlan: executionPlan{ + tookAction: true, + planMocks: func( + roomStorage *mockports.MockRoomStorage, + roomManager *mockports.MockRoomManager, + instanceStorage *mockports.MockGameRoomInstanceStorage, + schedulerStorage *mockports.MockSchedulerStorage, + operationManager *mockports.MockOperationManager, + autoscaler *mockports.MockAutoscaler, + ) { + gameRoomIDs := []string{"room-1", "room-2"} + instances := []*game_room.Instance{ + { + ID: "room-1", + Status: game_room.InstanceStatus{ + Type: game_room.InstanceReady, + }, + }, + { + ID: "room-2", + Status: game_room.InstanceStatus{ + Type: game_room.InstanceReady, + }, + }, + } + newScheduler := newValidScheduler(&autoscalingEnabled) + newScheduler.Spec.Version = "v2" + gameRoom1 := &game_room.GameRoom{ + ID: gameRoomIDs[0], + SchedulerID: genericSchedulerAutoscalingEnabled.Name, + Status: game_room.GameStatusReady, + LastPingAt: time.Now(), + CreatedAt: time.Now(), + Version: genericSchedulerAutoscalingEnabled.Spec.Version, + } + gameRoom2 := &game_room.GameRoom{ + ID: gameRoomIDs[1], + SchedulerID: genericSchedulerAutoscalingEnabled.Name, + Status: game_room.GameStatusOccupied, + LastPingAt: time.Now(), + Version: genericSchedulerAutoscalingEnabled.Spec.Version, + } + + // load + roomStorage.EXPECT().GetAllRoomIDs(gomock.Any(), gomock.Any()).Return(gameRoomIDs, nil) + instanceStorage.EXPECT().GetAllInstances(gomock.Any(), gomock.Any()).Return(instances, nil) + + // findAvailableAndExpiredRooms + roomStorage.EXPECT().GetRoom(gomock.Any(), newScheduler.Name, gameRoomIDs[0]).Return(gameRoom1, nil) + roomStorage.EXPECT().GetRoom(gomock.Any(), newScheduler.Name, gameRoomIDs[1]).Return(gameRoom2, nil) + + // getDesiredNumberOfRooms + schedulerStorage.EXPECT().GetScheduler(gomock.Any(), gomock.Any()).Return(newScheduler, nil) + autoscaler.EXPECT().CalculateDesiredNumberOfRooms(gomock.Any(), newScheduler).Return(2, nil) + + // Check for rolling update + roomStorage.EXPECT().GetRoom(gomock.Any(), newScheduler.Name, gameRoomIDs[0]).Return(gameRoom1, nil) + roomStorage.EXPECT().GetRoom(gomock.Any(), newScheduler.Name, gameRoomIDs[1]).Return(gameRoom2, nil) + + op := operation.New(newScheduler.Name, definition.Name(), nil) + + // Perform rolling update + roomManager.EXPECT().SchedulerMaxSurge(gomock.Any(), newScheduler).Return(1, nil) + operationManager.EXPECT().CreatePriorityOperation(gomock.Any(), newScheduler.Name, &add.Definition{Amount: 1}).Return(op, nil) + operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()).Times(2) + roomStorage.EXPECT().GetRoomIDsByStatus(gomock.Any(), newScheduler.Name, game_room.GameStatusOccupied).Return(gameRoomIDs, nil) + roomStorage.EXPECT().GetRoomIDsByStatus(gomock.Any(), newScheduler.Name, game_room.GameStatusReady).Return(gameRoomIDs, nil) + operationManager.EXPECT().CreateOperation(gomock.Any(), newScheduler.Name, &remove.Definition{RoomsIDs: gameRoomIDs, Reason: remove.RollingUpdateReplace}).Return(op, nil) + + // Shouldn't call autoscale + schedulerStorage.EXPECT().UpdateScheduler(gomock.Any(), gomock.Any()).Times(0) }, }, }, @@ -1207,6 +1388,7 @@ func TestSchedulerHealthController_Execute(t *testing.T) { t.Run(testCase.title, func(t *testing.T) { mockCtrl := gomock.NewController(t) roomsStorage := mockports.NewMockRoomStorage(mockCtrl) + roomManager := mockports.NewMockRoomManager(mockCtrl) instanceStorage := mockports.NewMockGameRoomInstanceStorage(mockCtrl) schedulerStorage := mockports.NewMockSchedulerStorage(mockCtrl) operationManager := mockports.NewMockOperationManager(mockCtrl) @@ -1216,9 +1398,9 @@ func TestSchedulerHealthController_Execute(t *testing.T) { RoomInitializationTimeout: 4 * time.Minute, RoomDeletionTimeout: 4 * time.Minute, } - executor := healthcontroller.NewExecutor(roomsStorage, instanceStorage, schedulerStorage, operationManager, autoscaler, config) + executor := healthcontroller.NewExecutor(roomsStorage, roomManager, instanceStorage, schedulerStorage, operationManager, autoscaler, config) - testCase.executionPlan.planMocks(roomsStorage, instanceStorage, schedulerStorage, operationManager, autoscaler) + testCase.executionPlan.planMocks(roomsStorage, roomManager, instanceStorage, schedulerStorage, operationManager, autoscaler) ctx := context.Background() err := executor.Execute(ctx, genericOperation, testCase.definition) diff --git a/internal/core/operations/providers/operation_providers.go b/internal/core/operations/providers/operation_providers.go index a21c7beaf..ea61e882b 100644 --- a/internal/core/operations/providers/operation_providers.go +++ b/internal/core/operations/providers/operation_providers.go @@ -95,9 +95,9 @@ func ProvideExecutors( executors[addrooms.OperationName] = addrooms.NewExecutor(roomManager, schedulerStorage, operationManager, addRoomsConfig) executors[removerooms.OperationName] = removerooms.NewExecutor(roomManager, roomStorage, operationManager) executors[test.OperationName] = test.NewExecutor() - executors[switchversion.OperationName] = switchversion.NewExecutor(roomManager, schedulerManager, operationManager, roomStorage) + executors[switchversion.OperationName] = switchversion.NewExecutor(schedulerManager, operationManager) executors[newversion.OperationName] = newversion.NewExecutor(roomManager, schedulerManager, operationManager, newSchedulerVersionConfig) - executors[healthcontroller.OperationName] = healthcontroller.NewExecutor(roomStorage, instanceStorage, schedulerStorage, operationManager, autoscaler, healthControllerConfig) + executors[healthcontroller.OperationName] = healthcontroller.NewExecutor(roomStorage, roomManager, instanceStorage, schedulerStorage, operationManager, autoscaler, healthControllerConfig) executors[storagecleanup.OperationName] = storagecleanup.NewExecutor(operationStorage) executors[deletescheduler.OperationName] = deletescheduler.NewExecutor(schedulerStorage, schedulerCache, instanceStorage, operationStorage, operationManager, runtime) diff --git a/internal/core/operations/rooms/remove/definition.go b/internal/core/operations/rooms/remove/definition.go index 0ec64011f..66555c1fe 100644 --- a/internal/core/operations/rooms/remove/definition.go +++ b/internal/core/operations/rooms/remove/definition.go @@ -38,6 +38,7 @@ const ( NewVersionValidationFinished string = "new_version_validation_finished" SwitchVersionRollback string = "switch_version_rollback" SwitchVersionReplace string = "switch_version_replace" + RollingUpdateReplace string = "rolling_update_replace" ) const OperationName = "remove_rooms" diff --git a/internal/core/operations/schedulers/switchversion/executor.go b/internal/core/operations/schedulers/switchversion/executor.go index d5ac58a9c..a9c39e1fd 100644 --- a/internal/core/operations/schedulers/switchversion/executor.go +++ b/internal/core/operations/schedulers/switchversion/executor.go @@ -25,56 +25,30 @@ package switchversion import ( "context" "fmt" - "sync" "github.com/topfreegames/maestro/internal/core/logs" - "github.com/topfreegames/maestro/internal/core/operations/rooms/remove" "github.com/topfreegames/maestro/internal/core/ports" - "github.com/avast/retry-go/v4" "github.com/topfreegames/maestro/internal/core/entities" - "github.com/topfreegames/maestro/internal/core/entities/game_room" "github.com/topfreegames/maestro/internal/core/entities/operation" "github.com/topfreegames/maestro/internal/core/operations" "go.uber.org/zap" - "golang.org/x/sync/errgroup" ) type Executor struct { - roomManager ports.RoomManager - schedulerManager ports.SchedulerManager - operationManager ports.OperationManager - roomStorage ports.RoomStorage - roomsBeingReplaced *sync.Map - newCreatedRooms map[string][]*game_room.GameRoom - newCreatedRoomsLock sync.Mutex + schedulerManager ports.SchedulerManager + operationManager ports.OperationManager } var _ operations.Executor = (*Executor)(nil) -func NewExecutor(roomManager ports.RoomManager, schedulerManager ports.SchedulerManager, operationManager ports.OperationManager, roomStorage ports.RoomStorage) *Executor { - // TODO(caio.rodrigues): change map to store a list of ids (less memory used) - newCreatedRoomsMap := make(map[string][]*game_room.GameRoom) - +func NewExecutor(schedulerManager ports.SchedulerManager, operationManager ports.OperationManager) *Executor { return &Executor{ - roomManager: roomManager, - schedulerManager: schedulerManager, - operationManager: operationManager, - roomStorage: roomStorage, - roomsBeingReplaced: &sync.Map{}, - newCreatedRooms: newCreatedRoomsMap, - newCreatedRoomsLock: sync.Mutex{}, + schedulerManager: schedulerManager, + operationManager: operationManager, } } -// Execute the process of switching a scheduler active version consists of the following: -// 1. Creates "replace" goroutines (same number as MaxSurge); -// 2. Each goroutine will listen to a channel and create a new room using the -// new configuration. After the room is ready, it will then delete the room -// being replaced; -// 3. List all game rooms that need to be replaced and produce them into the -// replace goroutines channel; -// 4. Switch the active version func (ex *Executor) Execute(ctx context.Context, op *operation.Operation, definition operations.Definition) error { logger := zap.L().With( zap.String(logs.LogFieldSchedulerName, op.SchedulerName), @@ -97,32 +71,6 @@ func (ex *Executor) Execute(ctx context.Context, op *operation.Operation, defini return getSchedulerErr } - replacePods, err := ex.shouldReplacePods(ctx, scheduler) - if err != nil { - logger.Error("error deciding if should replace pods", zap.Error(err)) - shouldReplacePodsErr := fmt.Errorf("error deciding if should replace pods: %w", err) - ex.operationManager.AppendOperationEventToExecutionHistory(ctx, op, shouldReplacePodsErr.Error()) - return shouldReplacePodsErr - } - - if replacePods { - maxSurgeNum, err := ex.roomManager.SchedulerMaxSurge(ctx, scheduler) - if err != nil { - logger.Error("error fetching scheduler max surge", zap.Error(err)) - maxSurgeErr := fmt.Errorf("error fetching scheduler max surge: %w", err) - ex.operationManager.AppendOperationEventToExecutionHistory(ctx, op, maxSurgeErr.Error()) - return maxSurgeErr - } - - err = ex.startReplaceRoomsLoop(ctx, logger, maxSurgeNum, *scheduler, op) - if err != nil { - logger.Error("error replacing rooms", zap.Error(err)) - replaceRoomsErr := fmt.Errorf("error replacing rooms: %w", err) - ex.operationManager.AppendOperationEventToExecutionHistory(ctx, op, replaceRoomsErr.Error()) - return replaceRoomsErr - } - } - logger.Sugar().Debugf("switching version to %v", scheduler.Spec.Version) scheduler.State = entities.StateInSync err = ex.schedulerManager.UpdateScheduler(ctx, scheduler) @@ -133,179 +81,14 @@ func (ex *Executor) Execute(ctx context.Context, op *operation.Operation, defini return updateSchedulerErr } - ex.clearNewCreatedRooms(op.SchedulerName) logger.Info("scheduler update finishes with success") return nil } func (ex *Executor) Rollback(ctx context.Context, op *operation.Operation, definition operations.Definition, executeErr error) error { - logger := zap.L().With( - zap.String(logs.LogFieldSchedulerName, op.SchedulerName), - zap.String(logs.LogFieldOperationDefinition, definition.Name()), - zap.String(logs.LogFieldOperationPhase, "Rollback"), - zap.String(logs.LogFieldOperationID, op.ID), - ) - logger.Info("starting Rollback routine") - - err := ex.deleteNewCreatedRooms(ctx, logger, op.SchedulerName, remove.SwitchVersionRollback) - ex.clearNewCreatedRooms(op.SchedulerName) - if err != nil { - logger.Error("error deleting newly created rooms", zap.Error(err)) - deleteCreatedRoomsErr := fmt.Errorf("error rolling back created rooms: %w", err) - ex.operationManager.AppendOperationEventToExecutionHistory(ctx, op, deleteCreatedRoomsErr.Error()) - return err - } - - logger.Info("finished Rollback routine") return nil } func (ex *Executor) Name() string { return OperationName } - -func (ex *Executor) deleteNewCreatedRooms(ctx context.Context, logger *zap.Logger, schedulerName string, reason string) error { - logger.Info("deleting created rooms since switching active version had error - start") - for _, room := range ex.newCreatedRooms[schedulerName] { - err := ex.roomManager.DeleteRoom(ctx, room, reason) - if err != nil { - logger.Error("failed to deleted recent created room", zap.Error(err)) - return err - } - logger.Sugar().Debugf("deleted room \"%s\" successfully", room.ID) - } - logger.Info("deleting created rooms since switching active version had error - end successfully") - return nil -} - -func (ex *Executor) startReplaceRoomsLoop(ctx context.Context, logger *zap.Logger, maxSurgeNum int, scheduler entities.Scheduler, op *operation.Operation) error { - logger.Info("replacing rooms loop - start") - roomsChan := make(chan *game_room.GameRoom) - errs, ctx := errgroup.WithContext(ctx) - - var totalRoomsAmount int - var err error - err = retry.Do(func() error { - totalRoomsAmount, err = ex.roomStorage.GetRoomCount(ctx, op.SchedulerName) - return err - }, retry.Attempts(10)) - if err != nil { - return err - } - - for i := 0; i < maxSurgeNum; i++ { - errs.Go(func() error { - return ex.replaceRoom(logger, roomsChan, ex.roomManager, scheduler) - }) - } - -roomsListLoop: - for { - var rooms []*game_room.GameRoom - err = retry.Do(func() error { - rooms, err = ex.roomManager.ListRoomsWithDeletionPriority(ctx, scheduler.Name, scheduler.Spec.Version, maxSurgeNum, ex.roomsBeingReplaced) - return err - }, retry.Attempts(10)) - - if err != nil { - return fmt.Errorf("failed to list rooms for deletion") - } - for _, room := range rooms { - ex.roomsBeingReplaced.Store(room.ID, true) - - select { - case roomsChan <- room: - case <-ctx.Done(): - break roomsListLoop - } - } - - ex.reportOperationProgress(ctx, logger, totalRoomsAmount, op) - - if len(rooms) == 0 { - break - } - } - - // close the rooms change and ensure all replace goroutines are gone - close(roomsChan) - - // Wait for possible errors from goroutines - if err := errs.Wait(); err != nil { - return err - } - ex.reportOperationProgress(ctx, logger, totalRoomsAmount, op) - logger.Info("replacing rooms loop - finish") - return nil -} - -func (ex *Executor) replaceRoom(logger *zap.Logger, roomsChan chan *game_room.GameRoom, roomManager ports.RoomManager, scheduler entities.Scheduler) error { - - // we're going to use a separated context for each replaceRoom since we - // don't want to cancel the replacement in the middle (like creating a room and - // then left the old one (without deleting it). - ctx := context.Background() - - for { - room, ok := <-roomsChan - if !ok { - return nil - } - - gameRoom, _, err := roomManager.CreateRoom(ctx, scheduler, false) - if err != nil { - logger.Error("error creating room", zap.Error(err)) - } - - err = roomManager.DeleteRoom(ctx, room, remove.SwitchVersionReplace) - if err != nil { - logger.Warn("failed to delete room", zap.Error(err)) - ex.roomsBeingReplaced.Delete(room.ID) - return err - } - - ex.roomsBeingReplaced.Delete(room.ID) - if gameRoom == nil { - return nil - } - - logger.Sugar().Debugf("replaced room \"%s\" with \"%s\"", room.ID, gameRoom.ID) - ex.appendToNewCreatedRooms(scheduler.Name, gameRoom) - } -} - -func (ex *Executor) appendToNewCreatedRooms(schedulerName string, gameRoom *game_room.GameRoom) { - ex.newCreatedRoomsLock.Lock() - defer ex.newCreatedRoomsLock.Unlock() - ex.newCreatedRooms[schedulerName] = append(ex.newCreatedRooms[schedulerName], gameRoom) -} - -func (ex *Executor) clearNewCreatedRooms(schedulerName string) { - delete(ex.newCreatedRooms, schedulerName) -} - -func (ex *Executor) shouldReplacePods(ctx context.Context, newScheduler *entities.Scheduler) (bool, error) { - actualActiveScheduler, err := ex.schedulerManager.GetActiveScheduler(ctx, newScheduler.Name) - if err != nil { - return false, err - } - return actualActiveScheduler.IsMajorVersion(newScheduler), nil -} - -func (ex *Executor) reportOperationProgress(ctx context.Context, logger *zap.Logger, totalAmount int, op *operation.Operation) { - if totalAmount == 0 { - return - } - - amountReplaced := ex.amountReplaced(op.SchedulerName) - currentPercentageRate := 100 * amountReplaced / totalAmount - - msg := fmt.Sprintf("Conclusion: %v%%. Amount of rooms replaced: %v", currentPercentageRate, amountReplaced) - logger.Debug(msg) - ex.operationManager.AppendOperationEventToExecutionHistory(ctx, op, msg) -} - -func (ex *Executor) amountReplaced(schedulerName string) int { - amountReplaced := len(ex.newCreatedRooms[schedulerName]) - return amountReplaced -} diff --git a/internal/core/operations/schedulers/switchversion/executor_test.go b/internal/core/operations/schedulers/switchversion/executor_test.go index aad0f6076..87b1c90e7 100644 --- a/internal/core/operations/schedulers/switchversion/executor_test.go +++ b/internal/core/operations/schedulers/switchversion/executor_test.go @@ -28,11 +28,9 @@ package switchversion_test import ( "context" "errors" - "fmt" "testing" - "time" - "github.com/topfreegames/maestro/internal/core/operations/rooms/remove" + "github.com/topfreegames/maestro/internal/core/operations/rooms/add" "github.com/topfreegames/maestro/internal/core/operations/schedulers/switchversion" "github.com/topfreegames/maestro/internal/core/ports" @@ -49,11 +47,9 @@ import ( // mockRoomAndSchedulerManager struct that holds all the mocks necessary for the // operation executor. type mockRoomAndSchedulerAndOperationManager struct { - roomManager *mockports.MockRoomManager schedulerManager *mockports.MockSchedulerManager operationManager *mockports.MockOperationManager portAllocator *mockports.MockPortAllocator - roomStorage *mockports.MockRoomStorage instanceStorage *mockports.MockGameRoomInstanceStorage runtime *mockports.MockRuntime eventsService ports.EventsService @@ -70,281 +66,64 @@ func TestExecutor_Execute(t *testing.T) { } newMajorScheduler := newValidSchedulerV2() - newMajorScheduler.PortRange.Start = 1000 - newMajorScheduler.MaxSurge = "3" - newMinorScheduler := newValidSchedulerV2() - newMinorScheduler.Spec.Version = "v1.1.0" - - activeScheduler := newValidSchedulerV2() - activeScheduler.Spec.Version = "v1.0.0" - - maxSurge := 3 - - t.Run("should succeed - Execute switch active version operation replacing pods", func(t *testing.T) { + t.Run("should succeed - Execute switch active version operation", func(t *testing.T) { definition := &switchversion.Definition{NewActiveVersion: newMajorScheduler.Spec.Version} - mocks := newMockRoomAndSchedulerManager(mockCtrl) - mocks.roomManager.EXPECT().SchedulerMaxSurge(gomock.Any(), gomock.Any()).Return(3, nil) - - var gameRoomListCycle1 []*game_room.GameRoom - var gameRoomListCycle2 []*game_room.GameRoom - var gameRoomListCycle3 []*game_room.GameRoom - for i := 0; i < maxSurge; i++ { - gameRoomListCycle1 = append(gameRoomListCycle1, &game_room.GameRoom{ - ID: fmt.Sprintf("room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - }) - } - for i := maxSurge; i < maxSurge*2; i++ { - gameRoomListCycle2 = append(gameRoomListCycle2, &game_room.GameRoom{ - ID: fmt.Sprintf("room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - }) - } mocks.schedulerManager.EXPECT().GetSchedulerByVersion(context.Background(), newMajorScheduler.Name, definition.NewActiveVersion).Return(newMajorScheduler, nil) - mocks.schedulerManager.EXPECT().GetActiveScheduler(context.Background(), newMajorScheduler.Name).Return(activeScheduler, nil) - - mocks.roomStorage.EXPECT().GetRoomCount(gomock.Any(), newMajorScheduler.Name).Return(len(append(gameRoomListCycle1, gameRoomListCycle2...)), nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle1, nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle2, nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle3, nil) - - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()).Times(4) - - for i := range append(gameRoomListCycle1, gameRoomListCycle2...) { - gameRoom := &game_room.GameRoom{ - ID: fmt.Sprintf("new-room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - } - mocks.roomManager.EXPECT().CreateRoom(gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoom, nil, nil) - } - mocks.roomManager.EXPECT().DeleteRoom(gomock.Any(), gomock.Any(), remove.SwitchVersionReplace).Return(nil).MaxTimes(len(append(gameRoomListCycle1, gameRoomListCycle2...))) - mocks.schedulerManager.EXPECT().UpdateScheduler(gomock.Any(), newMajorScheduler).Return(nil) - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) + executor := switchversion.NewExecutor(mocks.schedulerManager, mocks.operationManager) execErr := executor.Execute(context.Background(), &operation.Operation{SchedulerName: newMajorScheduler.Name}, definition) - require.Nil(t, execErr) - }) - - t.Run("should succeed - Execute switch active version operation not replacing pods", func(t *testing.T) { - noReplaceDefinition := &switchversion.Definition{NewActiveVersion: newMinorScheduler.Spec.Version} - mocks := newMockRoomAndSchedulerManager(mockCtrl) - - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(activeScheduler, nil) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMinorScheduler.Name, newMinorScheduler.Spec.Version).Return(newMinorScheduler, nil) - mocks.schedulerManager.EXPECT().UpdateScheduler(gomock.Any(), gomock.Any()).Return(nil) - - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - execErr := executor.Execute(context.Background(), &operation.Operation{SchedulerName: newMinorScheduler.Name}, noReplaceDefinition) - require.Nil(t, execErr) - }) - - t.Run("should succeed - Execute switch active version operation (no running rooms)", func(t *testing.T) { - definition := &switchversion.Definition{NewActiveVersion: newMajorScheduler.Spec.Version} - - mocks := newMockRoomAndSchedulerManager(mockCtrl) - mocks.roomManager.EXPECT().SchedulerMaxSurge(gomock.Any(), gomock.Any()).Return(maxSurge, nil) - - var emptyGameRoom []*game_room.GameRoom - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(emptyGameRoom, nil) - - mocks.roomStorage.EXPECT().GetRoomCount(gomock.Any(), newMajorScheduler.Name).Return(0, nil) - - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(activeScheduler, nil) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMajorScheduler.Name, newMajorScheduler.Spec.Version).Return(newMajorScheduler, nil) - mocks.schedulerManager.EXPECT().UpdateScheduler(gomock.Any(), gomock.Any()).Return(nil) - - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - execErr := executor.Execute(context.Background(), &operation.Operation{SchedulerName: activeScheduler.Name}, definition) - require.Nil(t, execErr) - }) - - t.Run("should succeed - Can't create room", func(t *testing.T) { - definition := &switchversion.Definition{NewActiveVersion: newMajorScheduler.Spec.Version} - mocks := newMockRoomAndSchedulerManager(mockCtrl) - - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(activeScheduler, nil) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMajorScheduler.Name, newMajorScheduler.Spec.Version).Return(newMajorScheduler, nil) - - mocks.roomManager.EXPECT().SchedulerMaxSurge(gomock.Any(), gomock.Any()).Return(3, nil) - - var gameRoomListCycle1 []*game_room.GameRoom - for i := 0; i < maxSurge; i++ { - gameRoomListCycle1 = append(gameRoomListCycle1, &game_room.GameRoom{ - ID: fmt.Sprintf("room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - }) - } - - mocks.roomStorage.EXPECT().GetRoomCount(gomock.Any(), newMajorScheduler.Name).Return(len(gameRoomListCycle1), nil) - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()).Times(3) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle1, nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return([]*game_room.GameRoom{}, nil).MaxTimes(1) - - mocks.roomManager.EXPECT().CreateRoom(gomock.Any(), gomock.Any(), gomock.Any()).Return(nil, nil, errors.New("error")).MaxTimes(maxSurge) - mocks.roomManager.EXPECT().DeleteRoom(gomock.Any(), gomock.Any(), remove.SwitchVersionReplace).Return(nil).MaxTimes(maxSurge) - - mocks.schedulerManager.EXPECT().UpdateScheduler(gomock.Any(), gomock.Any()).Return(nil) - - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - execErr := executor.Execute(context.Background(), &operation.Operation{SchedulerName: newMajorScheduler.Name}, definition) require.Nil(t, execErr) }) - t.Run("should fail - Can't update scheduler (switch active version on database)", func(t *testing.T) { - noReplaceDefinition := &switchversion.Definition{NewActiveVersion: newMinorScheduler.Spec.Version} + t.Run("should fail - Invalid definition received", func(t *testing.T) { + invalidDef := &add.Definition{Amount: 2} mocks := newMockRoomAndSchedulerManager(mockCtrl) - op := &operation.Operation{SchedulerName: newMinorScheduler.Name} - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(activeScheduler, nil) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMinorScheduler.Name, newMinorScheduler.Spec.Version).Return(newMinorScheduler, nil) - mocks.schedulerManager.EXPECT().UpdateScheduler(gomock.Any(), gomock.Any()).Return(errors.New("error")) - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), op, "error updating scheduler with new active version: error") + executor := switchversion.NewExecutor(mocks.schedulerManager, mocks.operationManager) + execErr := executor.Execute(context.Background(), &operation.Operation{SchedulerName: newMajorScheduler.Name}, invalidDef) - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - execErr := executor.Execute(context.Background(), op, noReplaceDefinition) require.NotNil(t, execErr) }) - t.Run("should fail - Can't delete room", func(t *testing.T) { + t.Run("should fail - Can not get scheduler by version", func(t *testing.T) { + getSchedErr := errors.New("foobar") definition := &switchversion.Definition{NewActiveVersion: newMajorScheduler.Spec.Version} mocks := newMockRoomAndSchedulerManager(mockCtrl) - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(activeScheduler, nil) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMajorScheduler.Name, newMajorScheduler.Spec.Version).Return(newMajorScheduler, nil) - mocks.roomManager.EXPECT().SchedulerMaxSurge(gomock.Any(), gomock.Any()).Return(maxSurge, nil) - - var gameRoomListCycle1 []*game_room.GameRoom - for i := 0; i < maxSurge; i++ { - gameRoomListCycle1 = append(gameRoomListCycle1, &game_room.GameRoom{ - ID: fmt.Sprintf("room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - }) - } - - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()).MaxTimes(3) + mocks.schedulerManager.EXPECT().GetSchedulerByVersion(context.Background(), newMajorScheduler.Name, definition.NewActiveVersion).Return(nil, getSchedErr) + mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()).Times(1) - mocks.roomStorage.EXPECT().GetRoomCount(gomock.Any(), newMajorScheduler.Name).Return(len(gameRoomListCycle1), nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle1, nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return([]*game_room.GameRoom{}, nil).MaxTimes(1) - - mocks.roomManager.EXPECT().CreateRoom(gomock.Any(), gomock.Any(), gomock.Any()).Return(nil, nil, nil).MaxTimes(maxSurge) - mocks.roomManager.EXPECT().DeleteRoom(gomock.Any(), gomock.Any(), remove.SwitchVersionReplace).Return(errors.New("error")).MaxTimes(maxSurge) - - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) + executor := switchversion.NewExecutor(mocks.schedulerManager, mocks.operationManager) execErr := executor.Execute(context.Background(), &operation.Operation{SchedulerName: newMajorScheduler.Name}, definition) - require.NotNil(t, execErr) - }) - - t.Run("should fail - Can't find max surge", func(t *testing.T) { - definition := &switchversion.Definition{NewActiveVersion: newMajorScheduler.Spec.Version} - op := &operation.Operation{SchedulerName: newMajorScheduler.Name} - mocks := newMockRoomAndSchedulerManager(mockCtrl) - - mocks.roomManager.EXPECT().SchedulerMaxSurge(gomock.Any(), gomock.Any()).Return(0, errors.New("error")) - - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(activeScheduler, nil) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMajorScheduler.Name, newMajorScheduler.Spec.Version).Return(newMajorScheduler, nil) - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), op, "error fetching scheduler max surge: error") - - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - execErr := executor.Execute(context.Background(), op, definition) require.NotNil(t, execErr) + require.ErrorIs(t, execErr, getSchedErr) }) - t.Run("should fail - Can't list rooms to delete", func(t *testing.T) { + t.Run("should fail - Can not update scheduler", func(t *testing.T) { + updateSchedErr := errors.New("foobar") definition := &switchversion.Definition{NewActiveVersion: newMajorScheduler.Spec.Version} - op := &operation.Operation{SchedulerName: newMajorScheduler.Name} mocks := newMockRoomAndSchedulerManager(mockCtrl) - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(activeScheduler, nil) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMajorScheduler.Name, newMajorScheduler.Spec.Version).Return(newMajorScheduler, nil) - - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), op, "error replacing rooms: failed to list rooms for deletion") - - mocks.roomManager.EXPECT().SchedulerMaxSurge(gomock.Any(), gomock.Any()).Return(maxSurge, nil) - - mocks.roomStorage.EXPECT().GetRoomCount(gomock.Any(), newMajorScheduler.Name).Return(0, nil) - - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil, errors.New("error")).Times(10) - - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - execErr := executor.Execute(context.Background(), op, definition) - require.NotNil(t, execErr) - }) - - t.Run("should fail - Can't count total rooms amount", func(t *testing.T) { - definition := &switchversion.Definition{NewActiveVersion: newMajorScheduler.Spec.Version} - op := &operation.Operation{SchedulerName: newMajorScheduler.Name} - mocks := newMockRoomAndSchedulerManager(mockCtrl) - - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(activeScheduler, nil) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMajorScheduler.Name, newMajorScheduler.Spec.Version).Return(newMajorScheduler, nil) - - mocks.roomManager.EXPECT().SchedulerMaxSurge(gomock.Any(), gomock.Any()).Return(maxSurge, nil) - - mocks.roomStorage.EXPECT().GetRoomCount(gomock.Any(), newMajorScheduler.Name).Return(0, errors.New("error")).Times(10) - - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), op, gomock.Any()) + mocks.schedulerManager.EXPECT().GetSchedulerByVersion(context.Background(), newMajorScheduler.Name, definition.NewActiveVersion).Return(newMajorScheduler, nil) + mocks.schedulerManager.EXPECT().UpdateScheduler(gomock.Any(), gomock.Any()).Return(updateSchedErr) + mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()).Times(1) - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) + executor := switchversion.NewExecutor(mocks.schedulerManager, mocks.operationManager) execErr := executor.Execute(context.Background(), &operation.Operation{SchedulerName: newMajorScheduler.Name}, definition) - require.NotNil(t, execErr) - }) - - t.Run("should fail - Can't get new scheduler", func(t *testing.T) { - definition := &switchversion.Definition{NewActiveVersion: newMajorScheduler.Spec.Version} - op := &operation.Operation{SchedulerName: newMajorScheduler.Name} - mocks := newMockRoomAndSchedulerManager(mockCtrl) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMajorScheduler.Name, newMajorScheduler.Spec.Version).Return(nil, errors.New("error")) - - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), op, "error fetching scheduler version to be switched to: error") - - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - execErr := executor.Execute(context.Background(), op, definition) - require.NotNil(t, execErr) - }) - - t.Run("should fail - Can't get active scheduler", func(t *testing.T) { - definition := &switchversion.Definition{NewActiveVersion: newMajorScheduler.Spec.Version} - op := &operation.Operation{SchedulerName: newMajorScheduler.Name} - mocks := newMockRoomAndSchedulerManager(mockCtrl) - - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMajorScheduler.Name, newMajorScheduler.Spec.Version).Return(newMajorScheduler, nil) - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(nil, errors.New("error")) - - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), op, "error deciding if should replace pods: error") - - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - execErr := executor.Execute(context.Background(), op, definition) require.NotNil(t, execErr) + require.ErrorIs(t, execErr, updateSchedErr) }) } func TestExecutor_Rollback(t *testing.T) { mockCtrl := gomock.NewController(t) - mocks := newMockRoomAndSchedulerManager(mockCtrl) err := validations.RegisterValidations() if err != nil { @@ -352,248 +131,45 @@ func TestExecutor_Rollback(t *testing.T) { } newMajorScheduler := newValidSchedulerV2() - newMajorScheduler.PortRange.Start = 1000 - newMajorScheduler.MaxSurge = "3" - - newMinorScheduler := newValidSchedulerV2() - newMinorScheduler.Spec.Version = "v1.1.0" - - activeScheduler := newValidSchedulerV2() - activeScheduler.Spec.Version = "v1.0.0" - - maxSurge := 3 - - definition := &switchversion.Definition{NewActiveVersion: newMajorScheduler.Spec.Version} - - t.Run("should succeed - Execute on error if operation finishes (no created rooms)", func(t *testing.T) { - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - err = executor.Rollback(context.Background(), &operation.Operation{}, definition, nil) - require.NoError(t, err) - }) - - t.Run("should succeed - Execute on error if operation finishes (created rooms)", func(t *testing.T) { - - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(activeScheduler, nil) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMajorScheduler.Name, newMajorScheduler.Spec.Version).Return(newMajorScheduler, nil) - mocks.roomManager.EXPECT().SchedulerMaxSurge(gomock.Any(), gomock.Any()).Return(3, nil) - - var gameRoomListCycle1 []*game_room.GameRoom - var gameRoomListCycle2 []*game_room.GameRoom - var gameRoomListCycle3 []*game_room.GameRoom - for i := 0; i < maxSurge; i++ { - gameRoomListCycle1 = append(gameRoomListCycle1, &game_room.GameRoom{ - ID: fmt.Sprintf("room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - }) - } - for i := maxSurge; i < maxSurge*2; i++ { - gameRoomListCycle2 = append(gameRoomListCycle2, &game_room.GameRoom{ - ID: fmt.Sprintf("room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - }) - } - - mocks.roomStorage.EXPECT().GetRoomCount(gomock.Any(), newMajorScheduler.Name).Return(len(append(gameRoomListCycle1, gameRoomListCycle2...)), nil) - - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()).Times(5) - - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle1, nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle2, nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle3, nil) - - for i := range append(gameRoomListCycle1, gameRoomListCycle2...) { - gameRoom := &game_room.GameRoom{ - ID: fmt.Sprintf("new-room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - } - mocks.roomManager.EXPECT().CreateRoom(gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoom, nil, nil) - mocks.roomManager.EXPECT().DeleteRoom(gomock.Any(), gomock.Any(), remove.SwitchVersionReplace).Return(nil) - } - - mocks.schedulerManager.EXPECT().UpdateScheduler(gomock.Any(), gomock.Any()).Return(errors.New("error")) - - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - op := &operation.Operation{ - ID: "op", - DefinitionName: definition.Name(), - SchedulerName: newMajorScheduler.Name, - CreatedAt: time.Now(), - } - execErr := executor.Execute(context.Background(), op, definition) - require.NotNil(t, execErr) - for range append(gameRoomListCycle1, gameRoomListCycle2...) { - mocks.roomManager.EXPECT().DeleteRoom(gomock.Any(), gomock.Any(), remove.SwitchVersionRollback).Return(nil) - } - - err = executor.Rollback(context.Background(), op, definition, nil) - require.NoError(t, err) - }) - - t.Run("should succeed - a create room fail, then a delete room fails and triggers rollback", func(t *testing.T) { - - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(activeScheduler, nil) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMajorScheduler.Name, newMajorScheduler.Spec.Version).Return(newMajorScheduler, nil) - mocks.roomManager.EXPECT().SchedulerMaxSurge(gomock.Any(), gomock.Any()).Return(3, nil) - - var gameRoomListCycle1 []*game_room.GameRoom - var gameRoomListCycle2 []*game_room.GameRoom - var gameRoomListCycle3 []*game_room.GameRoom - for i := 0; i < maxSurge; i++ { - gameRoomListCycle1 = append(gameRoomListCycle1, &game_room.GameRoom{ - ID: fmt.Sprintf("room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - }) - } - for i := maxSurge; i < maxSurge*2; i++ { - gameRoomListCycle2 = append(gameRoomListCycle2, &game_room.GameRoom{ - ID: fmt.Sprintf("room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - }) - } - - mocks.roomStorage.EXPECT().GetRoomCount(gomock.Any(), newMajorScheduler.Name).Return(len(append(gameRoomListCycle1, gameRoomListCycle2...)), nil) - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()).MinTimes(0) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle1, nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle2, nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle3, nil) - - allRooms := append(gameRoomListCycle1, gameRoomListCycle2...) - for i := range allRooms { - if i == len(allRooms)-1 { - mocks.roomManager.EXPECT().CreateRoom(gomock.Any(), gomock.Any(), gomock.Any()).Return(nil, nil, errors.New("error")) - mocks.roomManager.EXPECT().DeleteRoom(gomock.Any(), gomock.Any(), remove.SwitchVersionReplace).Return(errors.New("error")) - continue - } - gameRoom := &game_room.GameRoom{ - ID: fmt.Sprintf("new-room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - } - mocks.roomManager.EXPECT().CreateRoom(gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoom, nil, nil) - mocks.roomManager.EXPECT().DeleteRoom(gomock.Any(), gomock.Any(), remove.SwitchVersionReplace).Return(nil) - } - - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - op := &operation.Operation{ - ID: "op", - DefinitionName: definition.Name(), - SchedulerName: newMajorScheduler.Name, - CreatedAt: time.Now(), - } - execErr := executor.Execute(context.Background(), op, definition) - require.NotNil(t, execErr) - - for i := range allRooms { - if i == len(allRooms)-1 { - continue - } - mocks.roomManager.EXPECT().DeleteRoom(gomock.Any(), gomock.Any(), remove.SwitchVersionRollback).Return(nil) - } + t.Run("empty rollback", func(t *testing.T) { + execErr := errors.New("foobar") + definition := &switchversion.Definition{NewActiveVersion: newMajorScheduler.Spec.Version} + mocks := newMockRoomAndSchedulerManager(mockCtrl) + executor := switchversion.NewExecutor(mocks.schedulerManager, mocks.operationManager) + rollbackErr := executor.Rollback(context.Background(), &operation.Operation{SchedulerName: newMajorScheduler.Name}, definition, execErr) - err = executor.Rollback(context.Background(), op, definition, nil) - require.NoError(t, err) + require.Nil(t, rollbackErr) }) +} - t.Run("should fail - error deleting rooms", func(t *testing.T) { - mocks.schedulerManager.EXPECT().GetActiveScheduler(gomock.Any(), activeScheduler.Name).Return(activeScheduler, nil) - mocks.schedulerManager.EXPECT().GetSchedulerByVersion(gomock.Any(), newMajorScheduler.Name, newMajorScheduler.Spec.Version).Return(newMajorScheduler, nil) - mocks.roomManager.EXPECT().SchedulerMaxSurge(gomock.Any(), gomock.Any()).Return(maxSurge, nil) - - var gameRoomListCycle1 []*game_room.GameRoom - var gameRoomListCycle2 []*game_room.GameRoom - var gameRoomListCycle3 []*game_room.GameRoom - for i := 0; i < maxSurge; i++ { - gameRoomListCycle1 = append(gameRoomListCycle1, &game_room.GameRoom{ - ID: fmt.Sprintf("room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - }) - } - for i := maxSurge; i < maxSurge*2; i++ { - gameRoomListCycle2 = append(gameRoomListCycle2, &game_room.GameRoom{ - ID: fmt.Sprintf("room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - }) - } - mocks.roomStorage.EXPECT().GetRoomCount(gomock.Any(), newMajorScheduler.Name).Return(len(append(gameRoomListCycle1, gameRoomListCycle2...)), nil) - mocks.operationManager.EXPECT().AppendOperationEventToExecutionHistory(gomock.Any(), gomock.Any(), gomock.Any()).MinTimes(0) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle1, nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle2, nil) - mocks.roomManager.EXPECT().ListRoomsWithDeletionPriority(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoomListCycle3, nil) - - for i := range append(gameRoomListCycle1, gameRoomListCycle2...) { - gameRoom := &game_room.GameRoom{ - ID: fmt.Sprintf("new-room-%v", i), - SchedulerID: newMajorScheduler.Name, - Version: activeScheduler.Spec.Version, - Status: game_room.GameStatusReady, - LastPingAt: time.Now(), - } - mocks.roomManager.EXPECT().CreateRoom(gomock.Any(), gomock.Any(), gomock.Any()).Return(gameRoom, nil, nil) - mocks.roomManager.EXPECT().DeleteRoom(gomock.Any(), gomock.Any(), remove.SwitchVersionReplace).Return(nil) - } - - mocks.schedulerManager.EXPECT().UpdateScheduler(gomock.Any(), gomock.Any()).Return(errors.New("error")) - - executor := switchversion.NewExecutor(mocks.roomManager, mocks.schedulerManager, mocks.operationManager, mocks.roomStorage) - op := &operation.Operation{ - ID: "op", - DefinitionName: definition.Name(), - SchedulerName: newMajorScheduler.Name, - CreatedAt: time.Now(), - } - execErr := executor.Execute(context.Background(), op, definition) - require.NotNil(t, execErr) +func TestExecutor_Name(t *testing.T) { + mockCtrl := gomock.NewController(t) - mocks.roomManager.EXPECT().DeleteRoom(gomock.Any(), gomock.Any(), remove.SwitchVersionRollback).Return(errors.New("error")) + t.Run("returns operation name", func(t *testing.T) { + mocks := newMockRoomAndSchedulerManager(mockCtrl) + executor := switchversion.NewExecutor(mocks.schedulerManager, mocks.operationManager) + exName := executor.Name() - err = executor.Rollback(context.Background(), op, definition, nil) - require.Error(t, err) + require.Equal(t, "switch_active_version", exName) }) } func newMockRoomAndSchedulerManager(mockCtrl *gomock.Controller) *mockRoomAndSchedulerAndOperationManager { portAllocator := mockports.NewMockPortAllocator(mockCtrl) - roomStorage := mockports.NewMockRoomStorage(mockCtrl) instanceStorage := mockports.NewMockGameRoomInstanceStorage(mockCtrl) runtime := mockports.NewMockRuntime(mockCtrl) eventsForwarderService := mockports.NewMockEventsService(mockCtrl) schedulerStorage := mockports.NewMockSchedulerStorage(mockCtrl) schedulerCache := mockports.NewMockSchedulerCache(mockCtrl) - roomManager := mockports.NewMockRoomManager(mockCtrl) schedulerManager := mockports.NewMockSchedulerManager(mockCtrl) operationManager := mockports.NewMockOperationManager(mockCtrl) return &mockRoomAndSchedulerAndOperationManager{ - roomManager, schedulerManager, operationManager, portAllocator, - roomStorage, instanceStorage, runtime, eventsForwarderService,