Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: start dashmate services depending on status #503

Merged
merged 33 commits into from
Jul 12, 2023
Merged

refactor: start dashmate services depending on status #503

Changes from 8 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
3c6c3a3
feat: start and stop dashmate depending on status
pshenmic May 16, 2023
ba6a4cf
fix: zero ssl check
pshenmic May 18, 2023
d08a3b7
fix: ansible lint
pshenmic May 18, 2023
42227a9
Merge branch 'master' into fix/dashmate-status
pshenmic May 18, 2023
1eb6539
fix: code review
pshenmic May 18, 2023
12a8d98
fix: revert some changes from separate pr
pshenmic May 18, 2023
51c5968
feat: add start platform
pshenmic May 18, 2023
788e861
fix: replace drive with tenderdash
pshenmic May 18, 2023
1adec64
Merge branch 'master' into fix/dashmate-status
pshenmic Jun 22, 2023
b656904
feat: smart start / restart logic
pshenmic Jun 25, 2023
7fff8ce
fix: template idempotency
pshenmic Jun 26, 2023
e9cb556
fix: remove comments
pshenmic Jun 26, 2023
414a3e9
chore: remove force
pshenmic Jun 28, 2023
a4bbad3
Merge branch 'master' into fix/dashmate-status
pshenmic Jul 3, 2023
02b08e5
chore: print
pshenmic Jul 3, 2023
0c92a06
chore: revert
pshenmic Jul 3, 2023
1b8484c
chore: fix print
pshenmic Jul 3, 2023
b5dbe2a
chore: revert
pshenmic Jul 3, 2023
7848224
fix: pretty print and fix var name
pshenmic Jul 5, 2023
0ce0f7d
fix: lint issues
pshenmic Jul 5, 2023
4a06b95
fix: cleanup
pshenmic Jul 5, 2023
f02b30d
fix: revert check
pshenmic Jul 5, 2023
cec1346
fix: escape
pshenmic Jul 5, 2023
b51fb42
fix: code review
pshenmic Jul 7, 2023
1bd0879
feat: extract dashmate version
pshenmic Jul 7, 2023
371c2ba
fix: lint fix
pshenmic Jul 7, 2023
21c6582
feat: improve conditions
pshenmic Jul 10, 2023
bbd8e38
feat: cleanup
pshenmic Jul 10, 2023
5ad5dfd
feat: look by not_started
pshenmic Jul 10, 2023
cdd4049
feat: refactor to core_running
pshenmic Jul 10, 2023
fa503a0
refactor: start/restart logic
shumkov Jul 10, 2023
ec8f134
fix: add bindIp
pshenmic Jul 10, 2023
883475f
fix: issues
pshenmic Jul 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 49 additions & 4 deletions ansible/roles/dashmate/tasks/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,8 +173,53 @@
changed_when: true
when: dashmate_branch is defined

# TODO: we should use status command to figure out if dashmate is running
# to get the status from the response
- name: Get status
ansible.builtin.command: "{{ dashmate_cmd }} status --format=json"
become: true
become_user: dashmate
changed_when: true
shumkov marked this conversation as resolved.
Show resolved Hide resolved
args:
chdir: '{{ dashmate_cwd }}'
register: dashmate_status

- name: Get service status
shumkov marked this conversation as resolved.
Show resolved Hide resolved
ansible.builtin.set_fact:
core_service_status: "{{ dashmate_status.stdout | from_json | json_query('core.serviceStatus') }}"
tenderdash_service_status: "{{ dashmate_status.stdout | from_json | json_query('platform.tenderdash.serviceStatus') }}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems we need to get all platform service statuses otherwise it will be inconsistency and conflict with the check in dashmate https://github.com/dashpay/platform/blob/2e984df0302fa94bdf187bd893328ec58f390aa0/packages/dashmate/src/docker/DockerCompose.js#L75

We using this function in start and restart commands I believe


- name: Fail if dashmate node in inconsistent state
ansible.builtin.fail:
msg: Dashmate status returned invalid service status
when: >
(core_service_status != 'up' and core_service_status != 'stopped') or
(tenderdash_service_status != 'up' and tenderdash_service_status != 'stopped')

# start platform when (core is up and platform is down)
Copy link
Member

@shumkov shumkov May 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we need to restart Core as well? What if dashmate code is updated or config is changed?

We can do restart logic smarter:

  1. Images are changes: restart core if core image changed and restart platform if platform images are changed
  2. dashmate config is changed
  3. dashmate is changed

@pshenmic @strophy What do you think guys?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, yesterday I had difficulty updating from Platform 0.24.2 to 0.24.4 because docker compose restart does not pull and use newer images, docker compose up -d is necessary for this. We definitely need a way of ensuring the necessary containers are restarted (or stopped, updated and started) when config changes. But changing the dashmate version probably shouldn't cause core to restart - this usually changes at the same time as the platform version only.

Copy link
Member

@shumkov shumkov May 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docker compose restart does not pull and use newer images

it's not, that's why we have dashmate update above

But changing the dashmate version probably shouldn't cause core to restart

We should because docker compose config could be changed.

Copy link
Member

@shumkov shumkov May 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about this logic:

  • check services status
    • core is running
    • drive is running
  • start ALL when (core is down and platform is down)
  • start platform when (core is up and platform is down)
  • determine what's need to be restarted:
    • any of platform images are updated: platform
    • core / sentinel images are updated: all
    • dashmate is changed: all
    • dashmate config is changed: all
  • restart platform (platform is not down and platform needs to be restarted)
  • restart all (all needs to be restarted)

# start ALL when (core is down and platform is down)
# restart ALL when (core is up and platform is up)

# Start platform in case core is running
- name: Start dashmate platform services
ansible.builtin.command: "{{ dashmate_cmd }} start --platform --verbose"
become: true
become_user: dashmate
args:
chdir: '{{ dashmate_cwd }}'
register: dashmate_start_platform
when: core_service_status == 'up' and tenderdash_service_status == 'stopped'
changed_when: dashmate_start_platform.rc == 0

# Start if not already running
- name: Start dashmate services
ansible.builtin.command: "{{ dashmate_cmd }} start --verbose"
become: true
become_user: dashmate
args:
chdir: '{{ dashmate_cwd }}'
register: dashmate_start
when: core_service_status == 'stopped' and tenderdash_service_status == 'stopped'
changed_when: dashmate_start.rc == 0

# Restart dashmate if already running
- name: Restart dashmate services
ansible.builtin.command: "{{ dashmate_cmd }} restart --verbose"
Expand All @@ -183,8 +228,8 @@
args:
chdir: '{{ dashmate_cwd }}'
register: dashmate_restart
shumkov marked this conversation as resolved.
Show resolved Hide resolved
when: core_service_status == 'up' and tenderdash_service_status == 'up'
changed_when: dashmate_restart.rc == 0
ignore_errors: true

# Start if not already running
- name: Start dashmate services
Expand All @@ -194,7 +239,7 @@
args:
chdir: '{{ dashmate_cwd }}'
register: dashmate_start
when: "dashmate_restart.rc != 0"
when: "service_status == 'stopped'"
changed_when: dashmate_start.rc == 0

# TODO Convenient tasks until we have this functionality in dashmate
Expand Down