Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: start dashmate services depending on status #503

Merged
merged 33 commits into from
Jul 12, 2023
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
3c6c3a3
feat: start and stop dashmate depending on status
pshenmic May 16, 2023
ba6a4cf
fix: zero ssl check
pshenmic May 18, 2023
d08a3b7
fix: ansible lint
pshenmic May 18, 2023
42227a9
Merge branch 'master' into fix/dashmate-status
pshenmic May 18, 2023
1eb6539
fix: code review
pshenmic May 18, 2023
12a8d98
fix: revert some changes from separate pr
pshenmic May 18, 2023
51c5968
feat: add start platform
pshenmic May 18, 2023
788e861
fix: replace drive with tenderdash
pshenmic May 18, 2023
1adec64
Merge branch 'master' into fix/dashmate-status
pshenmic Jun 22, 2023
b656904
feat: smart start / restart logic
pshenmic Jun 25, 2023
7fff8ce
fix: template idempotency
pshenmic Jun 26, 2023
e9cb556
fix: remove comments
pshenmic Jun 26, 2023
414a3e9
chore: remove force
pshenmic Jun 28, 2023
a4bbad3
Merge branch 'master' into fix/dashmate-status
pshenmic Jul 3, 2023
02b08e5
chore: print
pshenmic Jul 3, 2023
0c92a06
chore: revert
pshenmic Jul 3, 2023
1b8484c
chore: fix print
pshenmic Jul 3, 2023
b5dbe2a
chore: revert
pshenmic Jul 3, 2023
7848224
fix: pretty print and fix var name
pshenmic Jul 5, 2023
0ce0f7d
fix: lint issues
pshenmic Jul 5, 2023
4a06b95
fix: cleanup
pshenmic Jul 5, 2023
f02b30d
fix: revert check
pshenmic Jul 5, 2023
cec1346
fix: escape
pshenmic Jul 5, 2023
b51fb42
fix: code review
pshenmic Jul 7, 2023
1bd0879
feat: extract dashmate version
pshenmic Jul 7, 2023
371c2ba
fix: lint fix
pshenmic Jul 7, 2023
21c6582
feat: improve conditions
pshenmic Jul 10, 2023
bbd8e38
feat: cleanup
pshenmic Jul 10, 2023
5ad5dfd
feat: look by not_started
pshenmic Jul 10, 2023
cdd4049
feat: refactor to core_running
pshenmic Jul 10, 2023
fa503a0
refactor: start/restart logic
shumkov Jul 10, 2023
ec8f134
fix: add bindIp
pshenmic Jul 10, 2023
883475f
fix: issues
pshenmic Jul 10, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ansible/roles/dashmate/tasks/install.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,4 @@
ansible.builtin.apt:
deb: "{{ release_info.json.assets | selectattr('name', 'search', '_' + deb_architecture.stdout + '.deb') | map(attribute='browser_download_url') | first }}"
allow_downgrade: true
register: dashmate_install_result
shumkov marked this conversation as resolved.
Show resolved Hide resolved
106 changes: 95 additions & 11 deletions ansible/roles/dashmate/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@
owner: '{{ dashmate_user }}'
group: '{{ dashmate_group }}'
mode: "0644"
register: template_result

- name: Generate self-signed SSL certificate for DAPI
ansible.builtin.import_tasks: ./ssl/self_signed.yml
Expand Down Expand Up @@ -131,7 +132,7 @@
when: dashmate_branch is defined

- name: Update dashmate images
ansible.builtin.command: "{{ dashmate_cmd }} update --verbose"
ansible.builtin.command: "{{ dashmate_cmd }} update --format=json"
become: true
become_user: dashmate
register: dashmate_update
Expand All @@ -148,30 +149,113 @@
changed_when: true
when: dashmate_branch is defined

# TODO: we should use status command to figure out if dashmate is running
# to get the status from the response
# Restart dashmate if already running
- name: Restart dashmate services
ansible.builtin.command: "{{ dashmate_cmd }} restart --verbose"
- name: Get status
ansible.builtin.command: "{{ dashmate_cmd }} status --format=json"
become: true
become_user: dashmate
changed_when: true
shumkov marked this conversation as resolved.
Show resolved Hide resolved
args:
chdir: '{{ dashmate_cwd }}'
register: dashmate_restart
changed_when: dashmate_restart.rc == 0
ignore_errors: true
register: dashmate_status

- name: Get service status
shumkov marked this conversation as resolved.
Show resolved Hide resolved
ansible.builtin.set_fact:
core_docker_status: "{{ dashmate_status.stdout | from_json | json_query('core.dockerStatus') }}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if tenderdash is stopped it Drive or DAPI is started?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean? Should I rather check by drive container?

Copy link
Member

@shumkov shumkov Jul 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check if one of them started?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm worried that it will add more fragility to the code. We actually don't have them stopped separately from each other. Tenderdash container will be either in not_started state or running. If there are any issues happening, it will remain in restarting, exited, or any other states, and tasks will be skipped

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What will happen if tenderdash is stopped but any of platform services is running? DAPI for example?

tenderdash_docker_status: "{{ dashmate_status.stdout | from_json | json_query('platform.tenderdash.dockerStatus') }}"
core_images_updated: >-
{{
dashmate_update.stdout |
from_json |
json_query('[?name==`core`||name==`sentinel`]') |
sort(attribute='updated', reverse=true) |
first() |
json_query('updated')
}}
platform_images_updated: >-
{{
dashmate_update.stdout |
from_json |
json_query('[?name!=`core`&&name!=`sentinel`]') |
sort(attribute='updated', reverse=true) |
first() |
json_query('updated')
}}
dashmate_package_changed: "{{ dashmate_install_result.changed }}"
dashmate_config_changed: "{{ template_result.changed }}"

- name: Print status
ansible.builtin.debug:
msg: "\n
Core Status: {{ core_docker_status }}\n
Tenderdash Status: {{ tenderdash_docker_status }}\n
Core Images Updated: {{ core_images_updated }}\n
Platform Images Updated: {{ platform_images_updated }}\n
Dashmate Package Updated: {{ dashmate_package_changed }}\n
Dashmate Config Changed: {{ dashmate_config_changed }}
"

# Start logic:
# start ALL when (core is down and platform is down)
# start platform when (core is up and platform is down)

# Start if not already running
# Start all if everything is stopped (not started yet)
- name: Start dashmate services
ansible.builtin.command: "{{ dashmate_cmd }} start --verbose"
become: true
become_user: dashmate
args:
chdir: '{{ dashmate_cwd }}'
register: dashmate_start
when: "dashmate_restart.rc != 0"
when:
- core_docker_status == 'not_started'
- tenderdash_docker_status == 'not_started'
changed_when: dashmate_start.rc == 0

# Start platform in case core is running
- name: Start dashmate platform services
ansible.builtin.command: "{{ dashmate_cmd }} start --platform --verbose"
become: true
become_user: dashmate
args:
chdir: '{{ dashmate_cwd }}'
register: dashmate_start_platform
when:
- core_docker_status == 'running'
- tenderdash_docker_status == 'not_started'
changed_when: dashmate_start_platform.rc == 0

# Restart logic
strophy marked this conversation as resolved.
Show resolved Hide resolved
# if any of platform images were updated => restart platform
# if core or sentinel images updated => restart all
# dashmate version is changed => restart all
# dashmate config is changed => restart all

- name: Restart dashmate platform services
ansible.builtin.command: "{{ dashmate_cmd }} restart --platform --verbose"
become: true
become_user: dashmate
args:
chdir: '{{ dashmate_cwd }}'
register: dashmate_restart
when:
- core_docker_status == 'running' and tenderdash_docker_status == 'running'
shumkov marked this conversation as resolved.
Show resolved Hide resolved
- core_images_updated is false and platform_images_updated is true
- dashmate_package_changed is false and dashmate_config_changed is false
changed_when: dashmate_restart.rc == 0

- name: Restart dashmate services
ansible.builtin.command: "{{ dashmate_cmd }} restart --verbose"
become: true
become_user: dashmate
args:
chdir: '{{ dashmate_cwd }}'
register: dashmate_restart
shumkov marked this conversation as resolved.
Show resolved Hide resolved
when:
- core_docker_status == 'running' and tenderdash_docker_status == 'running'
- platform_images_updated is false
- core_images_updated is true or dashmate_package_changed is true or dashmate_config_changed is true
changed_when: dashmate_restart.rc == 0

# TODO Convenient tasks until we have this functionality in dashmate
# - name: Replace parameters in tenderdash config
# ansible.builtin.lineinfile:
Expand Down
23 changes: 10 additions & 13 deletions ansible/roles/dashmate/templates/dashmate.json.j2
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"configFormatVersion": "{{ dashmate_version | default('999.0.0') }}",
"configFormatVersion": "{{ dashmate_version | default(dashmate_config_version | default('999.0.0')) }}",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you we want to configure this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because when we install from branch, we don't have dashmate_version available and it will back off to 999.0.0 which will break template idempotency. On the first dashmate run, config version will be overwritten with the package default version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Yes you right. Would it be better to get dashmate version form dashmate itself (dashmate --version or package.json) so you won't need to configure it manually. 999.0.0 workaround won't be necessary as well.

"defaultConfigName": "{{ dash_network_name}}",
"defaultGroupName": null,
"configs": {
Expand All @@ -18,12 +18,10 @@
"p2p": {
"port": {{ dashd_port }},
"seeds": [
{% for seed in template_bootstrap_peers %}
{
{% for seed in template_bootstrap_peers %}{
"host": "{{ hostvars[seed].public_ip }}",
"port": {{ dashd_port }}
}{% if not loop.last %},{% endif %}
{% endfor %}
}{% if not loop.last %},{% endif %}{% endfor %}
]
},
"rpc": {
Expand Down Expand Up @@ -60,9 +58,9 @@
}
},
"devnet": {
"name": {% if dash_network == 'devnet' %}"{{ dash_devnet_name }}"{% else %}null{% endif %},
"minimumDifficultyBlocks": {{ dashd_minimumdifficultyblocks }},
"powTargetSpacing": {{ dashd_powtargetspacing }}
"name": {% if dash_network == 'devnet' %}"{{ dash_devnet_name }}"{% else %}null{% endif %},
"minimumDifficultyBlocks": {{ dashd_minimumdifficultyblocks }},
"powTargetSpacing": {{ dashd_powtargetspacing }}
},
"logIps": 0,
"indexes": true,
Expand Down Expand Up @@ -145,9 +143,8 @@
"id": "{{ seed_nodes[seed].node_key.id }}",
"host": "{{ hostvars[seed].public_ip }}",
"port": {{ tendermint_p2p_port }}
}{% if not loop.last %},{% endif %}
{% endfor %}
]
}{% if not loop.last %},{% endif %}{% endfor %}
]
},
"rpc": {
"port": {{ tendermint_rpc_port }}
Expand Down Expand Up @@ -238,7 +235,7 @@
},
"withdrawals": {
"contract": {
"id": "4fJLR2GYTPFdomuTVvNy3VRrvWgvkKPzqehEBpNf2nk6"
"id": "4fJLR2GYTPFdomuTVvNy3VRrvWgvkKPzqehEBpNf2nk6"
},
"masterPublicKey": "{{ withdrawals_hd_master_public_key }}",
"secondPublicKey": "{{ withdrawals_hd_second_public_key }}"
Expand All @@ -251,7 +248,7 @@
},
"api": {
"enable": true,
"port": "{{ dashmate_api_port }}"
"port": {{ dashmate_api_port }}
}
}
},
Expand Down