Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue with Backuping MLNXOS Devices Using Oxidized #3357

Open
graby0 opened this issue Jan 2, 2025 · 3 comments
Open

Issue with Backuping MLNXOS Devices Using Oxidized #3357

graby0 opened this issue Jan 2, 2025 · 3 comments

Comments

@graby0
Copy link

graby0 commented Jan 2, 2025 •
edited by robertcheramy
Loading

Hello,

In my infrastructure, I manage a large number of network devices that I back up using Oxidized. However, I am encountering an issue with backing up devices running the MLNXOS system.

From the Linux shell, I am able to connect to the switch via SSH and execute all the necessary commands without any problems.

However, the following error messages appear in the Oxidized logs:

W, [2024-11-14T10:29:09.589014 #144957]  WARN -- : IP raised Net::SSH::AuthenticationFailed with msg "Authentication failed for user backup@IP"
W, [2024-11-14T10:29:10.372208 #144957]  WARN -- : CMP/CMP-PUL-SWC02-M2 status no_connection, retry attempt 1
W, [2024-11-14T10:29:10.591862 #144957]  WARN -- : IP raised Net::SSH::AuthenticationFailed with msg "Authentication failed for user backup@IP"
W, [2024-11-14T10:29:11.374222 #144957]  WARN -- : CMP/CMP-PUL-SWC02-M2 status no_connection, retries exhausted, giving up


Log in debug level:
D, [2024-12-16T15:09:28.402377 #19416] DEBUG -- : lib/oxidized/worker.rb: Added Switch_name to the job queue
D, [2024-12-16T15:09:28.402410 #19416] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-12-16T15:09:28.402549 #19416] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for Switch_name at 2024-12-16 14:09:28 UTC
D, [2024-12-16T15:09:28.402708 #19416] DEBUG -- : lib/oxidized/input/ssh.rb: Connecting to Switch_name
D, [2024-12-16T15:09:28.402968 #19416] DEBUG -- : AUTH METHODS::["none", "publickey", "password"]
D, [2024-12-16T15:09:28.403529 #19416] DEBUG -- net.ssh.transport.session[744]: establishing connection to IP:22
D, [2024-12-16T15:09:28.405269 #19416] DEBUG -- net.ssh.transport.session[744]: connection established
I, [2024-12-16T15:09:28.405317 #19416]  INFO -- net.ssh.transport.server_version[758]: negotiating protocol version
D, [2024-12-16T15:09:28.405328 #19416] DEBUG -- net.ssh.transport.server_version[758]: local is `SSH-2.0-Ruby/Net::SSH_7.2.1 x86_64-linux'
D, [2024-12-16T15:09:28.509251 #19416] DEBUG -- net.ssh.transport.server_version[758]: remote is `SSH-2.0-OpenSSH_8.0'
I, [2024-12-16T15:09:28.509667 #19416]  INFO -- net.ssh.transport.algorithms[76c]: sending KEXINIT
D, [2024-12-16T15:09:28.509781 #19416] DEBUG -- socket[780]: queueing packet nr 0 type 20 len 1436
D, [2024-12-16T15:09:28.509846 #19416] DEBUG -- socket[780]: sent 1440 bytes
D, [2024-12-16T15:09:28.510347 #19416] DEBUG -- socket[780]: read 616 bytes
D, [2024-12-16T15:09:28.510404 #19416] DEBUG -- socket[780]: received packet nr 0 type 20 len 612
I, [2024-12-16T15:09:28.510426 #19416]  INFO -- net.ssh.transport.algorithms[76c]: got KEXINIT from server
I, [2024-12-16T15:09:28.510463 #19416]  INFO -- net.ssh.transport.algorithms[76c]: negotiating algorithms
D, [2024-12-16T15:09:28.510517 #19416] DEBUG -- net.ssh.transport.algorithms[76c]: negotiated:
* kex: diffie-hellman-group14-sha256
* host_key: ssh-rsa
* encryption_server: aes256-ctr
* encryption_client: aes256-ctr
* hmac_client: [email protected]
* hmac_server: [email protected]
* compression_client: none
* compression_server: none
* language_client:
* language_server:
D, [2024-12-16T15:09:28.510528 #19416] DEBUG -- net.ssh.transport.algorithms[76c]: exchanging keys
D, [2024-12-16T15:09:28.512934 #19416] DEBUG -- socket[780]: queueing packet nr 1 type 30 len 268
D, [2024-12-16T15:09:28.512997 #19416] DEBUG -- socket[780]: sent 272 bytes
D, [2024-12-16T15:09:28.548316 #19416] DEBUG -- socket[780]: read 1104 bytes
D, [2024-12-16T15:09:28.548450 #19416] DEBUG -- socket[780]: received packet nr 1 type 31 len 1084
D, [2024-12-16T15:09:28.551124 #19416] DEBUG -- socket[780]: queueing packet nr 2 type 21 len 20
D, [2024-12-16T15:09:28.551214 #19416] DEBUG -- socket[780]: sent 24 bytes
D, [2024-12-16T15:09:28.551254 #19416] DEBUG -- socket[780]: received packet nr 2 type 21 len 12
D, [2024-12-16T15:09:28.551479 #19416] DEBUG -- net.ssh.authentication.session[794]: beginning authentication of `backup'
D, [2024-12-16T15:09:28.551529 #19416] DEBUG -- socket[780]: using encrypt-then-mac
D, [2024-12-16T15:09:28.551568 #19416] DEBUG -- socket[780]: queueing packet nr 3 type 5 len 32
D, [2024-12-16T15:09:28.551584 #19416] DEBUG -- socket[780]: sent 100 bytes
D, [2024-12-16T15:09:28.594341 #19416] DEBUG -- socket[780]: read 100 bytes
D, [2024-12-16T15:09:28.594520 #19416] DEBUG -- socket[780]: received packet nr 3 type 6 len 32
D, [2024-12-16T15:09:28.594604 #19416] DEBUG -- net.ssh.authentication.session[794]: trying none
D, [2024-12-16T15:09:28.594685 #19416] DEBUG -- socket[780]: using encrypt-then-mac
D, [2024-12-16T15:09:28.594728 #19416] DEBUG -- socket[780]: queueing packet nr 4 type 50 len 48
D, [2024-12-16T15:09:28.594786 #19416] DEBUG -- socket[780]: sent 116 bytes
D, [2024-12-16T15:09:28.604548 #19416] DEBUG -- socket[780]: read 232 bytes
D, [2024-12-16T15:09:28.604725 #19416] DEBUG -- socket[780]: received packet nr 4 type 53 len 48
I, [2024-12-16T15:09:28.604770 #19416]  INFO -- net.ssh.authentication.session[794]: HPE Management Console

D, [2024-12-16T15:09:28.604817 #19416] DEBUG -- socket[780]: received packet nr 5 type 51 len 48
D, [2024-12-16T15:09:28.604850 #19416] DEBUG -- net.ssh.authentication.session[794]: allowed methods: publickey,keyboard-interactive
D, [2024-12-16T15:09:28.604881 #19416] DEBUG -- net.ssh.authentication.methods.none[7a8]: none failed
D, [2024-12-16T15:09:28.604904 #19416] DEBUG -- net.ssh.authentication.session[794]: trying publickey
D, [2024-12-16T15:09:28.605117 #19416] DEBUG -- net.ssh.authentication.agent[7bc]: connecting to ssh-agent
E, [2024-12-16T15:09:28.605153 #19416] ERROR -- net.ssh.authentication.agent[7bc]: could not connect to ssh-agent: Agent not configured
D, [2024-12-16T15:09:28.605232 #19416] DEBUG -- net.ssh.authentication.methods.publickey[7e4]: trying publickey (ca:02:c0:81:b1:e7:1f:43:ea:d3:65:82:92:25:c9:37) alg rsa-sha2-256
D, [2024-12-16T15:09:28.605297 #19416] DEBUG -- socket[780]: using encrypt-then-mac
D, [2024-12-16T15:09:28.605335 #19416] DEBUG -- socket[780]: queueing packet nr 5 type 50 len 368
D, [2024-12-16T15:09:28.605383 #19416] DEBUG -- socket[780]: sent 436 bytes
D, [2024-12-16T15:09:28.615347 #19416] DEBUG -- socket[780]: read 116 bytes
D, [2024-12-16T15:09:28.615494 #19416] DEBUG -- socket[780]: received packet nr 6 type 51 len 48
D, [2024-12-16T15:09:28.615550 #19416] DEBUG -- net.ssh.authentication.session[794]: allowed methods: publickey,keyboard-interactive
D, [2024-12-16T15:09:28.615576 #19416] DEBUG -- net.ssh.authentication.methods.publickey[7e4]: trying publickey (ca:02:c0:81:b1:e7:1f:43:ea:d3:65:82:92:25:c9:37) alg ssh-rsa
D, [2024-12-16T15:09:28.615656 #19416] DEBUG -- socket[780]: using encrypt-then-mac
D, [2024-12-16T15:09:28.615861 #19416] DEBUG -- socket[780]: queueing packet nr 6 type 50 len 352
D, [2024-12-16T15:09:28.615938 #19416] DEBUG -- socket[780]: sent 420 bytes
D, [2024-12-16T15:09:28.625840 #19416] DEBUG -- socket[780]: read 116 bytes
D, [2024-12-16T15:09:28.626012 #19416] DEBUG -- socket[780]: received packet nr 7 type 51 len 48
D, [2024-12-16T15:09:28.626074 #19416] DEBUG -- net.ssh.authentication.session[794]: allowed methods: publickey,keyboard-interactive
E, [2024-12-16T15:09:28.626106 #19416] ERROR -- net.ssh.authentication.session[794]: all authorization methods failed (tried none, publickey)
W, [2024-12-16T15:09:28.626194 #19416]  WARN -- : IP raised Net::SSH::AuthenticationFailed with msg "Authentication failed for user backup@IP"
D, [2024-12-16T15:09:28.626221 #19416] DEBUG -- : lib/oxidized/node.rb: Oxidized::SSH failed for Switch_name
D, [2024-12-16T15:09:28.626243 #19416] DEBUG -- : lib/oxidized/job.rb: Config fetched for Switch_name at 2024-12-16 14:09:28 UTC
W, [2024-12-16T15:09:29.403547 #19416]  WARN -- : Switch_name status no_connection, retry attempt 1
D, [2024-12-16T15:09:29.403611 #19416] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 2 of 441
D, [2024-12-16T15:09:29.403647 #19416] DEBUG -- : lib/oxidized/worker.rb: Added Switch_name to the job queue
D, [2024-12-16T15:09:29.403663 #19416] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-12-16T15:09:29.403725 #19416] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for Switch_name at 2024-12-16 14:09:29 UTC
D, [2024-12-16T15:09:29.403871 #19416] DEBUG -- : lib/oxidized/input/ssh.rb: Connecting to Switch_name
D, [2024-12-16T15:09:29.403955 #19416] DEBUG -- : AUTH METHODS::["none", "publickey", "password"]
D, [2024-12-16T15:09:29.404500 #19416] DEBUG -- net.ssh.transport.session[7f8]: establishing connection to IP:22
D, [2024-12-16T15:09:29.406144 #19416] DEBUG -- net.ssh.transport.session[7f8]: connection established
I, [2024-12-16T15:09:29.406201 #19416]  INFO -- net.ssh.transport.server_version[80c]: negotiating protocol version
D, [2024-12-16T15:09:29.406213 #19416] DEBUG -- net.ssh.transport.server_version[80c]: local is `SSH-2.0-Ruby/Net::SSH_7.2.1 x86_64-linux'
D, [2024-12-16T15:09:29.510154 #19416] DEBUG -- net.ssh.transport.server_version[80c]: remote is `SSH-2.0-OpenSSH_8.0'
I, [2024-12-16T15:09:29.510569 #19416]  INFO -- net.ssh.transport.algorithms[820]: sending KEXINIT
D, [2024-12-16T15:09:29.510692 #19416] DEBUG -- socket[834]: queueing packet nr 0 type 20 len 1436
D, [2024-12-16T15:09:29.510744 #19416] DEBUG -- socket[834]: sent 1440 bytes
D, [2024-12-16T15:09:29.511230 #19416] DEBUG -- socket[834]: read 616 bytes
D, [2024-12-16T15:09:29.511288 #19416] DEBUG -- socket[834]: received packet nr 0 type 20 len 612
I, [2024-12-16T15:09:29.511322 #19416]  INFO -- net.ssh.transport.algorithms[820]: got KEXINIT from server
I, [2024-12-16T15:09:29.511369 #19416]  INFO -- net.ssh.transport.algorithms[820]: negotiating algorithms
D, [2024-12-16T15:09:29.511433 #19416] DEBUG -- net.ssh.transport.algorithms[820]: negotiated:
* kex: diffie-hellman-group14-sha256
* host_key: ssh-rsa
* encryption_server: aes256-ctr
* encryption_client: aes256-ctr
* hmac_client: [email protected]
* hmac_server: [email protected]
* compression_client: none
* compression_server: none
* language_client:
* language_server:
D, [2024-12-16T15:09:29.511450 #19416] DEBUG -- net.ssh.transport.algorithms[820]: exchanging keys
D, [2024-12-16T15:09:29.513696 #19416] DEBUG -- socket[834]: queueing packet nr 1 type 30 len 268
D, [2024-12-16T15:09:29.513744 #19416] DEBUG -- socket[834]: sent 272 bytes
D, [2024-12-16T15:09:29.549069 #19416] DEBUG -- socket[834]: read 1104 bytes
D, [2024-12-16T15:09:29.549252 #19416] DEBUG -- socket[834]: received packet nr 1 type 31 len 1084
D, [2024-12-16T15:09:29.551659 #19416] DEBUG -- socket[834]: queueing packet nr 2 type 21 len 20
D, [2024-12-16T15:09:29.552014 #19416] DEBUG -- socket[834]: sent 24 bytes
D, [2024-12-16T15:09:29.552063 #19416] DEBUG -- socket[834]: received packet nr 2 type 21 len 12
D, [2024-12-16T15:09:29.552298 #19416] DEBUG -- net.ssh.authentication.session[848]: beginning authentication of `backup'
D, [2024-12-16T15:09:29.552349 #19416] DEBUG -- socket[834]: using encrypt-then-mac
D, [2024-12-16T15:09:29.552392 #19416] DEBUG -- socket[834]: queueing packet nr 3 type 5 len 32
D, [2024-12-16T15:09:29.552409 #19416] DEBUG -- socket[834]: sent 100 bytes
D, [2024-12-16T15:09:29.595225 #19416] DEBUG -- socket[834]: read 100 bytes
D, [2024-12-16T15:09:29.595467 #19416] DEBUG -- socket[834]: received packet nr 3 type 6 len 32
D, [2024-12-16T15:09:29.595574 #19416] DEBUG -- net.ssh.authentication.session[848]: trying none
D, [2024-12-16T15:09:29.595671 #19416] DEBUG -- socket[834]: using encrypt-then-mac
D, [2024-12-16T15:09:29.595706 #19416] DEBUG -- socket[834]: queueing packet nr 4 type 50 len 48
D, [2024-12-16T15:09:29.595761 #19416] DEBUG -- socket[834]: sent 116 bytes
D, [2024-12-16T15:09:29.605507 #19416] DEBUG -- socket[834]: read 232 bytes
D, [2024-12-16T15:09:29.605617 #19416] DEBUG -- socket[834]: received packet nr 4 type 53 len 48
I, [2024-12-16T15:09:29.605666 #19416]  INFO -- net.ssh.authentication.session[848]: HPE Management Console

D, [2024-12-16T15:09:29.605715 #19416] DEBUG -- socket[834]: received packet nr 5 type 51 len 48
D, [2024-12-16T15:09:29.605743 #19416] DEBUG -- net.ssh.authentication.session[848]: allowed methods: publickey,keyboard-interactive
D, [2024-12-16T15:09:29.605781 #19416] DEBUG -- net.ssh.authentication.methods.none[85c]: none failed
D, [2024-12-16T15:09:29.605810 #19416] DEBUG -- net.ssh.authentication.session[848]: trying publickey
D, [2024-12-16T15:09:29.606074 #19416] DEBUG -- net.ssh.authentication.agent[870]: connecting to ssh-agent
E, [2024-12-16T15:09:29.606122 #19416] ERROR -- net.ssh.authentication.agent[870]: could not connect to ssh-agent: Agent not configured
D, [2024-12-16T15:09:29.606221 #19416] DEBUG -- net.ssh.authentication.methods.publickey[898]: trying publickey (ca:02:c0:81:b1:e7:1f:43:ea:d3:65:82:92:25:c9:37) alg rsa-sha2-256
D, [2024-12-16T15:09:29.606288 #19416] DEBUG -- socket[834]: using encrypt-then-mac
D, [2024-12-16T15:09:29.606320 #19416] DEBUG -- socket[834]: queueing packet nr 5 type 50 len 368
D, [2024-12-16T15:09:29.606353 #19416] DEBUG -- socket[834]: sent 436 bytes
D, [2024-12-16T15:09:29.616123 #19416] DEBUG -- socket[834]: read 116 bytes
D, [2024-12-16T15:09:29.616216 #19416] DEBUG -- socket[834]: received packet nr 6 type 51 len 48
D, [2024-12-16T15:09:29.616254 #19416] DEBUG -- net.ssh.authentication.session[848]: allowed methods: publickey,keyboard-interactive
D, [2024-12-16T15:09:29.616279 #19416] DEBUG -- net.ssh.authentication.methods.publickey[898]: trying publickey (ca:02:c0:81:b1:e7:1f:43:ea:d3:65:82:92:25:c9:37) alg ssh-rsa
D, [2024-12-16T15:09:29.616337 #19416] DEBUG -- socket[834]: using encrypt-then-mac
D, [2024-12-16T15:09:29.616372 #19416] DEBUG -- socket[834]: queueing packet nr 6 type 50 len 352
D, [2024-12-16T15:09:29.616406 #19416] DEBUG -- socket[834]: sent 420 bytes
D, [2024-12-16T15:09:29.626510 #19416] DEBUG -- socket[834]: read 116 bytes
D, [2024-12-16T15:09:29.626841 #19416] DEBUG -- socket[834]: received packet nr 7 type 51 len 48
D, [2024-12-16T15:09:29.627015 #19416] DEBUG -- net.ssh.authentication.session[848]: allowed methods: publickey,keyboard-interactive
E, [2024-12-16T15:09:29.627131 #19416] ERROR -- net.ssh.authentication.session[848]: all authorization methods failed (tried none, publickey)
W, [2024-12-16T15:09:29.627261 #19416]  WARN -- : IP raised Net::SSH::AuthenticationFailed with msg "Authentication failed for user backup@IP"
D, [2024-12-16T15:09:29.627287 #19416] DEBUG -- : lib/oxidized/node.rb: Oxidized::SSH failed for Switch_name
D, [2024-12-16T15:09:29.627311 #19416] DEBUG -- : lib/oxidized/job.rb: Config fetched for Switch_name at 2024-12-16 14:09:29 UTC
W, [2024-12-16T15:09:30.405218 #19416]  WARN -- : Switch_name status no_connection, retries exhausted, giving up

Switch Version (logged in from Centos7):

[oxidized@OXIDIZED oxidized]$ ssh backup@IP
HPE Management Console
Password:
Last login: Tue Dec 24 10:49:55 CET 2024 from 10.127.200.10 on pts/0
Number of total successful connections since last 1 days: 49
HPE Switch
CMP-PUL-SWC02-M2 [switch_name: standby] > enable
CMP-PUL-SWC02-M2 [switch_name standby] # show version
Product name:      Onyx
Product release:   3.10.3004
Build ID:          #1-dev
Build date:        2022-08-01 18:41:38
Target arch:       x86_64
Target hw:         x86_64
Built by:          sw-r2d2-bot@782111439e91
Version summary:   X86_64 3.10.3004 2022-08-01 18:41:38 x86_64

Product model:     x86onie
Host ID:           Host_ID
System serial num: Serial_number
System UUID:       System_UUID

Uptime:            387d 0h 49m 47.008s
CPU load averages: 0.04 / 0.07 / 0.04
Number of CPUs:    4
System memory:     2716 MB used / 5087 MB free / 7803 MB total
Swap:              0 MB used / 0 MB free / 0 MB total

The entry in the router.db file for the device does not differ from other switches that are being backed up successfully:
Switch_name:group:mlnxos:IP:backup:my_password:false:22:false:false:

Additionally, I have a default model for these types of devices:
/usr/local/rvm/gems/ruby-3.0.0/gems/oxidized-0.29.1/lib/oxidized/model/mlnxos.rb

For additional context, here is my setup:

  • Oxidized version: 0.29.1
  • Ruby version: 3.0.0
  • OS: CentOS Linux 7

I would appreciate any guidance on resolving this issue. If additional information is required, I’d be happy to provide it.
@robertcheramy
Copy link
Collaborator

Updated Issue for readability

@robertcheramy
Copy link
Collaborator

Please try if it works with the latest Oxidized version.

@graby0
Copy link
Author

graby0 commented Jan 7, 2025 •
edited
Loading

Updating Ruby from version 3.0.0 to version 3.1.4 and Oxidizez from version 0.29.1 to version 0.31 did not fix the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@graby0 @robertcheramy