"Invalid signature" appears at random from different commands

[beaugunderson]

I'm on Ubuntu 12.10 and hitting this very often which makes scripting the tools a huge pain.

Here's my simple repro:

#!/bin/bash

while true; do
  sdc-listmachines --limit 100 --offset 0

  EXIT_CODE=$?

  echo "Exit code: $EXIT_CODE"

  if [[ $EXIT_CODE -ne 0 ]]; then
    exit $EXIT_CODE
  fi
done

[mcavage]

Hi Beau,

Can you dump your node version and the version of this library you're using?

[beaugunderson]

◼ reginald/root ~ node -v
v0.8.22
◼ reginald/root ~ npm -g ls smartdc
/usr/local/lib
└── [email protected]

[mcavage]

Ok, I'm getting an ubuntu machine up to see if I can repro; anecdotally I couldn't on smartos or mac.

[mcavage]

Oh, and I should have asked - are you using SSH keys locally or with the agent, and is your key DSA or RSA (and either way, what size)?

[beaugunderson]

Here's env | grep SDC:

SDC_CLI_ACCOUNT=singlyinc
SDC_CLI_IDENTITY=/root/.ssh/id_rsa
SDC_CLI_URL=https://us-east-1.api.joyentcloud.com/
SDC_CLI_KEY_ID=id_rsa

I should note that this morning it took ~50 iterations of that repro before I saw the issue.

If you have logging on the Joyent end the IP I'm coming from is 173.255.212.44.

On Wednesday, March 27, 2013 at 12:59 PM, Mark Cavage wrote:

Oh, and I should have asked - are you using SSH keys locally or with the agent, and is your key DSA or RSA (and either way, what size)?

—
Reply to this email directly or view it on GitHub (#26 (comment)).

[mcavage]

Ok - I was able to get this to happen from an ubuntu client - I'm looking
into it.

On Wed, Mar 27, 2013 at 1:31 PM, Beau Gunderson [email protected]:

Here's env | grep SDC:

SDC_CLI_ACCOUNT=singlyinc
SDC_CLI_IDENTITY=/root/.ssh/id_rsa
SDC_CLI_URL=https://us-east-1.api.joyentcloud.com/
SDC_CLI_KEY_ID=id_rsa

I should note that this morning it took ~50 iterations of that repro
before I saw the issue.

If you have logging on the Joyent end the IP I'm coming from is
173.255.212.44.

On Wednesday, March 27, 2013 at 12:59 PM, Mark Cavage wrote:

Oh, and I should have asked - are you using SSH keys locally or with the
agent, and is your key DSA or RSA (and either way, what size)?

—
Reply to this email directly or view it on GitHub (
#26 (comment)).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-15551212
.

[mcavage]

Hi Beau,

Just to confirm, I can repro this, but I need some debug logs from production to diagnose further - which currently requires a restart, so this will take a little bit to get to the bottom of - I'm sorry for the trouble this is causing you; this is definitely not expected.

m

[beaugunderson]

Glad you were able to confirm, thanks for looking into this! :)

I have some code in place to retry if an sdc- command fails with Invalid signature so I'm OK for the short-term.

On Wednesday, March 27, 2013 at 3:34 PM, Mark Cavage wrote:

Hi Beau,
Just to confirm, I can repro this, but I need some debug logs from production to diagnose further - which currently requires a restart, so this will take a little bit to get to the bottom of - I'm sorry for the trouble this is causing you; this is definitely not expected.
m

—
Reply to this email directly or view it on GitHub (#26 (comment)).

[mcavage]

Hi Beau,

Just as an update, since it's been a couple days, I confirmed there's something where the client is signing a date header that is sometimes different than the one the server receives (i.e., it's off by exactly 1 second every time). I'm not sure yet whether the problem is node-smartdc, restify, or node itself yet - this is definitely a new regression though that didn't used to happen. I'll update the ticket again when I have a code fix.

m

[mcavage]

Hey Beau,

Ok, that took less time to track down than I thought -- can you try using it out of the #v6.5 branch and let me know if it fixes your cases? It should - the problem was that the Date header was getting dropped as the restify client was looking for date. I'll put a better fix into restify to take it case-insensitive, but for now I just made smartdc send what's expected. I suspect this broke when smartdc moved onto a newer restify, as the old one didn't try to do this.

[beaugunderson]

It's done 800 iterations without failing, seems like it's fixed to me! Thanks for the quick work. :)

[mcavage]

Great!

On Fri, Mar 29, 2013 at 4:19 PM, Beau Gunderson [email protected]:

It's done 800 iterations without failing, seems like it's fixed to me!
Thanks for the quick work. :)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-15665195
.