Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coinbase: Update exchange implementation #1480

Open
wants to merge 111 commits into
base: master
Choose a base branch
from

Conversation

cranktakular
Copy link
Collaborator

@cranktakular cranktakular commented Feb 14, 2024

PR Description

Updating Coinbase to the Advanced Trade and Sign In With Coinbase APIs, as well as a few tiny fixes of other parts of the codebase found along the way.

Type of change

Please delete options that are not relevant and add an x in [] as item is complete.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How has this been tested

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration and
also consider improving test coverage whilst working on a certain feature or package.

  • go test ./... -race
  • golangci-lint run

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation and regenerated documentation via the documentation tool
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally and on Github Actions with my changes

The only tests which seem to be failing (exchanges/kucoin, database/models/postgres, config, and common/file) are parts that I haven't substantively changed, and which seem to be failing on master too.

Continual enhance of Coinbase tests

The revamp continues

Oh jeez the Orderbook part's unfinished don't look

Coinbase revamp, Orderbook still unfinished
V3 done, onto V2

Coinbase revamp nears completion

Coinbase revamp nears completion

Test commit should fail

Coinbase revamp nears completion
@thrasher- thrasher- changed the title Coinbase api revamp Coinbase: Update exchange implementation Feb 14, 2024
@thrasher- thrasher- requested a review from a team February 14, 2024 23:40
Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy with your changes, thanks! There's an intermitted exchange_wrapper_standards tests breaking now post-master-merge. I think I'm good once that's sorted

Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just have a question on a comment. Then after that if you could rebase/merge I can approve

exchanges/subscription/template_test.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately it has been difficult to test the scenario due to CBP websocket not respecting config subscriptions:

 "subscriptions": [
     {
      "enabled": false,
      "channel": "heartbeat"
     },
     {
      "enabled": false,
      "channel": "status"
     },
     {
      "enabled": false,
      "channel": "ticker",
      "asset": "spot"
     },
     {
      "enabled": false,
      "channel": "candles",
      "asset": "spot"
     },
     {
      "enabled": false,
      "channel": "allTrades",
      "asset": "spot"
     },
     {
      "enabled": false,
      "channel": "orderbook",
      "asset": "spot"
     },
     {
      "enabled": true,
      "channel": "account",
      "authenticated": true
     },
     {
      "enabled": false,
      "channel": "ticker_batch",
      "asset": "spot"
     }
[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","product_ids":["BTC-USD"],"channel":"candles","timestamp":"1730245901"}
[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","channel":"heartbeats","timestamp":"1730245901"}
[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","channel":"user","signature":"POP","api_key":"CAT","timestamp":"1730245901"}
[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","product_ids":["BTC-USD"],"channel":"market_trades","timestamp":"1730245901"}
[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","product_ids":["BTC-USD"],"channel":"level2","timestamp":"1730245901"}
[DEBUG] | WEBSOCKET | 30/10/2024 10:51:41 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","product_ids":["BTC-USD"],"channel":"ticker","timestamp":"1730245901"}

This will need to be fixed

Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not many requests other than the reversion I already commented on. Good stuff

Type: oType,
Side: oSide,
Status: oStatus,
AssetType: asset.Spot,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the usage of asset.All for "user" and that this can return productType of either SPOT or FUTURE I think you should be returning the appropriate asset type

Comment on lines 494 to 509
func statusToStandardStatus(stat string) (order.Status, error) {
switch stat {
case "received":
return order.New, nil
case "open":
return order.Active, nil
case "done":
return order.Filled, nil
case "match":
return order.PartiallyFilled, nil
case "change", "activate":
return order.Active, nil
default:
return order.UnknownStatus, fmt.Errorf("%w %v", errUnrecognisedStatusType, stat)
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These may have changed since you did this:
https://docs.cdp.coinbase.com/advanced-trade/docs/ws-channels#user-channel
I note that CB seems to be doing things in upper case and I'm not seeing case handling anywhere

Comment on lines 262 to 275
if err != nil {
return warnString, err
}
var oSide order.Side
oSide, err = order.StringToOrderSide(wsUser[i].Orders[j].OrderSide)
if err != nil {
return warnString, err
}
var oStatus order.Status
oStatus, err = statusToStandardStatus(wsUser[i].Orders[j].Status)
if err != nil {
return warnString, err
}
sliToSend = append(sliToSend, order.Detail{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the following just for these instances of if err != nil

if err != nil {
	c.Websocket.DataHandler <- order.ClassificationError{
				Exchange: c.Name,
				Err:      err,
	}
}

Comment on lines 38 to 40
// FullPayloadSubscribe flushes and changes full subscription on websocket
// connection by subscribing with full default stream channel list
FullPayloadSubscribe bool `json:"fullPayloadSubscribe,omitempty"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This appears unused and the only addition

Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good stuff minor requests.

startDateString = "start_date"
endDateString = "end_date"

errIntervalNotSupported = "interval not supported"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return an error then 🚀

if err := c.SendHTTPRequest(ctx, exchange.RestSpot, path, &orderbook); err != nil {
return nil, err
// GetProductBookV3 returns a list of bids/asks for a single product
func (c *CoinbasePro) GetProductBookV3(ctx context.Context, productID string, limit uint16, authenticated bool) (*ProductBook, error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not done


return fee
// UnmarshalJSON unmarshals the JSON input into a UnixTimestamp type
func (t *UnixTimestamp) UnmarshalJSON(b []byte) error {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove UnixTimestamp type and use types.Time

} else if c.Equal(currency.EUR) {
fee = 0.15
// UnmarshalJSON unmarshals the JSON input into a UnixTimestamp type
func (t *UnixTimestampMilli) UnmarshalJSON(b []byte) error {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove UnixTimestampMilli and use types.Time

}

// TestWsAuth dials websocket, sends login request.
func TestWsAuth(t *testing.T) {
if !c.Websocket.IsEnabled() && !c.API.AuthenticatedWebsocketSupport || !sharedtestvalues.AreAPICredentialsSet(c) {
// t.Parallel()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reinstate?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just gonna rm that, since other tests interfere with it.


// AmCur is a sub-struct used in ListNotificationsSubData, WalletData, TransactionData,
// DeposWithdrData, and PaymentMethodData
type AmCur struct {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AmCur -> AmountWithCurrency

// OrderBookResp holds information on bids and asks for a particular currency pair, used for unmarshalling in
// GetProductBookV1
type OrderBookResp struct {
Bids [][3]interface{} `json:"bids"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please convert interface{} -> any across this PR.

// ValCur is a sub-struct used in the types Account, NativeAndRaw, DetailedPortfolioResponse,
// FuturesBalanceSummary, ListFuturesSweepsResponse, PerpetualsPortfolioSummary, PerpPositionDetail,
// FeeStruct, AmScale, and ConvertResponse
type ValCur struct {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValCur -> ValueWithCurrency

Comment on lines 538 to 542
timestamp, err := time.Parse(time.RFC3339, string(data))
if err != nil {
return time.Time{}, err
}
return timestamp, nil
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
timestamp, err := time.Parse(time.RFC3339, string(data))
if err != nil {
return time.Time{}, err
}
return timestamp, nil
return time.Parse(time.RFC3339, string(data))

}

// strategyDecoder is a helper function that converts a Coinbase Pro time in force string to a few standardised bools
func strategyDecoder(str string) (bool, bool, error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

linter issue here

Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are going to have to swap over to using JWT authentication. I cannot test the old key format.

requestFmt := &currency.PairFormat{Delimiter: currency.DashDelimiter, Uppercase: true}
configFmt := &currency.PairFormat{Delimiter: currency.DashDelimiter, Uppercase: true}
err := c.SetGlobalPairsManager(requestFmt, configFmt, asset.Spot)
err := c.SetGlobalPairsManager(requestFmt, configFmt, asset.Spot, asset.Futures)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add:

diff --git a/exchanges/coinbasepro/coinbasepro_wrapper.go b/exchanges/coinbasepro/coinbasepro_wrapper.go
index a675e7ba9..8ca87ef5a 100644
--- a/exchanges/coinbasepro/coinbasepro_wrapper.go
+++ b/exchanges/coinbasepro/coinbasepro_wrapper.go
@@ -44,6 +44,10 @@ func (c *CoinbasePro) SetDefaults() {
        if err != nil {
                log.Errorln(log.ExchangeSys, err)
        }
+       err = c.DisableAssetWebsocketSupport(asset.Futures)
+       if err != nil {
+               log.Errorln(log.ExchangeSys, err)
+       }
        c.Features = exchange.Features{
                Supports: exchange.FeaturesSupported{
                        REST:      true,
}

As there are currently no websocket market data support for futures.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You want that done, even though there is futures websocket support, limited to the user's orders?

sort.Sort(trade.ByDate(resp))
return resp, nil
// GetRecentTrades returns the most recent trades for a currency and asset
func (c *CoinbasePro) GetRecentTrades(_ context.Context, _ currency.Pair, _ asset.Item) ([]trade.Data, error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
func (c *CoinbasePro) GetRecentTrades(_ context.Context, _ currency.Pair, _ asset.Item) ([]trade.Data, error) {
func (c *CoinbasePro) GetRecentTrades(context.Context, currency.Pair, asset.Item) ([]trade.Data, error) {

Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick review on new changes. Then I can do a thorough review before next week.

Comment on lines 1524 to 1547
// The code below mostly works, but seems to lead to bad results on the signature step. Deferring until later
// head := map[string]any{"kid": creds.Key, "typ": "JWT", "alg": "ES256", "nonce": nonce}
// headJSON, err := json.Marshal(head)
// if err != nil {
// return "", time.Time{}, err
// }
// headEncode := base64URLEncode(headJSON)
// regTime := time.Now()
// body := map[string]any{"iss": "cdp", "nbf": regTime.Unix(), "exp": regTime.Add(time.Minute * 2).Unix(), "sub": creds.Key /*, "aud": "retail_rest_api_proxy"*/}
// if uri != "" {
// body["uri"] = uri
// }
// bodyJSON, err := json.Marshal(body)
// if err != nil {
// return "", time.Time{}, err
// }
// bodyEncode := base64URLEncode(bodyJSON)
// hash := sha256.Sum256([]byte(headEncode + "." + bodyEncode))
// sig, err := ecdsa.SignASN1(rand.Reader, key, hash[:])
// if err != nil {
// return "", time.Time{}, err
// }
// sigEncode := base64URLEncode(sig)
// return headEncode + "." + bodyEncode + "." + sigEncode, regTime, nil
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this for?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As comment at the top of the block says, it's my implementation of the JWT protocol, so we won't have to import a separate package to handle that. Alas, there's some issue I can't identify with generating the signature, which has it create an invalid one, so this can't be used right now.

return headEncode + "." + bodyEncode + "." + sigEncode, nil
// GetWSJWT returns a JWT, using a stored one of it's provided, and generating a new one otherwise
func (c *CoinbasePro) GetWSJWT() (string, error) {
c.mut.RLock()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: You can simplify this with a normal mutex and just defer the unlock. Then you don't need to worry about most of whats going on below. We can optimise later if this turns into a bottleneck which it most likely won't be.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a point in doing that now, considering how I've already written this handling?

If so, let me know, and I'll refactor it.

// GetWSJWT returns a JWT, using a stored one of it's provided, and generating a new one otherwise
func (c *CoinbasePro) GetWSJWT() (string, error) {
c.mut.RLock()
if c.jwtLastRegen.Add(time.Minute * 2).After(time.Now()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of JWTLasTRegen just assign an expiry value so the addition op doesn't need to happen.

Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work with the JWT support! Final minor comments

t.Skip(stream.ErrWebsocketNotEnabled.Error())
}
var dialer websocket.Dialer
err := c.Websocket.Conn.Dial(&dialer, http.Header{})
require.NoError(t, err, "Dial must not error")
require.NoError(t, err)
go c.wsReadData()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This design can cause issue on c.wsReadData errors because of the defer. You get a panic for a negative waitgroup. That and TestMain also calls this which is likely what causes the negative waitgroup.

edit: My issue might actually be with TestMain, why does it dial? Your other tests don't need it. Remove the dialing from TestMain

Copy link
Collaborator Author

@cranktakular cranktakular Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without dialing in TestMain, TestSubscribeUnsubscribe errors out when run as a single test.

Seems like negative waitgroup would come about since this never increments the waitgroup, but readData decrements it when closing. I'll add an increment in this call, so maybe that'll fix it?

}
if !partialSkip {
_, err = c.ScheduleFuturesSweep(context.Background(), 0.001337)
assert.NoError(t, err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems more appropriate to require

id, err := uuid.NewV4()
assert.NoError(t, err)
resp, err := c.PlaceOrder(context.Background(), id.String(), testPair.String(), order.Sell.String(), "", order.Limit.String(), "", "CROSS", "", testAmount, testPrice, 0, 9999, false, time.Now().Add(time.Hour))
assert.NoError(t, err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems more appropriate to require

Copy link
Collaborator Author

@cranktakular cranktakular Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that's necessary. The results of this test aren't used later, and this test returning an error doesn't prevent resp itself from being checked for being empty.

Do you just suggest that switch so, on an error, only one message pops up for it?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert.NotEmpty(t, resp, errExpectedNonEmpty) isn't very useful if the result is an error

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understandable, but it also doesn't seem great to stop the test on the first error if there's novel stuff afterwards which isn't impacted by that error. I've rewritten many tests, including this one, with that in mind.

sharedtestvalues.SkipTestIfCredentialsUnset(t, c)
skipTestIfLowOnFunds(t)
resp, err := c.PreviewOrder(context.Background(), testPair.String(), "BUY", "MARKET", "", "ISOLATED", 0, testAmount, 0, 0, 0, 0, false, false, false, time.Time{})
assert.NoError(t, err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

require would be more appropriate

Comment on lines +580 to +582
if len(pmID) == 0 {
t.Skip(skipPayMethodNotFound)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know we talked about this stuff before, I'm not requesting the change for this PR. In future PRs, I don't think you should be doing this.
I think it is more appropriate to provide made up data than to do this. The testing that we do when we check the endpoints is to see whether the endpoint will complain from say "invalid id" versus "missing param" it helps to identify issues with your implementations even if the ID themselves are not valid. A user can always provide their own IDs in the fields.
Plus it simplifies your tests so you don't need to get so much

Thankfully, removing them all and testing the endpoints gave me the results I was after

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alas, my next PR's already largely set up in this sorta way (._.)

Comment on lines 450 to 451
// The new JWT-based keys don't have permissions to edit portfolios they aren't assigned to, causing this to fail
if err != nil && err.Error() != errPortfolioNameDuplicate {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment and the error check do not match. I think you should remove && err.Error() != errPortfolioNameDuplicate
It is acceptable for authenticated tests to fail on execution for reasons such as "this already exists". I think hardcoding an error response to a specific json string will be very likely to fail in future anyway, even if you have a duplicate portfolio

Is the comment about something else?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah the comment's about something else, I probably should've placed it one line lower to reduce confusion.

assert.ErrorIs(t, err, errNameEmpty)
sharedtestvalues.SkipTestIfCredentialsUnset(t, c, canManipulateRealOrders)
_, err = c.CreatePortfolio(context.Background(), "GCT Test Portfolio")
if err != nil && err.Error() != errPortfolioNameDuplicate {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove && err.Error() != errPortfolioNameDuplicate

@@ -3,30 +3,61 @@ package coinbasepro
import (
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[DEBUG] | WEBSOCKET | 14/01/2025 14:34:06 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Message received: {"channel":"ticker","client_id":"","timestamp":"2025-01-14T03:34:05.207247116Z","sequence_num":6,"events":[{"type":"snapshot","tickers":[{"type":"ticker","product_id":"BTC-USD","price":"94873.88","volume_24_h":"13052.57027986","low_24_h":"89028.64","high_24_h":"95254.56","low_52_w":"38501","high_52_w":"108388.88","price_percent_chg_24_h":"0.58560330817365","best_bid":"94886.93","best_ask":"94886.94","best_bid_quantity":"0.00065345","best_ask_quantity":"0.03299181"}]}]}
[DEBUG] | WEBSOCKET | 14/01/2025 14:34:06 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Message received: {"channel":"ticker","client_id":"","timestamp":"2025-01-14T03:34:05.207270615Z","sequence_num":7,"events":[{"type":"snapshot","tickers":[{"type":"ticker","product_id":"ETH-USD","price":"3167.46","volume_24_h":"169211.37256592","low_24_h":"2913.75","high_24_h":"3260.94","low_52_w":"2116.02","high_52_w":"4109","price_percent_chg_24_h":"-2.71002460307953","best_bid":"3166.88","best_ask":"3166.89","best_bid_quantity":"2.23297081","best_ask_quantity":"0.00130411"}]}]}
[DEBUG] | WEBSOCKET | 14/01/2025 14:34:06 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Message received: {"channel":"subscriptions","client_id":"","timestamp":"2025-01-14T03:34:05.207286942Z","sequence_num":8,"events":[{"subscriptions":{"candles":["ETH-USD","BTC-USD"],"heartbeats":["heartbeats"],"ticker":["BTC-USD","ETH-USD"],"user":["3a054eaf-0164-5ca2-872c-a0d64bbc0d79"]}}]}
[ERROR] | WEBSOCKET | 14/01/2025 14:34:06 | could not sync new data for CoinbasePro BTC USD spot Ticker
[DEBUG] | SYNC | 14/01/2025 14:34:06 | CoinbasePro Ticker sync complete ETH-USD [3/6].

Eth works, btc doesn't

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This continues with orderbook:

[DEBUG] | WEBSOCKET | 14/01/2025 14:34:08 | CoinbasePro wss://advanced-trade-ws.coinbase.com: Message received: {"channel":"l2_data","client_id":"","timestamp":"2025-01-14T03:34:05.875997282Z","sequence_num":17,"events":[{"type":"update","product_id":"BTC-USD","updates":[{"side":"bid","event_time":"2025-01-14T03:34:05.194124Z","price_level":"94686.13","new_quantity":"0"},{"side":"bid","event_time":"2025-01-14T03:34:05.194124Z","price_level":"83026.07","new_quantity":"0.00121044"},{"side":"offer","event_time":"2025-01-14T03:34:05.194124Z","price_level":"94905.66","new_quantity":"0"},{"side":"offer","event_time":"2025-01-14T03:34:05.194124Z","price_level":"94942.44","new_quantity":"0"},{"side":"offer","event_time":"2025-01-14T03:34:05.194124Z","price_level":"94942.45","new_quantity":"0.00052708"},{"side":"offer","event_time":"2025-01-14T03:34:05.194124Z","price_level":"94943.86","new_quantity":"0.2634787"}]}]}
[ERROR] | WEBSOCKET | 14/01/2025 14:34:08 | could not sync new data for CoinbasePro BTC USD spot Orderbook

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its because I have BTC-USDC as the enabled pair. Not sure why BTC USD is being returned. I think this is something you've highlighted in the past. Needs to get resolved though

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what we can do to resolve that. Looking into things, it seems like they're pruning it down on their end:

CoinbasePro wss://advanced-trade-ws.coinbase.com: Sending message: {"type":"subscribe","product_ids":["BTC-USD","BTC-USDC","ETH-USD","ETH-USDC"],"channel":"ticker","timestamp":"1736833475"}

CoinbasePro wss://advanced-trade-ws.coinbase.com: Message received: {"channel":"subscriptions","client_id":"","timestamp":"2025-01-14T05:44:36.121096821Z","sequence_num":2,"events":[{"subscriptions":{"ticker":["BTC-USD","ETH-USD"]}}]}

All I can see us doing is some sort of reinterpretation. Taking USD tickers received in websocket as referring to USDC, or saying that USDC is invalid for websocket.

Copy link
Collaborator

@gloriousCode gloriousCode Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will need to be fixed before a merge though. Looks like data already returns with this kind of translation:

https://api.coinbase.com/api/v3/brokerage/products?limit=0&product_type=SPOT
products":[{"product_id":"BTC-USDC", ... ,"alias":"BTC-USD",

Let us know if you need help

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a few ideas on how it could be handled, and think many of them are easy enough to implement, but what sort of approach do y'all think is best?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My immediate thought would be something like

type butts struct {
 associatedAliases map[alias]*[]currencies
 m sync.Mutex
}

Add it to Coinbase as an unexported property and when UpdateTradablePairs is called reset associatedAliases and populate it with the alias data for each enabled pair. When order book data comes in, lock, publish to all currency pairs associated with the alias. In the example above BTC-USD data would come in and coinbase_websocket would publish to BTC USDC

There's likely nuances that I haven't considered and I'm happy to look into whatever you do as well.

sharedtestvalues.SkipTestIfCredentialsUnset(t, c)
resp, err = c.GetProductBookV3(context.Background(), testPair, 4, -1, true)
assert.NoError(t, err)
require.NoError(t, err)
Copy link
Collaborator

@gloriousCode gloriousCode Jan 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as an FYI, not a required change, asserts' have bool responses, so if you don't like the require, you can do

if assert.NoError(errawr) {
    assert.NotEmpty(kasjhdfaksdjfhasldkfh)
}

@@ -3,30 +3,61 @@ package coinbasepro
import (
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My immediate thought would be something like

type butts struct {
 associatedAliases map[alias]*[]currencies
 m sync.Mutex
}

Add it to Coinbase as an unexported property and when UpdateTradablePairs is called reset associatedAliases and populate it with the alias data for each enabled pair. When order book data comes in, lock, publish to all currency pairs associated with the alias. In the example above BTC-USD data would come in and coinbase_websocket would publish to BTC USDC

There's likely nuances that I haven't considered and I'm happy to look into whatever you do as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
high priority review me This pull request is ready for review test(s) fix This is to denote the PR is fixing a build test issue
Projects
Status: In review
Development

Successfully merging this pull request may close these issues.

5 participants