parseEsResponse not able to parse a "document not found" response

[gmarpons]

Version: OpensSearch 1.3.2.

It throws the following exception:

EsProtocolException
  { esProtoExMessage = "Original error was: Non-200 status code Error parse failure was: Error in $: key \"status\" not found",
    esProtoExResponse = "{\"_index\":XXXXX,\"_type\":\"_doc\",\"_id\":YYYYY,\"found\":false}"
  }

[blackheaven]

Did you use eitherDecodeResponse or parseEsResponse?

[gmarpons]

I'm using parseEsResponse, but now I'm thinking to switch to eitherDecodeResponse and avoid throwing, even if the value I receive in case of failure is less structured.

But the documentation of parseEsResponse says that any EsProtocolException should be reported.

The exception is thrown when I call getDocument with a non-existent document.

[blackheaven]

It's a bit weird, we have a test for this scenario, maybe the new interface will fix ambiguity.

I also have to make tests run against OpenSearch.

[1chb]

I get the same exception, also using Elasticsearch v 1.3.2, see below for details.

I checked the parse code, and the problem seems to be that ES does not provide a status field in its response, just as the exception message says:
"Original error was: Non-200 status code Error parse failure was: Error in $: key \"status\" not found"
The last part of the above failure message is provided by Aeson.

The ES response in this case contains these fields:

{
  "_index": "XXX",
  "_type": "_doc",
  "_id": "XXX",
  "_version": 1,
  "result": "not_found",
  "_shards": { "total": 2, "successful": 1, "failed": 0 },
  "_seq_no": 20124,
  "_primary_term": 1
}

But a status field is required by the JSON parser for EsError:

instance FromJSON EsError where
  parseJSON (Object v) =
    EsError
      <$> v .: "status"   <<<=== REQUIRED FIELD
      <*> (v .: "error" <|> (v .: "error" >>= (.: "reason")))
  parseJSON _ = empty

The above parser is used in:

parseEsResponse :: ( MonadThrow m, FromJSON body ) => BHResponse body -> m (ParsedEsResponse body)
parseEsResponse response
  | isSuccess response = case eitherDecode body of
      Right a -> return (Right a)
      Left err -> tryParseError err
  | otherwise = tryParseError "Non-200 status code"
  where
    body = responseBody $ getResponse response
    tryParseError originalError =
      case eitherDecode body of   <<<=== body CONTAINS THE ES RESPONSE.
        Right e -> return (Left e)      <<<=== THE TYPE OF e IS EsError
        -- Failed to parse the error message.  <<<=== BECAUSE IN THIS CASE THERE IS NO status FIELD
        Left err -> explode ("Original error was: " <> originalError <> " Error parse failure was: " <> err)
    explode errorMsg = throwM $ EsProtocolException (T.pack errorMsg) body

In the above code, isSuccess returns False, because the status is Not Found (should be 404, I think), so it continues to the otherwise branch where it executes tryParseError (still compatible with the exception message). This function fails to decode the ES response (body) and explodes (throws the exception).

My ES version is:

{
  "name" : "XXX",
  "cluster_name" : "XXX:test-opensearch",
  "cluster_uuid" : "XXX",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.3.2",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2023-03-12T18:08:54.377848Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

[blackheaven]

Hello @1chb,

Thanks for the report.

Are you using hackage version or the git version?

IIRC, it was fixed on the main branch, I'll have to do a release.

[1chb]

Hackage.

[blackheaven]

I have made a release