Raw response is not present when the response model is an Iterable

[chrishiste]

Describe the bug
When using the instructor library, there seems to be an issue with the raw response not being available when the response model is an Iterable.

To Reproduce

from typing import Iterable

import instructor
from openai import OpenAI
from pydantic import BaseModel


client = instructor.patch(OpenAI())


class UserDetail(BaseModel):
    name: str
    age: int


users = client.chat.completions.create(
    model="gpt-3.5-turbo",
    response_model=Iterable[UserDetail],
    messages=[
        {"role": "user", "content": "Give me 3 users"},
    ],
)

print(users) # [UserDetail(name='Alice', age=25), UserDetail(name='Bob', age=30), UserDetail(name='Charlie', age=35)]
assert users._raw_response is not None # AttributeError: 'list' object has no attribute '_raw_response'

Expected behavior
The expected behavior is that the raw response should be available and accessible.

[indigoviolet]

I was just about to file another issue related to this.

I think an instructor.patch-ed client should always return a wrapper object (let's call it) InstructorResponseWrapper, with attributes: raw_response, response_model_instance (which would be None if validation_error is non-None, validation_error (etc), multi_task_model. It could have a raise_for_validation() flag or method if that is needed.

This would allow a user to fall back to other strategies or have debug metadata if the Pydantic casting failed. Currently it's a bit of a take-it-all or leave-it.

[Tedfulk]

Your question got me digging and help me put some pieces together about instructor. Your error makes sense because at high level this line response_model=Iterable[UserDetail] says, what is expected to be returned is a regular python list/iterable. And that's what you get back. But lists in python don't have a method or attributes called '_raw_response'.
You must specify a specific model in the response_model param. For example if you wanted a list of users back with the raw response.

from typing import List

import instructor
from openai import OpenAI
from pydantic import BaseModel


client = instructor.patch(
    OpenAI()
)


class UserDetail(BaseModel):
    name: str
    age: int



class UserDetailList(BaseModel):
    users: List[UserDetail]


users = client.chat.completions.create(
    model="gpt-3.5-turbo",
    response_model=UserDetailList,
    messages=[
        {"role": "user", "content": "Give me 3 users"},
    ],
)


print(users._raw_response)
# ChatCompletion(id='chatcmpl-8lBAU77cgwRjc3u5cvlH050m9TEAK', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=None, role='assistant', function_call=FunctionCall(arguments='{\n  "users": [\n    { "name": "John", "age": 25 },\n    { "name": "Emily", "age": 30 },\n    { "name": "Michael", "age": 35 }\n  ]\n}', name='UserDetailList'), tool_calls=None))], created=1706255038, model='gpt-3.5-turbo-0613', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=51, prompt_tokens=80, total_tokens=131))

The _raw_response is tied to the parent model UserDetailList not the child model UserDetail. Judging by all his examples though this is the intent. I could be wrong but this was my understanding. Hope that's helpful.

[indigoviolet]

@Tedfulk I think it would make a lot of sense to try to provide the raw response in some form (see my suggestion above) -- there is valuable metadata in there.

[Tedfulk]

@indigoviolet sorry I didn't see your reply in time before I posted. Hmm that does sound nice. I basically just use _raw_response to check out responses coming back or token limit, and do another round of validation if need be. Although pydantic and insructor do a good job of validating the llms response, or errors. Usually cuz the open source models cant figure out json.

Idk I'm still leaning towards how they are now by having the response model be a specific model I can expect to be returned with a hard pattern to adhere to.

[chrishiste]

@Tedfulk I'm using the same workaround at the moment but it requires more boilerplate. Would be nice if the returned value was always an object with the default value correctly typed to the response_model value but with extra properties like @indigoviolet is describing. For this Iterator case maybe a simple __str__ or __repr__ would work? Hiding the fact that the response is an object

[jxnl]

Yeah, the iterable type hint really is just to help when stream=True Honestly, I think using the parent class is probably the right way forward if you do want to access those attributes.

[eware-godaddy]

Unfortunately using the parent model workaround which is not of type Iterable currently doesn't allow for streaming in the current implementation from what I can tell. Would be nice to be able to have both, streaming and access to the raw response somehow, even if by some other mechanism like a context manager.

[jacobtohahn]

I'm running into the same issue as @chrishiste. For now, I'm going to workaround using a parent model as described.

That being said, Extracting Tasks using Iterable in the docs makes it seem as if this is supposed to work in general with models (regardless of if they also include the raw response), so this probably is a bug. Is there any update on a fix besides the workaround?