client.prompts.log({ ...params }) -> Humanloop.CreatePromptLogResponse
-
-
-
Log to a Prompt.
You can use query parameters
version_id
, orenvironment
, to target an existing version of the Prompt. Otherwise, the default deployed version will be chosen.Instead of targeting an existing version explicitly, you can instead pass in Prompt details in the request body. In this case, we will check if the details correspond to an existing version of the Prompt. If they do not, we will create a new version. This is helpful in the case where you are storing or deriving your Prompt details in code.
-
-
-
await client.prompts.log({ path: "persona", prompt: { model: "gpt-4", template: [ { role: "system", content: "You are {{person}}. Answer questions as this person. Do not break character.", }, ], }, messages: [ { role: "user", content: "What really happened at Roswell?", }, ], inputs: { person: "Trump", }, createdAt: "2024-07-19T00:29:35.178992", providerLatency: 6.5931549072265625, outputMessage: { content: "Well, you know, there is so much secrecy involved in government, folks, it's unbelievable. They don't want to tell you everything. They don't tell me everything! But about Roswell, it's a very popular question. I know, I just know, that something very, very peculiar happened there. Was it a weather balloon? Maybe. Was it something extraterrestrial? Could be. I'd love to go down and open up all the classified documents, believe me, I would. But they don't let that happen. The Deep State, folks, the Deep State. They're unbelievable. They want to keep everything a secret. But whatever the truth is, I can tell you this: it's something big, very very big. Tremendous, in fact.", role: "assistant", }, promptTokens: 100, outputTokens: 220, promptCost: 0.00001, outputCost: 0.0002, finishReason: "stop", });
-
-
-
request:
Humanloop.PromptLogRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.updateLog(id, logId, { ...params }) -> Humanloop.LogResponse
-
-
-
Update a Log.
Update the details of a Log with the given ID.
-
-
-
await client.prompts.updateLog("id", "log_id");
-
-
-
id:
string
β Unique identifier for Prompt.
-
logId:
string
β Unique identifier for the Log.
-
request:
Humanloop.PromptLogUpdateRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.callStream({ ...params }) -> core.Stream
-
-
-
Call a Prompt.
Calling a Prompt calls the model provider before logging the request, responses and metadata to Humanloop.
You can use query parameters
version_id
, orenvironment
, to target an existing version of the Prompt. Otherwise the default deployed version will be chosen.Instead of targeting an existing version explicitly, you can instead pass in Prompt details in the request body. In this case, we will check if the details correspond to an existing version of the Prompt. If they do not, we will create a new version. This is helpful in the case where you are storing or deriving your Prompt details in code.
-
-
-
await client.prompts.callStream({ versionId: "string", environment: "string", path: "string", id: "string", messages: [ { content: "string", name: "string", toolCallId: "string", role: "user", toolCalls: [ { id: "string", type: "function", function: { name: "string", arguments: undefined, }, }, ], }, ], toolChoice: "none", prompt: { model: "string", endpoint: undefined, template: undefined, provider: undefined, maxTokens: undefined, temperature: undefined, topP: undefined, stop: undefined, presencePenalty: undefined, frequencyPenalty: undefined, other: undefined, seed: undefined, responseFormat: undefined, tools: undefined, linkedTools: undefined, attributes: undefined, }, inputs: { string: { key: "value", }, }, source: "string", metadata: { string: { key: "value", }, }, startTime: "2024-01-15T09:30:00Z", endTime: "2024-01-15T09:30:00Z", sourceDatapointId: "string", traceParentId: "string", user: "string", promptsCallStreamRequestEnvironment: "string", save: true, logId: "string", providerApiKeys: { openai: "string", ai21: "string", mock: "string", anthropic: "string", bedrock: "string", cohere: "string", openaiAzure: "string", openaiAzureEndpoint: "string", }, numSamples: 1, returnInputs: true, logprobs: 1, suffix: "string", });
-
-
-
request:
Humanloop.PromptsCallStreamRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.call({ ...params }) -> Humanloop.PromptCallResponse
-
-
-
Call a Prompt.
Calling a Prompt calls the model provider before logging the request, responses and metadata to Humanloop.
You can use query parameters
version_id
, orenvironment
, to target an existing version of the Prompt. Otherwise the default deployed version will be chosen.Instead of targeting an existing version explicitly, you can instead pass in Prompt details in the request body. In this case, we will check if the details correspond to an existing version of the Prompt. If they do not, we will create a new version. This is helpful in the case where you are storing or deriving your Prompt details in code.
-
-
-
await client.prompts.call({ path: "persona", prompt: { model: "gpt-4", template: [ { role: "system", content: "You are stockbot. Return latest prices.", }, ], tools: [ { name: "get_stock_price", description: "Get current stock price", parameters: { type: "object", properties: { ticker_symbol: { type: "string", name: "Ticker Symbol", description: "Ticker symbol of the stock", }, }, required: [], }, }, ], }, messages: [ { role: "user", content: "latest apple", }, ], });
-
-
-
request:
Humanloop.PromptsCallRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.list({ ...params }) -> core.Page
-
-
-
Get a list of all Prompts.
-
-
-
await client.prompts.list({ size: 1, });
-
-
-
request:
Humanloop.ListPromptsGetRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.upsert({ ...params }) -> Humanloop.PromptResponse
-
-
-
Create a Prompt or update it with a new version if it already exists.
Prompts are identified by the
ID
or theirpath
. The parameters (i.e. the prompt template, temperature, model etc.) determine the versions of the Prompt.If you provide a commit message, then the new version will be committed; otherwise it will be uncommitted. If you try to commit an already committed version, an exception will be raised.
-
-
-
await client.prompts.upsert({ path: "Personal Projects/Coding Assistant", model: "gpt-4o", endpoint: "chat", template: [ { content: "You are a helpful coding assistant specialising in {{language}}", role: "system", }, ], provider: "openai", maxTokens: -1, temperature: 0.7, commitMessage: "Initial commit", });
-
-
-
request:
Humanloop.PromptRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.get(id, { ...params }) -> Humanloop.PromptResponse
-
-
-
Retrieve the Prompt with the given ID.
By default, the deployed version of the Prompt is returned. Use the query parameters
version_id
orenvironment
to target a specific version of the Prompt.
-
-
-
await client.prompts.get("pr_30gco7dx6JDq4200GVOHa");
-
-
-
id:
string
β Unique identifier for Prompt.
-
request:
Humanloop.GetPromptsIdGetRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.delete(id) -> void
-
-
-
Delete the Prompt with the given ID.
-
-
-
await client.prompts.delete("pr_30gco7dx6JDq4200GVOHa");
-
-
-
id:
string
β Unique identifier for Prompt.
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.move(id, { ...params }) -> Humanloop.PromptResponse
-
-
-
Move the Prompt to a different path or change the name.
-
-
-
await client.prompts.move("pr_30gco7dx6JDq4200GVOHa", { path: "new directory/new name", });
-
-
-
id:
string
β Unique identifier for Prompt.
-
request:
Humanloop.UpdatePromptRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.listVersions(id, { ...params }) -> Humanloop.ListPrompts
-
-
-
Get a list of all the versions of a Prompt.
-
-
-
await client.prompts.listVersions("pr_30gco7dx6JDq4200GVOHa", { status: "committed", });
-
-
-
id:
string
β Unique identifier for Prompt.
-
request:
Humanloop.ListVersionsPromptsIdVersionsGetRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.commit(id, versionId, { ...params }) -> Humanloop.PromptResponse
-
-
-
Commit a version of the Prompt with a commit message.
If the version is already committed, an exception will be raised.
-
-
-
await client.prompts.commit("pr_30gco7dx6JDq4200GVOHa", "prv_F34aba5f3asp0", { commitMessage: "Reiterated point about not discussing sentience", });
-
-
-
id:
string
β Unique identifier for Prompt.
-
versionId:
string
β Unique identifier for the specific version of the Prompt.
-
request:
Humanloop.CommitRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.deletePromptVersion(id, versionId) -> void
-
-
-
Delete a version of the Prompt.
-
-
-
await client.prompts.deletePromptVersion("id", "version_id");
-
-
-
id:
string
β Unique identifier for Prompt.
-
versionId:
string
β Unique identifier for the specific version of the Prompt.
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.setDeployment(id, environmentId, { ...params }) -> Humanloop.PromptResponse
-
-
-
Deploy Prompt to an Environment.
Set the deployed version for the specified Environment. This Prompt will be used for calls made to the Prompt in this Environment.
-
-
-
await client.prompts.setDeployment("id", "environment_id", { versionId: "version_id", });
-
-
-
id:
string
β Unique identifier for Prompt.
-
environmentId:
string
β Unique identifier for the Environment to deploy the Version to.
-
request:
Humanloop.SetDeploymentPromptsIdEnvironmentsEnvironmentIdPostRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.removeDeployment(id, environmentId) -> void
-
-
-
Remove deployed Prompt from the Environment.
Remove the deployed version for the specified Environment. This Prompt will no longer be used for calls made to the Prompt in this Environment.
-
-
-
await client.prompts.removeDeployment("id", "environment_id");
-
-
-
id:
string
β Unique identifier for Prompt.
-
environmentId:
string
β Unique identifier for the Environment to remove the deployment from.
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.listEnvironments(id) -> Humanloop.FileEnvironmentResponse[]
-
-
-
List all Environments and their deployed versions for the Prompt.
-
-
-
await client.prompts.listEnvironments("pr_30gco7dx6JDq4200GVOHa");
-
-
-
id:
string
β Unique identifier for Prompt.
-
requestOptions:
Prompts.RequestOptions
-
-
client.prompts.updateMonitoring(id, { ...params }) -> Humanloop.PromptResponse
-
-
-
Activate and deactivate Evaluators for monitoring the Prompt.
An activated Evaluator will automatically be run on all new Logs within the Prompt for monitoring purposes.
-
-
-
await client.prompts.updateMonitoring("pr_30gco7dx6JDq4200GVOHa", { activate: [ { evaluatorVersionId: "evv_1abc4308abd", }, ], });
-
-
-
id:
string
-
request:
Humanloop.EvaluatorActivationDeactivationRequest
-
requestOptions:
Prompts.RequestOptions
-
-
client.tools.log({ ...params }) -> Humanloop.CreateToolLogResponse
-
-
-
Log to a Tool.
You can use query parameters
version_id
, orenvironment
, to target an existing version of the Tool. Otherwise the default deployed version will be chosen.Instead of targeting an existing version explicitly, you can instead pass in Tool details in the request body. In this case, we will check if the details correspond to an existing version of the Tool, if not we will create a new version. This is helpful in the case where you are storing or deriving your Tool details in code.
-
-
-
await client.tools.log({ path: "math-tool", tool: { function: { name: "multiply", description: "Multiply two numbers", parameters: { type: "object", properties: { a: { type: "number", }, b: { type: "number", }, }, required: ["a", "b"], }, }, }, inputs: { a: 5, b: 7, }, output: "35", });
-
-
-
request:
Humanloop.ToolLogRequest
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.update(id, logId, { ...params }) -> Humanloop.LogResponse
-
-
-
Update a Log.
Update the details of a Log with the given ID.
-
-
-
await client.tools.update("id", "log_id");
-
-
-
id:
string
β Unique identifier for Prompt.
-
logId:
string
β Unique identifier for the Log.
-
request:
Humanloop.ToolLogUpdateRequest
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.list({ ...params }) -> core.Page
-
-
-
Get a list of all Tools.
-
-
-
await client.tools.list({ size: 1, });
-
-
-
request:
Humanloop.ListToolsGetRequest
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.upsert({ ...params }) -> Humanloop.ToolResponse
-
-
-
Create a Tool or update it with a new version if it already exists.
Tools are identified by the
ID
or theirpath
. The name, description and parameters determine the versions of the Tool.If you provide a commit message, then the new version will be committed; otherwise it will be uncommitted. If you try to commit an already committed version, an exception will be raised.
-
-
-
await client.tools.upsert({ path: "math-tool", function: { name: "multiply", description: "Multiply two numbers", parameters: { type: "object", properties: { a: { type: "number", }, b: { type: "number", }, }, required: ["a", "b"], }, }, commitMessage: "Initial commit", });
-
-
-
request:
Humanloop.ToolRequest
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.get(id, { ...params }) -> Humanloop.ToolResponse
-
-
-
Retrieve the Tool with the given ID.
By default, the deployed version of the Tool is returned. Use the query parameters
version_id
orenvironment
to target a specific version of the Tool.
-
-
-
await client.tools.get("tl_789ghi");
-
-
-
id:
string
β Unique identifier for Tool.
-
request:
Humanloop.GetToolsIdGetRequest
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.delete(id) -> void
-
-
-
Delete the Tool with the given ID.
-
-
-
await client.tools.delete("tl_789ghi");
-
-
-
id:
string
β Unique identifier for Tool.
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.move(id, { ...params }) -> Humanloop.ToolResponse
-
-
-
Move the Tool to a different path or change the name.
-
-
-
await client.tools.move("tl_789ghi", { path: "new directory/new name", });
-
-
-
id:
string
β Unique identifier for Tool.
-
request:
Humanloop.UpdateToolRequest
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.listVersions(id, { ...params }) -> Humanloop.ListTools
-
-
-
Get a list of all the versions of a Tool.
-
-
-
await client.tools.listVersions("tl_789ghi", { status: "committed", });
-
-
-
id:
string
β Unique identifier for the Tool.
-
request:
Humanloop.ListVersionsToolsIdVersionsGetRequest
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.commit(id, versionId, { ...params }) -> Humanloop.ToolResponse
-
-
-
Commit a version of the Tool with a commit message.
If the version is already committed, an exception will be raised.
-
-
-
await client.tools.commit("tl_789ghi", "tv_012jkl", { commitMessage: "Initial commit", });
-
-
-
id:
string
β Unique identifier for Tool.
-
versionId:
string
β Unique identifier for the specific version of the Tool.
-
request:
Humanloop.CommitRequest
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.deleteToolVersion(id, versionId) -> void
-
-
-
Delete a version of the Tool.
-
-
-
await client.tools.deleteToolVersion("id", "version_id");
-
-
-
id:
string
β Unique identifier for Tool.
-
versionId:
string
β Unique identifier for the specific version of the Tool.
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.setDeployment(id, environmentId, { ...params }) -> Humanloop.ToolResponse
-
-
-
Deploy Tool to an Environment.
Set the deployed version for the specified Environment. This Prompt will be used for calls made to the Tool in this Environment.
-
-
-
await client.tools.setDeployment("tl_789ghi", "staging", { versionId: "tv_012jkl", });
-
-
-
id:
string
β Unique identifier for Tool.
-
environmentId:
string
β Unique identifier for the Environment to deploy the Version to.
-
request:
Humanloop.SetDeploymentToolsIdEnvironmentsEnvironmentIdPostRequest
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.removeDeployment(id, environmentId) -> void
-
-
-
Remove deployed Tool from the Environment.
Remove the deployed version for the specified Environment. This Tool will no longer be used for calls made to the Tool in this Environment.
-
-
-
await client.tools.removeDeployment("tl_789ghi", "staging");
-
-
-
id:
string
β Unique identifier for Tool.
-
environmentId:
string
β Unique identifier for the Environment to remove the deployment from.
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.listEnvironments(id) -> Humanloop.FileEnvironmentResponse[]
-
-
-
List all Environments and their deployed versions for the Tool.
-
-
-
await client.tools.listEnvironments("tl_789ghi");
-
-
-
id:
string
β Unique identifier for Tool.
-
requestOptions:
Tools.RequestOptions
-
-
client.tools.updateMonitoring(id, { ...params }) -> Humanloop.ToolResponse
-
-
-
Activate and deactivate Evaluators for monitoring the Tool.
An activated Evaluator will automatically be run on all new Logs within the Tool for monitoring purposes.
-
-
-
await client.tools.updateMonitoring("tl_789ghi", { activate: [ { evaluatorVersionId: "evv_1abc4308abd", }, ], });
-
-
-
id:
string
-
request:
Humanloop.EvaluatorActivationDeactivationRequest
-
requestOptions:
Tools.RequestOptions
-
-
client.datasets.list({ ...params }) -> core.Page
-
-
-
List all Datasets.
-
-
-
await client.datasets.list({ size: 1, });
-
-
-
request:
Humanloop.ListDatasetsGetRequest
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.upsert({ ...params }) -> Humanloop.DatasetResponse
-
-
-
Create a Dataset or update it with a new version if it already exists.
Datasets are identified by the
ID
or theirpath
. The datapoints determine the versions of the Dataset.By default, the new Dataset version will be set to the list of Datapoints provided in the request. You can also create a new version by adding or removing Datapoints from an existing version by specifying
action
asadd
orremove
respectively. In this case, you may specify theversion_id
orenvironment
query parameters to identify the existing version to base the new version on. If neither is provided, the default deployed version will be used.If you provide a commit message, then the new version will be committed; otherwise it will be uncommitted. If you try to commit an already committed version, an exception will be raised.
Humanloop also deduplicates Datapoints. If you try to add a Datapoint that already exists, it will be ignored. If you intentionally want to add a duplicate Datapoint, you can add a unique identifier to the Datapoint's inputs such as
{_dedupe_id: <unique ID>}
.
-
-
-
await client.datasets.upsert({ path: "test-questions", datapoints: [ { inputs: { question: "What is the capital of France?", }, target: { answer: "Paris", }, }, { inputs: { question: "Who wrote Hamlet?", }, target: { answer: "William Shakespeare", }, }, ], action: "set", commitMessage: "Add two new questions and answers", });
-
-
-
request:
Humanloop.DatasetRequest
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.get(id, { ...params }) -> Humanloop.DatasetResponse
-
-
-
Retrieve the Dataset with the given ID.
Unless
include_datapoints
is set totrue
, the response will not include the Datapoints. Use the List Datapoints endpoint (GET /{id}/datapoints
) to efficiently retrieve Datapoints for a large Dataset.By default, the deployed version of the Dataset is returned. Use the query parameters
version_id
orenvironment
to target a specific version of the Dataset.
-
-
-
await client.datasets.get("ds_b0baF1ca7652", { versionId: "dsv_6L78pqrdFi2xa", includeDatapoints: true, });
-
-
-
id:
string
β Unique identifier for Dataset.
-
request:
Humanloop.GetDatasetsIdGetRequest
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.delete(id) -> void
-
-
-
Delete the Dataset with the given ID.
-
-
-
await client.datasets.delete("id");
-
-
-
id:
string
β Unique identifier for Dataset.
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.move(id, { ...params }) -> Humanloop.DatasetResponse
-
-
-
Move the Dataset to a different path or change the name.
-
-
-
await client.datasets.move("id");
-
-
-
id:
string
β Unique identifier for Dataset.
-
request:
Humanloop.UpdateDatasetRequest
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.listDatapoints(id, { ...params }) -> core.Page
-
-
-
List all Datapoints for the Dataset with the given ID.
-
-
-
await client.datasets.listDatapoints("ds_b0baF1ca7652", { size: 1, });
-
-
-
id:
string
β Unique identifier for Dataset.
-
request:
Humanloop.ListDatapointsDatasetsIdDatapointsGetRequest
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.listVersions(id, { ...params }) -> Humanloop.ListDatasets
-
-
-
Get a list of the versions for a Dataset.
-
-
-
await client.datasets.listVersions("ds_b0baF1ca7652", { status: "committed", });
-
-
-
id:
string
β Unique identifier for Dataset.
-
request:
Humanloop.ListVersionsDatasetsIdVersionsGetRequest
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.commit(id, versionId, { ...params }) -> Humanloop.DatasetResponse
-
-
-
Commit a version of the Dataset with a commit message.
If the version is already committed, an exception will be raised.
-
-
-
await client.datasets.commit("ds_b0baF1ca7652", "dsv_6L78pqrdFi2xa", { commitMessage: "initial commit", });
-
-
-
id:
string
β Unique identifier for Dataset.
-
versionId:
string
β Unique identifier for the specific version of the Dataset.
-
request:
Humanloop.CommitRequest
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.deleteDatasetVersion(id, versionId) -> void
-
-
-
Delete a version of the Dataset.
-
-
-
await client.datasets.deleteDatasetVersion("id", "version_id");
-
-
-
id:
string
β Unique identifier for Dataset.
-
versionId:
string
β Unique identifier for the specific version of the Dataset.
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.uploadCsv(file, id, { ...params }) -> Humanloop.DatasetResponse
-
-
-
Add Datapoints from a CSV file to a Dataset.
This will create a new committed version of the Dataset with the Datapoints from the CSV file.
If either
version_id
orenvironment
is provided, the new version will be based on the specified version, with the Datapoints from the CSV file added to the existing Datapoints in the version. If neitherversion_id
norenvironment
is provided, the new version will be based on the version of the Dataset that is deployed to the default Environment.
-
-
-
await client.datasets.uploadCsv(fs.createReadStream("/path/to/your/file"), "id", { commitMessage: "commit_message", });
-
-
-
file:
File | fs.ReadStream | Blob
-
id:
string
-
request:
Humanloop.BodyUploadCsvDatasetsIdDatapointsCsvPost
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.setDeployment(id, environmentId, { ...params }) -> Humanloop.DatasetResponse
-
-
-
Deploy Dataset to Environment.
Set the deployed version for the specified Environment.
-
-
-
await client.datasets.setDeployment("ds_b0baF1ca7652", "staging", { versionId: "dsv_6L78pqrdFi2xa", });
-
-
-
id:
string
β Unique identifier for Dataset.
-
environmentId:
string
β Unique identifier for the Environment to deploy the Version to.
-
request:
Humanloop.SetDeploymentDatasetsIdEnvironmentsEnvironmentIdPostRequest
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.removeDeployment(id, environmentId) -> void
-
-
-
Remove deployed Dataset from Environment.
Remove the deployed version for the specified Environment.
-
-
-
await client.datasets.removeDeployment("ds_b0baF1ca7652", "staging");
-
-
-
id:
string
β Unique identifier for Dataset.
-
environmentId:
string
β Unique identifier for the Environment to remove the deployment from.
-
requestOptions:
Datasets.RequestOptions
-
-
client.datasets.listEnvironments(id) -> Humanloop.FileEnvironmentResponse[]
-
-
-
List all Environments and their deployed versions for the Dataset.
-
-
-
await client.datasets.listEnvironments("id");
-
-
-
id:
string
β Unique identifier for Dataset.
-
requestOptions:
Datasets.RequestOptions
-
-
client.evaluators.log({ ...params }) -> Humanloop.CreateEvaluatorLogResponse
-
-
-
Submit Evaluator judgment for an existing Log.
Creates a new Log. The evaluated Log will be set as the parent of the created Log.
-
-
-
await client.evaluators.log({ parentId: "parent_id", });
-
-
-
request:
Humanloop.CreateEvaluatorLogRequest
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.list({ ...params }) -> core.Page
-
-
-
Get a list of all Evaluators.
-
-
-
await client.evaluators.list({ size: 1, });
-
-
-
request:
Humanloop.ListEvaluatorsGetRequest
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.upsert({ ...params }) -> Humanloop.EvaluatorResponse
-
-
-
Create an Evaluator or update it with a new version if it already exists.
Evaluators are identified by the
ID
or theirpath
. The spec provided determines the version of the Evaluator.If you provide a commit message, then the new version will be committed; otherwise it will be uncommitted. If you try to commit an already committed version, an exception will be raised.
-
-
-
await client.evaluators.upsert({ path: "Shared Evaluators/Accuracy Evaluator", spec: { argumentsType: "target_required", returnType: "number", evaluatorType: "python", code: "def evaluate(answer, target):\n return 0.5", }, commitMessage: "Initial commit", });
-
-
-
request:
Humanloop.EvaluatorRequest
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.get(id, { ...params }) -> Humanloop.EvaluatorResponse
-
-
-
Retrieve the Evaluator with the given ID.
By default, the deployed version of the Evaluator is returned. Use the query parameters
version_id
orenvironment
to target a specific version of the Evaluator.
-
-
-
await client.evaluators.get("ev_890bcd");
-
-
-
id:
string
β Unique identifier for Evaluator.
-
request:
Humanloop.GetEvaluatorsIdGetRequest
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.delete(id) -> void
-
-
-
Delete the Evaluator with the given ID.
-
-
-
await client.evaluators.delete("ev_890bcd");
-
-
-
id:
string
β Unique identifier for Evaluator.
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.move(id, { ...params }) -> Humanloop.EvaluatorResponse
-
-
-
Move the Evaluator to a different path or change the name.
-
-
-
await client.evaluators.move("ev_890bcd", { path: "new directory/new name", });
-
-
-
id:
string
β Unique identifier for Evaluator.
-
request:
Humanloop.UpdateEvaluatorRequest
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.listVersions(id, { ...params }) -> Humanloop.ListEvaluators
-
-
-
Get a list of all the versions of an Evaluator.
-
-
-
await client.evaluators.listVersions("ev_890bcd");
-
-
-
id:
string
β Unique identifier for the Evaluator.
-
request:
Humanloop.ListVersionsEvaluatorsIdVersionsGetRequest
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.commit(id, versionId, { ...params }) -> Humanloop.EvaluatorResponse
-
-
-
Commit a version of the Evaluator with a commit message.
If the version is already committed, an exception will be raised.
-
-
-
await client.evaluators.commit("ev_890bcd", "evv_012def", { commitMessage: "Initial commit", });
-
-
-
id:
string
β Unique identifier for Prompt.
-
versionId:
string
β Unique identifier for the specific version of the Evaluator.
-
request:
Humanloop.CommitRequest
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.deleteEvaluatorVersion(id, versionId) -> void
-
-
-
Delete a version of the Evaluator.
-
-
-
await client.evaluators.deleteEvaluatorVersion("id", "version_id");
-
-
-
id:
string
β Unique identifier for Evaluator.
-
versionId:
string
β Unique identifier for the specific version of the Evaluator.
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.setDeployment(id, environmentId, { ...params }) -> Humanloop.EvaluatorResponse
-
-
-
Deploy Evaluator to an Environment.
Set the deployed version for the specified Environment. This Evaluator will be used for calls made to the Evaluator in this Environment.
-
-
-
await client.evaluators.setDeployment("ev_890bcd", "staging", { versionId: "evv_012def", });
-
-
-
id:
string
β Unique identifier for Evaluator.
-
environmentId:
string
β Unique identifier for the Environment to deploy the Version to.
-
request:
Humanloop.SetDeploymentEvaluatorsIdEnvironmentsEnvironmentIdPostRequest
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.removeDeployment(id, environmentId) -> void
-
-
-
Remove deployed Evaluator from the Environment.
Remove the deployed version for the specified Environment. This Evaluator will no longer be used for calls made to the Evaluator in this Environment.
-
-
-
await client.evaluators.removeDeployment("ev_890bcd", "staging");
-
-
-
id:
string
β Unique identifier for Evaluator.
-
environmentId:
string
β Unique identifier for the Environment to remove the deployment from.
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.listEnvironments(id) -> Humanloop.FileEnvironmentResponse[]
-
-
-
List all Environments and their deployed versions for the Evaluator.
-
-
-
await client.evaluators.listEnvironments("ev_890bcd");
-
-
-
id:
string
β Unique identifier for Evaluator.
-
requestOptions:
Evaluators.RequestOptions
-
-
client.evaluators.updateMonitoring(id, { ...params }) -> Humanloop.EvaluatorResponse
-
-
-
Activate and deactivate Evaluators for monitoring the Evaluator.
An activated Evaluator will automatically be run on all new Logs within the Evaluator for monitoring purposes.
-
-
-
await client.evaluators.updateMonitoring("id", {});
-
-
-
id:
string
-
request:
Humanloop.EvaluatorActivationDeactivationRequest
-
requestOptions:
Evaluators.RequestOptions
-
-
client.flows.log({ ...params }) -> Humanloop.CreateFlowLogResponse
-
-
-
Log to a Flow.
You can use query parameters
version_id
, orenvironment
, to target an existing version of the Flow. Otherwise, the default deployed version will be chosen.
-
-
-
await client.flows.log({ logId: "medqa_experiment_0001", id: "fl_6o701g4jmcanPVHxdqD0O", flow: { attributes: { prompt: { template: "You are a helpful assistant helping with medical anamnesis", model: "gpt-4o", temperature: 0.8, }, tool: { name: "retrieval_tool_v3", description: "Retrieval tool for MedQA.", source_code: "def retrieval_tool(question: str) -> str:\n pass\n", }, }, }, inputs: { question: "Patient with a history of diabetes and hypertension presents with chest pain and shortness of breath.", }, output: "The patient is likely experiencing a myocardial infarction. Immediate medical attention is required.", traceStatus: "incomplete", startTime: "2024-07-08T22:40:35", endTime: "2024-07-08T22:40:39", });
-
-
-
request:
Humanloop.FlowLogRequest
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.get(id, { ...params }) -> Humanloop.FlowResponse
-
-
-
Retrieve the Flow with the given ID.
By default, the deployed version of the Flow is returned. Use the query parameters
version_id
orenvironment
to target a specific version of the Flow.
-
-
-
await client.flows.get("fl_6o701g4jmcanPVHxdqD0O");
-
-
-
id:
string
β Unique identifier for Flow.
-
request:
Humanloop.GetFlowsIdGetRequest
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.delete(id) -> void
-
-
-
Delete the Flow with the given ID.
-
-
-
await client.flows.delete("fl_6o701g4jmcanPVHxdqD0O");
-
-
-
id:
string
β Unique identifier for Flow.
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.move(id, { ...params }) -> Humanloop.FlowResponse
-
-
-
Move the Flow to a different path or change the name.
-
-
-
await client.flows.move("fl_6o701g4jmcanPVHxdqD0O", { path: "new directory/new name", });
-
-
-
id:
string
β Unique identifier for Flow.
-
request:
Humanloop.UpdateFlowRequest
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.list({ ...params }) -> core.Page
-
-
-
Get a list of Flows.
-
-
-
await client.flows.list({ size: 1, });
-
-
-
request:
Humanloop.ListFlowsGetRequest
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.upsert({ ...params }) -> Humanloop.FlowResponse
-
-
-
Create or update a Flow.
Flows can also be identified by the
ID
or theirpath
.If you provide a commit message, then the new version will be committed; otherwise it will be uncommitted. If you try to commit an already committed version, an exception will be raised.
-
-
-
await client.flows.upsert({ path: "Personal Projects/MedQA Flow", attributes: { prompt: { template: "You are a helpful medical assistant helping with medical anamnesis. Answer {{question}}", model: "gpt-4o", temperature: 0.8, }, tool: { name: "retrieval_tool_v3", description: "Retrieval tool for MedQA.", source_code: "def retrieval_tool(question: str) -> str:\n pass\n", }, commit_message: "Initial commit", }, });
-
-
-
request:
Humanloop.FlowRequest
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.updateLog(logId, { ...params }) -> Humanloop.FlowLogResponse
-
-
-
Update the status, inputs, output of a Flow Log.
Marking a Flow Log as complete will trigger any monitoring Evaluators to run. Inputs and output (or error) must be provided in order to mark it as complete.
The end_time log attribute will be set to match the time the log is marked as complete.
-
-
-
await client.flows.updateLog("medqa_experiment_0001", { inputs: { question: "Patient with a history of diabetes and normal tension presents with chest pain and shortness of breath.", }, output: "The patient is likely experiencing a myocardial infarction. Immediate medical attention is required.", traceStatus: "complete", });
-
-
-
logId:
string
β Unique identifier of the Flow Log.
-
request:
Humanloop.UpdateTraceRequest
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.listVersions(id, { ...params }) -> Humanloop.ListFlows
-
-
-
Get a list of all the versions of a Flow.
-
-
-
await client.flows.listVersions("fl_6o701g4jmcanPVHxdqD0O", { status: "committed", });
-
-
-
id:
string
β Unique identifier for Flow.
-
request:
Humanloop.ListVersionsFlowsIdVersionsGetRequest
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.commit(id, versionId, { ...params }) -> Humanloop.FlowResponse
-
-
-
Commit a version of the Flow with a commit message.
If the version is already committed, an exception will be raised.
-
-
-
await client.flows.commit("fl_6o701g4jmcanPVHxdqD0O", "flv_6o701g4jmcanPVHxdqD0O", { commitMessage: "RAG lookup tool bug fixing", });
-
-
-
id:
string
β Unique identifier for Flow.
-
versionId:
string
β Unique identifier for the specific version of the Flow.
-
request:
Humanloop.CommitRequest
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.deleteFlowVersion(id, versionId) -> void
-
-
-
Delete a version of the Flow.
-
-
-
await client.flows.deleteFlowVersion("id", "version_id");
-
-
-
id:
string
β Unique identifier for Flow.
-
versionId:
string
β Unique identifier for the specific version of the Flow.
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.setDeployment(id, environmentId, { ...params }) -> Humanloop.FlowResponse
-
-
-
Deploy Flow to an Environment.
Set the deployed version for the specified Environment. This Flow will be used for calls made to the Flow in this Environment.
-
-
-
await client.flows.setDeployment("fl_6o701g4jmcanPVHxdqD0O", "staging", { versionId: "flv_6o701g4jmcanPVHxdqD0O", });
-
-
-
id:
string
β Unique identifier for Flow.
-
environmentId:
string
β Unique identifier for the Environment to deploy the Version to.
-
request:
Humanloop.SetDeploymentFlowsIdEnvironmentsEnvironmentIdPostRequest
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.removeDeployment(id, environmentId) -> void
-
-
-
Remove deployed Flow from the Environment.
Remove the deployed version for the specified Environment. This Flow will no longer be used for calls made to the Flow in this Environment.
-
-
-
await client.flows.removeDeployment("fl_6o701g4jmcanPVHxdqD0O", "staging");
-
-
-
id:
string
β Unique identifier for Flow.
-
environmentId:
string
β Unique identifier for the Environment to remove the deployment from.
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.listEnvironments(id) -> Humanloop.FileEnvironmentResponse[]
-
-
-
List all Environments and their deployed versions for the Flow.
-
-
-
await client.flows.listEnvironments("fl_6o701g4jmcanPVHxdqD0O");
-
-
-
id:
string
β Unique identifier for Flow.
-
requestOptions:
Flows.RequestOptions
-
-
client.flows.updateMonitoring(id, { ...params }) -> Humanloop.FlowResponse
-
-
-
Activate and deactivate Evaluators for monitoring the Flow.
An activated Evaluator will automatically be run on all new "completed" Logs within the Flow for monitoring purposes.
-
-
-
await client.flows.updateMonitoring("fl_6o701g4jmcanPVHxdqD0O", { activate: [ { evaluatorVersionId: "evv_1abc4308abd", }, ], });
-
-
-
id:
string
-
request:
Humanloop.EvaluatorActivationDeactivationRequest
-
requestOptions:
Flows.RequestOptions
-
-
client.directories.list() -> Humanloop.DirectoryResponse[]
-
-
-
Retrieve a list of all Directories.
-
-
-
await client.directories.list();
-
-
-
requestOptions:
Directories.RequestOptions
-
-
client.directories.create({ ...params }) -> Humanloop.DirectoryResponse
-
-
-
Creates a Directory.
-
-
-
await client.directories.create();
-
-
-
request:
Humanloop.CreateDirectoryRequest
-
requestOptions:
Directories.RequestOptions
-
-
client.directories.get(id) -> Humanloop.DirectoryWithParentsAndChildrenResponse
-
-
-
Fetches a directory by ID.
-
-
-
await client.directories.get("id");
-
-
-
id:
string
β String ID of directory. Starts withdir_
.
-
requestOptions:
Directories.RequestOptions
-
-
client.directories.delete(id) -> void
-
-
-
Delete the Directory with the given ID.
The Directory must be empty (i.e. contain no Directories or Files).
-
-
-
await client.directories.delete("id");
-
-
-
id:
string
β Unique identifier for Directory. Starts withdir_
.
-
requestOptions:
Directories.RequestOptions
-
-
client.directories.update(id, { ...params }) -> Humanloop.DirectoryResponse
-
-
-
Update the Directory with the given ID.
-
-
-
await client.directories.update("id");
-
-
-
id:
string
β Unique identifier for Directory. Starts withdir_
.
-
request:
Humanloop.UpdateDirectoryRequest
-
requestOptions:
Directories.RequestOptions
-
-
client.files.list({ ...params }) -> Humanloop.PaginatedDataUnionPromptResponseToolResponseDatasetResponseEvaluatorResponseFlowResponse
-
-
-
Get a paginated list of files.
-
-
-
await client.files.list();
-
-
-
request:
Humanloop.FilesListRequest
-
requestOptions:
Files.RequestOptions
-
-
client.evaluations.list({ ...params }) -> core.Page
-
-
-
Retrieve a list of Evaluations for the specified File.
-
-
-
await client.evaluations.list({ fileId: "pr_30gco7dx6JDq4200GVOHa", size: 1, });
-
-
-
request:
Humanloop.ListEvaluationsGetRequest
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.create({ ...params }) -> Humanloop.EvaluationResponse
-
-
-
Create an Evaluation.
Create a new Evaluation by specifying the File to evaluate, and a name for the Evaluation. You can then add Runs to this Evaluation using the
POST /evaluations/{id}/runs
endpoint.
-
-
-
await client.evaluations.create({ evaluators: [ { versionId: "version_id", }, ], });
-
-
-
request:
Humanloop.CreateEvaluationRequest
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.addEvaluators(id, { ...params }) -> Humanloop.EvaluationResponse
-
-
-
Add Evaluators to an Evaluation.
The Evaluators will be run on the Logs generated for the Evaluation.
-
-
-
await client.evaluations.addEvaluators("id", { evaluators: [ { versionId: "version_id", }, ], });
-
-
-
id:
string
β Unique identifier for Evaluation.
-
request:
Humanloop.AddEvaluatorsRequest
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.removeEvaluator(id, evaluatorVersionId) -> Humanloop.EvaluationResponse
-
-
-
Remove an Evaluator from an Evaluation.
The Evaluator will no longer be run on the Logs in the Evaluation.
-
-
-
await client.evaluations.removeEvaluator("id", "evaluator_version_id");
-
-
-
id:
string
β Unique identifier for Evaluation.
-
evaluatorVersionId:
string
β Unique identifier for Evaluator Version.
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.get(id) -> Humanloop.EvaluationResponse
-
-
-
Get an Evaluation.
This includes the Evaluators associated with the Evaluation and metadata about the Evaluation, such as its name.
To get the Runs associated with the Evaluation, use the
GET /evaluations/{id}/runs
endpoint. To retrieve stats for the Evaluation, use theGET /evaluations/{id}/stats
endpoint.
-
-
-
await client.evaluations.get("ev_567yza");
-
-
-
id:
string
β Unique identifier for Evaluation.
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.delete(id) -> void
-
-
-
Delete an Evaluation.
The Runs and Evaluators in the Evaluation will not be deleted.
-
-
-
await client.evaluations.delete("ev_567yza");
-
-
-
id:
string
β Unique identifier for Evaluation.
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.listRunsForEvaluation(id) -> Humanloop.EvaluationRunsResponse
-
-
-
List all Runs for an Evaluation.
-
-
-
await client.evaluations.listRunsForEvaluation("id");
-
-
-
id:
string
β Unique identifier for Evaluation.
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.createRun(id, { ...params }) -> Humanloop.EvaluationRunResponse
-
-
-
Create an Evaluation Run.
Optionally specify the Dataset and version to be evaluated.
Humanloop will automatically start generating Logs and running Evaluators where
orchestrated=true
. If you are generating Logs yourself, you can setorchestrated=false
and then generate and submit the required Logs via the API.If
dataset
andversion
are provided, you can setuse_existing_logs=True
to reuse existing Logs, avoiding generating new Logs unnecessarily. Logs that are associated with the specified Version and havesource_datapoint_id
referencing a datapoint in the specified Dataset will be associated with the Run.To keep updated on the progress of the Run, you can poll the Run using the
GET /evaluations/{id}/runs
endpoint and check its status.
-
-
-
await client.evaluations.createRun("id");
-
-
-
id:
string
β Unique identifier for Evaluation.
-
request:
Humanloop.CreateRunRequest
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.addExistingRun(id, runId) -> unknown
-
-
-
Add an existing Run to the specified Evaluation.
This is useful if you want to compare the Runs in this Evaluation with an existing Run that exists within another Evaluation.
-
-
-
await client.evaluations.addExistingRun("id", "run_id");
-
-
-
id:
string
β Unique identifier for Evaluation.
-
runId:
string
β Unique identifier for Run.
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.removeRun(id, runId) -> void
-
-
-
Remove a Run from an Evaluation.
The Logs and Versions used in the Run will not be deleted. If this Run is used in any other Evaluations, it will still be available in those Evaluations.
-
-
-
await client.evaluations.removeRun("id", "run_id");
-
-
-
id:
string
β Unique identifier for Evaluation.
-
runId:
string
β Unique identifier for Run.
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.updateEvaluationRun(id, runId, { ...params }) -> Humanloop.EvaluationRunResponse
-
-
-
Update an Evaluation Run.
Specify
control=true
to use this Run as the control Run for the Evaluation. You can cancel a running/pending Run, or mark a Run that uses external or human Evaluators as completed.
-
-
-
await client.evaluations.updateEvaluationRun("id", "run_id");
-
-
-
id:
string
β Unique identifier for Evaluation.
-
runId:
string
β Unique identifier for Run.
-
request:
Humanloop.UpdateEvaluationRunRequest
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.addLogsToRun(id, runId, { ...params }) -> Humanloop.EvaluationRunResponse
-
-
-
Add the specified Logs to a Run.
-
-
-
await client.evaluations.addLogsToRun("id", "run_id", { logIds: ["log_ids"], });
-
-
-
id:
string
β Unique identifier for Evaluation.
-
runId:
string
β Unique identifier for Run.
-
request:
Humanloop.AddLogsToRunRequest
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.getStats(id) -> Humanloop.EvaluationStats
-
-
-
Get Evaluation Stats.
Retrieve aggregate stats for the specified Evaluation. This includes the number of generated Logs for each Run and the corresponding Evaluator statistics (such as the mean and percentiles).
-
-
-
await client.evaluations.getStats("id");
-
-
-
id:
string
β Unique identifier for Evaluation.
-
requestOptions:
Evaluations.RequestOptions
-
-
client.evaluations.getLogs(id, { ...params }) -> Humanloop.PaginatedDataEvaluationLogResponse
-
-
-
Get the Logs associated to a specific Evaluation.
This returns the Logs associated to all Runs within with the Evaluation.
-
-
-
await client.evaluations.getLogs("id");
-
-
-
id:
string
β String ID of evaluation. Starts withev_
orevr_
.
-
request:
Humanloop.GetLogsEvaluationsIdLogsGetRequest
-
requestOptions:
Evaluations.RequestOptions
-
-
client.logs.list({ ...params }) -> core.Page
-
-
-
List all Logs for the given filter criteria.
-
-
-
await client.logs.list({ fileId: "file_123abc", size: 1, });
-
-
-
request:
Humanloop.ListLogsGetRequest
-
requestOptions:
Logs.RequestOptions
-
-
client.logs.delete({ ...params }) -> void
-
-
-
Delete Logs with the given IDs.
-
-
-
await client.logs.delete({ id: "string", });
-
-
-
request:
Humanloop.LogsDeleteRequest
-
requestOptions:
Logs.RequestOptions
-
-
client.logs.get(id) -> Humanloop.LogResponse
-
-
-
Retrieve the Log with the given ID.
-
-
-
await client.logs.get("prv_Wu6zx1lAWJRqOyL8nWuZk");
-
-
-
id:
string
β Unique identifier for Log.
-
requestOptions:
Logs.RequestOptions
-
-