Skip to content

Commit

Permalink
remove HF Token from run.sh command (#62)
Browse files Browse the repository at this point in the history
* HF token no longer needed in run.sh
  • Loading branch information
lauranutanix authored Jul 8, 2024
1 parent c7a3d4c commit 3e123d1
Showing 1 changed file with 2 additions and 3 deletions.
5 changes: 2 additions & 3 deletions docs/gpt-in-a-box/kubernetes/v0.2/inference_server.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Run the following command for starting Kubeflow serving and running inference on the given input:
```
bash $WORK_DIR/llm/run.sh -n <MODEL_NAME> -g <NUM_GPUS> -f <NFS_ADDRESS_WITH_SHARE_PATH> -m <NFS_LOCAL_MOUNT_LOCATION> -e <KUBE_DEPLOYMENT_NAME> [OPTIONAL -d <INPUT_PATH> -v <REPO_COMMIT_ID> -t <HUGGINGFACE_HUB_TOKEN>]
bash $WORK_DIR/llm/run.sh -n <MODEL_NAME> -g <NUM_GPUS> -f <NFS_ADDRESS_WITH_SHARE_PATH> -m <NFS_LOCAL_MOUNT_LOCATION> -e <KUBE_DEPLOYMENT_NAME> [OPTIONAL -d <INPUT_PATH> -v <REPO_COMMIT_ID>]
```

* **n**: Name of a [validated model](validated_models.md)
Expand All @@ -12,7 +12,6 @@ bash $WORK_DIR/llm/run.sh -n <MODEL_NAME> -g <NUM_GPUS> -f <NFS_ADDRESS_WITH_SH
* **m**: Mount path to your nfs server to be used in the kube PV where model files and model archive file be stored
* **e**: Desired name of the deployment metadata (will be created)
* **v**: Commit ID of model's HuggingFace repository (optional, if not provided default set in model_config will be used)
* **t**: Your HuggingFace token. Needed for LLAMA(2) model.

Should print "Inference Run Successful" as a message once the Inference Server has successfully started.

Expand All @@ -29,7 +28,7 @@ bash $WORK_DIR/llm/run.sh -n falcon_7b -d data/qa -g 1 -e llm-deploy -f '1.1.1.1
```
For 1 GPU Inference with official Llama2-7B model and keep inference server alive:
```
bash $WORK_DIR/llm/run.sh -n llama2_7b -d data/summarize -g 1 -e llm-deploy -f '1.1.1.1:/llm' -m /mnt/llm -t <HUGGINGFACE_HUB_TOKEN>
bash $WORK_DIR/llm/run.sh -n llama2_7b -d data/summarize -g 1 -e llm-deploy -f '1.1.1.1:/llm' -m /mnt/llm
```

### Cleanup Inference deployment
Expand Down

0 comments on commit 3e123d1

Please sign in to comment.