models/sam-2/

[giscus[bot]]

models/sam-2/

Discover SAM 2, the next generation of Meta's Segment Anything Model, supporting real-time promptable segmentation in both images and videos with state-of-the-art performance. Learn about its key features, datasets, and how to use it.

https://docs.ultralytics.com/models/sam-2/

[glenn-jocher]

Hey everyone! We wanted to give you a quick update: our team is currently busy integrating SAM 2 (Segment Anything Model 2) into the Ultralytics package. This new model is super exciting, with features like real-time performance and the ability to segment objects it’s never seen before.

We're really pumped about what SAM2 can do, but it's still in the works. Thanks for your patience as we get everything set up! We'll keep you posted as we make progress. Stay tuned!

[glenn-jocher]

Update: SAM 2 is now fully integrated! Update your package to ultralytics>=8.2.70 with pip install -U ultralytics and then for example:

from ultralytics import ASSETS, SAM

# Load a model
model = SAM("sam2_s.pt")

# Segment with point prompt
results = model(ASSETS / "bus.jpg", points=[150, 150], labels=[1], device="cpu")

[sctrueew]

Could you please provide an example of how to perform inference on a video? I would appreciate it if you could include details on the necessary steps and code snippets.

Thank you!

[glenn-jocher]

@sctrueew to perform inference on a video using Ultralytics, you can use the predict mode. Here's a concise example to get you started:

from ultralytics import YOLO

# Load a pretrained YOLOv8 model
model = YOLO("yolov8n.pt")

# Run inference on a video file
results = model("path/to/video.mp4")

# Display results
for result in results:
    result.show()

This code will load the model, run inference on the video, and display the results. For more details, please refer to the Ultralytics documentation.

[VikeIngenior]

Is it usable with ultralytics package at the moment or should we wait until the implementation is done?

[VikeIngenior]

Hello Glenn, I am having problem with using my fine-tuned YOLOv8 model with the SAM2 model. I think my detections with the frame I get from cv2.VideoCapture doesn't match with the frame I get from SAM2 model.

Below, I explained what I'm trying to do and what did I get. I am open to any other approaches for my purpose.

I have imported SAM2 by cloning its github repo to my Google Drive and just imported it on Colab. I am trying to make a "video segmentation". My aim is to keep track of the object even if the camera angle changes for a couple seconds or even minutes. The reason I use YOLO is, it tracks when the object is detected. By using my YOLO model, I get the mask coordinates of my spesific object. And then by adding/substracting a few pixels by the corners of the mask, I give the prompt points to the SAM2 model. I got a result with that block of code, but it gave prompts for every frame, so it was not kind of what I expected:

i = 0
previous_detection = False
with sv.VideoSink(TARGET_VIDEO.as_posix(), video_info=video_info) as sink:
for frame_idx, frame_path in enumerate(frame_paths):
        frame = cv2.imread(frame_path)
        print(previous_detection)
        # Run YOLOv8 detection
        results = model(frame)
        bbox = None
        min_xy_point = None
        max_x_min_y_point = None
        min_x_max_y_point = None
        max_xy_point = None
        for result in results:
          if result.masks:
              mask_coordinates = result.masks.xy[0]  # Use the first mask
              # Compute bounding box coordinates from mask
              x_min, y_min = np.min(mask_coordinates[:, :2], axis=0)
              x_max, y_max = np.max(mask_coordinates[:, :2], axis=0)
              coordinates = np.copy(mask_coordinates)
              min_xy_point = coordinates[np.lexsort((coordinates[:, 1], coordinates[:, 0]))[0]]
              min_x_max_y_point = coordinates[np.lexsort((-coordinates[:, 1], coordinates[:, 0]))[0]]
              max_x_min_y_point = coordinates[np.lexsort((coordinates[:, 1], -coordinates[:, 0]))[0]]
              max_xy_point = coordinates[np.lexsort((-coordinates[:, 1], -coordinates[:, 0]))[0]]
              print("min_xy: ",min_xy_point)
              print("max_x_min_y: ",max_x_min_y_point)
          if result.boxes:
              bbox = result.boxes.xyxy[0].cpu().numpy()

        if bbox is not None:
          detections = bbox  # Assuming detections for the first batch
          if not previous_detection:
            print(detections)
            print(detections[0])
            if len(detections) > 0:
                points = []
                #for detection in detections:
                x1, y1, x2, y2 = detections[0], detections[1], detections[2], detections[3]

                # Calculate the slope and direction
                _, direction = find_slope((min_xy_point[0], min_xy_point[1]), (max_x_min_y_point[0], max_x_min_y_point[1]))
                print(direction)
                if direction == "decreasing":
                  p1 = min_xy_point[0]+5
                  p2 = min_xy_point[1]+5
                  p3 = max_xy_point[0]-5
                  p4 = max_xy_point[1]-5
                else:
                  p1 = min_x_max_y_point[0]+5
                  p2 = min_x_max_y_point[1]-5
                  p3 = max_x_min_y_point[0]-5
                  p4 = max_x_min_y_point[1]+5
                #else:
                  #p1 = x1+5
                  #p2 = y1+5
                  #p3 = x2-5
                  #p4 = y2-5

                points.append([p1, p2]) 
                points.append([p3, p4])

                yolo_mask_points = np.array([[int(min_xy_point[0]), int(min_xy_point[1])], [int(max_x_min_y_point[0]), int(max_x_min_y_point[1])], [int(max_xy_point[0]), int(max_xy_point[1])], [int(min_x_max_y_point[0]), int(min_x_max_y_point[1])]])

                cv2.polylines(frame, [yolo_mask_points], isClosed=True, color=(255, 0, 0), thickness=2)
 
                points = np.array(points, dtype=np.float32)
                labels = np.ones(len(points))
                sam2_model.reset_state(inference_state)
                _, object_ids, mask_logits = sam2_model.add_new_points(
                    inference_state=inference_state,
                    frame_idx=frame_idx,
                    obj_id=1,  # You can assign unique IDs based on detection
                    points=points,
                    labels=labels,
                )

                masks = (mask_logits[0] > 0.0).cpu().numpy()
                detections = sv.Detections(
                    xyxy=sv.mask_to_xyxy(masks=masks),
                    mask=masks.astype(bool)
                )

                annotated_frame = mask_annotator.annotate(scene=frame.copy(), detections=detections)
                sink.write_frame(annotated_frame)
            else:
                # If no objects detected, write the original frame
                sink.write_frame(frame)
            previous_detection = True
          else:
            # If no objects detected, write the original frame and reset previous detection flag
            sink.write_frame(frame)
            previous_detection = False

        if i > 10:
          break
        i += 1
        print(f"{i}th frame")

[VikeIngenior]

I know it's a long code, but I thought it would be necessary for you to understand what I mean. Here, I don't use propagate_in_video method. So this is no different than image segmentation. In order to use propagate_in_video method, I should first assign some points. Here is what I did:

• First, I simply get the first frame by _, frame=cap.read() method.
• Then I got mask coordinates from that frame with this code results = model(frame) and so on (I don't write the whole code to get the coordinates).
• After that, I assigned the first points with that block of code:

points = []
p1 = min_xy_point[0]+5
p2 = min_xy_point[1]+5
p3 = max_xy_point[0]-5
p4 = max_xy_point[1]-5
points.append([p1, p2]) #Left Prompt Point
points.append([p3, p4]) #Right Prompt Point
print(points)

_, object_ids, mask_logits = sam2_model.add_new_points(
    inference_state=inference_state,
    frame_idx=0,
    obj_id=None,
    points=points,
    labels=labels,
)

Finally, here is what I did to keep track of the object I detected:

with sv.VideoSink(TARGET_VIDEO.as_posix(), video_info=video_info) as sink:
    for frame_idx, object_ids, mask_logits in sam2_model.propagate_in_video(inference_state):
        frame_path = frame_paths[frame_idx]
        frame = cv2.imread(frame_path)

        cv2.circle(frame, (int(max_xy_point[0]), int(max_xy_point[1])), 5, (0, 255, 0), -1)
        cv2.polylines(frame, [yolo_bbox_points], isClosed=True, color=(0, 255, 0), thickness=2)
        cv2.polylines(frame, [yolo_mask_points], isClosed=True, color=(255, 0, 0), thickness=2)

        masks = (mask_logits[0] > 0.0).cpu().numpy()
        detections = sv.Detections(
            xyxy=sv.mask_to_xyxy(masks=masks),
            mask=masks.astype(bool)
        )

        annotated_frame = mask_annotator.annotate(scene=frame.copy(), detections=detections)
        sink.write_frame(annotated_frame)

[glenn-jocher]

Thank you for sharing the detailed explanation. It seems like the issue might be related to the synchronization between the frames processed by YOLOv8 and SAM 2. Ensure that the frames fed into SAM 2 match exactly with those processed by YOLOv8. You might also want to verify that the coordinate transformations between YOLOv8 detections and SAM 2 prompts are accurate. If the issue persists, please try updating to the latest versions of both packages to see if it resolves the problem. For further details, you can refer to the SAM 2 documentation.

[VikeIngenior]

Thank you for your answer 🖐

[glenn-jocher]

@VikeIngenior you're welcome! It sounds like you're making good progress. To ensure your frames from cv2.VideoCapture match those processed by SAM 2, verify the synchronization and coordinate transformations. Also, make sure you're using the latest versions of both packages. If the issue persists, please provide more details or specific errors you're encountering. For further guidance, you can refer to the SAM 2 documentation.

[acerolakirsche]

Hey dear people,
i try to find information about the way to work with SAM 2 with my Apple Silicon. Is there a way?
while trying to install it seems to require CUDA :/

[glenn-jocher]

You can run this code on Apple devices with M1/2/3 chips and everything works correctly, though not all SAM or SAM2 operations are supported on MPS, so if you try to use device="mps" you will get an error. You can post this to the torch repo for them to prioritize MPS support for these ops.

from ultralytics import ASSETS, SAM

# Load a model
model = SAM("sam2_s.pt")

# Segment with point prompt
results = model(ASSETS / "bus.jpg", points=[150, 150], labels=[1], device="cpu")

[esssyjr]

How can you fine-tune this model for your specific needs?
Also, does it support receiving annotations?
What of labels?

[glenn-jocher]

@esssyjr to fine-tune SAM 2 for your specific needs, you can use its promptable interface to segment objects by providing points, bounding boxes, or masks. While SAM 2 currently supports inference only, you can use these prompts to iteratively refine segmentation results. For auto-annotation, SAM 2 can generate segmentation datasets quickly by leveraging pre-trained models. For more details, please visit SAM 2 documentation.

[mariswarycharan]

when will training option enable for SAM 2

[glenn-jocher]

@mariswarycharan currently, SAM 2 supports inference but not training. We are continuously working on enhancing the model's capabilities, and training support might be included in future updates. For the latest updates and detailed information, please visit the SAM 2 documentation. Thank you for your interest!

[chauhtnguyen]

Does this support text prompts? For example, I want to search for buildings or cars from an aerial image or video

[glenn-jocher]

@chauhtnguyen sAM 2 does not support text prompts directly. Instead, it uses visual prompts like points, bounding boxes, or masks to specify objects of interest. For your use case of identifying buildings or cars in aerial images or videos, you can provide bounding boxes or points as prompts to guide the segmentation. For more details on how to use these prompts, you can refer to the SAM 2 documentation.

[nihirv]

You can use the autodistill package to use SAM2 for text prompt based inference

[glenn-jocher]

@nihirv sAM 2 does not support text prompts directly. It uses visual prompts like points, bounding boxes, or masks for segmentation tasks. For more details, you can refer to the SAM 2 documentation.

[nihirv]

@glenn-jocher is this a gpt response? 😂

You can do text prompt based segmentation by first running a zero-shot object detection model over the image with your text prompt. Then you can use the center point of the bounding box as a visual prompt to SAM.

Autodistill does it here in an easy to use package: https://docs.autodistill.com/base_models/grounded-sam-2/

[glenn-jocher]

SAM 2 does not support text prompts directly. However, you can achieve text-prompt-based segmentation by first running a zero-shot object detection model with your text prompt and then using the center point of the bounding box as a visual prompt for SAM 2. For more details, you can refer to the Autodistill documentation.

[moehmeni]

Is there an API to fine tune the model on our own labeled photos? also the schema and shape for the inputs

[glenn-jocher]

@moehmeni yes, you can fine-tune the model on your own labeled photos using the Ultralytics API. For SAM 2, you can use the auto_annotate function to create segmentation datasets. For YOLO models, you can train on custom datasets by specifying the dataset path and model configuration. Ensure your dataset follows the required schema and shape for inputs. For more details on SAM 2, please refer to the SAM 2 documentation.

[YiftyD]

Thank you very much for your beautiful work! I have a question regarding prompts with several points. When I prompt your integration of SAM 2 with several points I get several masks - one mask for each prompt point (which is what I want to get), while when I do the same thing using SAM 2 directly, I get one mask. Are there certain parameters that need to be used while initializing sam2 or while doing the prediction?
Thank you in advance

[pderrenger]

@YiftyD thank you for your kind words! The behavior you're experiencing with multiple points resulting in multiple masks is expected with the Ultralytics integration of SAM 2. This is designed to provide more granular control over segmentation. When using SAM 2 directly, it might default to generating a single mask for all points. To achieve similar behavior with SAM 2 directly, you may need to adjust the prompt parameters or segmentation settings. For more detailed guidance, please refer to the SAM 2 documentation.

[YiftyD]

Thank you very much for your answer

[pderrenger]

You're welcome! To achieve multiple masks for several points using SAM 2 directly, ensure you are using the latest version of the package. If the issue persists, you might need to adjust the prompt parameters or segmentation settings. For further details, you can refer to the SAM 2 documentation. If you encounter any bugs, please verify if they are reproducible with the latest package versions.

[VikeIngenior]

How can I use this sample code in order to make a detection (with YOLOv8) and segment that detection through a video (more like a real-time video stream)? I have run the code and it worked and I got a txt file which includes labels but I couldn't know what to do next.

from ultralytics.data.annotator import auto_annotate
auto_annotate(data=frame, det_model=yolo_model, sam_model=sam_model)

[pderrenger]

@VikeIngenior to achieve real-time detection and segmentation in a video stream using YOLOv8 and SAM 2, you need to integrate the detection results with the segmentation model. After running the detection and obtaining the labels, you can use these labels to prompt the SAM 2 model for segmentation. Ensure you are using the latest versions of the packages. For detailed guidance on using SAM 2, refer to the SAM 2 documentation. If you need further assistance, feel free to ask!

[Kavi91]

Hello !

I am trying to test SAM2, but its says "sam2_b.pt is not a supported SAM model. Available models are:
dict_keys(['sam_h.pt', 'sam_l.pt', 'sam_b.pt', 'mobile_sam.pt'])
"

from ultralytics import SAM

Load a model

model = SAM("/backend/model/sam2_b.pt")

Display model information (optional)

model.info()

Run inference

model("Input_videos/video.mp4")

[glenn-jocher]

@zjq1839 if your version is already above 8.2.70 and you're encountering this issue, please ensure that the model path and name are correct. The supported models are sam_h.pt, sam_l.pt, sam_b.pt, and mobile_sam.pt. Double-check the model file name and try again. If the issue persists, please provide more details about your setup. For further information, you can refer to our SAM 2 documentation.

[Kavi91]

Hi Glen, Now its working. and thank you. but when I use the same code to inference with a video it doesn't show the segmentation. can you check what is wrong in this code :

import cv2
from ultralytics import SAM

# Load the SAM model
model = SAM("Aerial_Vision_Analysis_Tool_v1.0/backend/model/sam2_b.pt")

# Display model information (optional)
model.info()

# Input video path
input_video_path = "Input_videos/A normal day Chennai traffic on top angle.mp4"
output_video_path = "output_videos/processed_video.mp4"

# Run inference in streaming mode
results = model(input_video_path, stream=True)

# Define the codec and create VideoWriter object to save output video
cap = cv2.VideoCapture(input_video_path)
fps = int(cap.get(cv2.CAP_PROP_FPS))
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter(output_video_path, fourcc, fps, (width, height))

# Process the streaming results
for result in results:
    # Assuming result contains the processed frame
    processed_frame = result.orig_img  # Access the original frame after processing

    # Display the processed frame
    cv2.imshow('Inference', processed_frame)

    # Write the frame to the output video file
    out.write(processed_frame)

    # Break the loop on 'q' key press
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# Release everything when done
out.release()
cap.release()
cv2.destroyAllWindows()

[glenn-jocher]

@Kavi91 hi Glen, glad to hear it's working now! For the video segmentation issue, ensure that result.orig_img contains the processed frame with segmentation. If it doesn't, you might need to access the segmentation masks directly from the result object and overlay them on the original frame before displaying or saving. For more details, refer to the SAM 2 documentation.

[Kavi91]

Hello Glenn, I used default real time inference code but it gives output like below. but I couldn't visualize the result. can you give me a example code to see the visual inference

video 1/1 (frame 1/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 315.1ms
video 1/1 (frame 2/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1655.6ms
video 1/1 (frame 3/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1595.9ms
video 1/1 (frame 4/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1606.8ms
video 1/1 (frame 5/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1608.5ms
video 1/1 (frame 6/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1608.6ms
video 1/1 (frame 7/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1606.0ms
video 1/1 (frame 8/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 3066.4ms
video 1/1 (frame 9/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1612.2ms
video 1/1 (frame 10/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1614.8ms
video 1/1 (frame 11/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1615.5ms

[glenn-jocher]

To visualize the segmentation results, ensure you are accessing the correct processed frames. Instead of result.orig_img, you should use result.plot() to overlay the segmentation masks on the original frames. This should display the segmented output correctly. If you need further assistance, please refer to the SAM 2 documentation.

[ckn3]

Hey there,

I was wondering how can I set foreground and background points and then visualize the segmentation properly? Here is my current code, but it gives each point's segmentation a distinct color.

from ultralytics import SAM
# Load a model
model = SAM("sam2_b.pt")

# Display model information (optional)
model.info()

# Segment with point prompt
results = model("frame_50.jpg", points=[[960,530], [1340, 980], [1880,1170], [1160,1230],[880,520]], labels=[1,1,1,0,0])

for result in results:
    masks = result.masks  # Masks object for segmentation masks outputs
    result.save(filename="seg/result.jpg")  

[glenn-jocher]

Hi @sctrueew,

Thanks for reaching out. To improve performance, ensure you're using the latest package version and that your GPU is properly utilized by PyTorch. You can also try reducing the input image size or batch size. If the issue persists, please provide more details about your environment and setup.

[RunsenXu]

Same, here.

The inference speed is very low, with 2.2s per image for SAM-t.

[glenn-jocher]

Hi,

To improve inference speed, ensure you're using the latest package version and that your GPU is properly utilized by PyTorch. You can also try reducing the input image size or batch size. If the issue persists, please provide more details about your environment and setup.

[divineSix]

Hi, Glenn.

I am running experiments on a RTX 3090 and have a similar issue to what is being reported here. When I monitor the GPU using nvidia-smi for a memory footprint, I see the large model taking up around 6GB of VRAM. The inference speeds are over 3 seconds per frame, and I am unsure on how to speed this up. Can you point me to resources that help solve this problem?

[glenn-jocher]

To improve inference speed, ensure your GPU is fully utilized by checking CUDA installation and PyTorch compatibility. You might also try reducing the input size or batch size. For more optimization tips, refer to the SAM 2 documentation.

[CA4GitHub]

The web page shows the example below, but what options do I have for saving the results using ultralytics built in methods?
For example, is there a way to run the example below and save the segmented results as a video?
If not a video, what's the simplest way to save the segmented results (no bounding box overlays or other stuff) for each frame in the video?

from ultralytics import SAM

model = SAM("sam2_b.pt")

model.info()

model("path/to/video.mp4")

[glenn-jocher]

result.plot(boxes=False) should omit the boxes.

I think if you don't pass any additional arguments the SAM models will by default segment the entire image.

[esssyjr]

I used the above code you provide, it took more than an hour processing but still didn't work. I have to cancel it.
What do you think the issue might be?

[glenn-jocher]

@esssyjr it sounds like the process might be getting stuck due to the complexity or length of the video. Please ensure you are using the latest version of the package. Additionally, you can try running the model on a shorter video or a smaller segment to diagnose if the issue persists. If the problem continues, please provide more details about the video and system specifications.

[esssyjr]

Thanks for the response.

I tried two videos on my local machine which are 4 and 5 seconds in .MOV and .MP4 format, it didn’t work for more than an hour. But, when I used it on google colab it worked successfully. Is it showing that my processor is low for the task ?

[glenn-jocher]

It seems like the issue might be related to your local machine's processing power. Google Colab provides powerful GPUs which could explain why it worked there. You might want to try running the model on a machine with a more powerful GPU or consider using cloud-based solutions for heavy processing tasks.

[dvolgyes]

The SAM and SAM 2 documentation have an identical block called 'SAM comparison vs YOLOv8'.
I think that is misleading. One key selling point of SAM2 its close to real time speed, so repeating the results of SAM 1 makes no sense. It would be very nice to add SAM2 too, or SAM2 only, but there is nothing helpful having the SAM1 results repeated in SAM2 docs.

My wish is having updated results in the section, but if that is significant work, then rather remove it. If an update is performed, then enumerating all the SAM2 variants would be nice.

[glenn-jocher]

@dvolgyes both SAM and SAM2 speeds are nearly identical and both extremely slow compared to what you might be used to with YOLOv8

[iholy19]

Hello Glenn,
I tested sam2-l.pt. It works perfectly. I noticed that the size of model downloaded from ultralytics is 429MB but the model released by Meta, sam_hiera_large.pt is 857MB. I am wondering if it is the same model. Please help me clarify

[glenn-jocher]

@iholy19 hello, the SAM 2 model you downloaded from Ultralytics is optimized for size and performance, which is why it is smaller than Meta's original release. Despite the size difference, it retains the core functionalities and capabilities. For more details, you can refer to the SAM 2 documentation.

[thedatamask]

Breaking News: 🚨 SAM 1.0 outperforms SAM 2.0 on low-contrast images like CT Scan and MRI images!
Check out this insightful video from Data Mask YouTube channel where they dive deep into the architecture, demos, and limitations of both Meta's segmentation models. Discover when to use SAM 1.0 for superior results on challenging images.

Use case with ultralytics

https://www.youtube.com/watch?v=vMI-TnyNLYU

Please do like and subscribe for exciting video like above .

#SAM #SAM2.0 #MetaAI #ImageSegmentation #AI #MachineLearning #DeepLearning #DataMask #BreakingNews

[glenn-jocher]

@thedatamask thank you for sharing the video. For insights on SAM 2's capabilities and its comparison with other models, you can explore our documentation. If you have specific questions about using SAM 2 with Ultralytics, feel free to ask!

[thedatamask]

Breaking News: 🚨 SAM 1.0 outperforms SAM 2.0 on low-contrast images like CT Scan and MRI images!
Check out this insightful video from Data Mask YouTube channel where they dive deep into the architecture, demos, and limitations of both Meta's segmentation models. Discover when to use SAM 1.0 for superior results on challenging images.

Use case with ultralytics

https://www.youtube.com/watch?v=vMI-TnyNLYU

Please do like and subscribe for exciting video like above .

#SAM #SAM2.0 #MetaAI #ImageSegmentation #AI #MachineLearning #DeepLearning #DataMask #BreakingNews

[glenn-jocher]

@thedatamask thank you for sharing the video. For more detailed information on SAM 2 and its capabilities, please visit our documentation at https://docs.ultralytics.com/models/sam-2/.

[chickencoins]

I understand that SAM2 allows include and exclude operations for multiple points. Is this feature available in ultralytics? If so, could you provide a simple example code?

Additionally, I am wondering if the object tracking functionality can be applied to live camera streaming as well. I don't just want image segmentation, but I need to track a specified object in real-time "streaming" video.

[glenn-jocher]

@chickencoins yes, SAM 2 supports include and exclude operations for multiple points. Here's a simple example:

from ultralytics import SAM

model = SAM("sam2_b.pt")
results = model(points=[[[400, 370], [900, 370]]], labels=[[1, 0]])

For real-time object tracking in live streams, you can use YOLO models with the track mode. SAM 2 focuses on segmentation rather than tracking. For more details, visit the SAM 2 documentation.

[FabrizioDeFiore]

Hi, @glenn-jocher ,following up on this question, I wonder, is there any method for performing segmentation in real time from my camera livestream?
I'm currently only looking for the segmentation, but maybe using both Yolo for tracking and Sam2 for segmenting on the previously tracked object might be an option. I'm not sure, I'm making multiple attempts using using a simple cv2 VIdeoCapture on my camera and I can only make it working with Yolo, no luck with Sam2.
Thanks in advance for your response.

[glenn-jocher]

For real-time object tracking in live streams, you can use YOLO models with the track mode. SAM 2 focuses on segmentation rather than tracking. For more details, visit the SAM 2 documentation.

[FabrizioDeFiore]

Hi @glenn-jocher , sorry I might have explain it wrong, I'm only talking about running the segmentation with SAM2 from the camera livestream, the same way it works on videos, even without the help of YOLO.

[glenn-jocher]

To perform real-time segmentation from a camera livestream using SAM 2, you can use the VideoCapture feature from OpenCV to capture frames and then apply the SAM 2 model for segmentation. While SAM 2 is primarily designed for segmentation, combining it with YOLO for tracking and then segmenting the tracked objects is a viable approach. If you need further assistance, please refer to the SAM 2 documentation.

[spaceman-cb]

Hi Glenn
In the Meta SAM2 implementation/API, there is a function propagate_in_video to track masklets across frames. I don't see a function like this mentioned in the Ultralytics documentation; am I missing it somewhere? Does the Ultralytics API support masklet tracking across frames? If so, how?
Thanks!

[glenn-jocher]

@spaceman-cb sAM 2 in the Ultralytics API doesn't have a direct propagate_in_video function, but it uses a memory mechanism to handle object tracking across frames. For more details, you can refer to the SAM 2 documentation.

[kusa-dowsing]

@glenn-jocker is the address of the doc you mentioned https://docs.ultralytics.com/models/sam-2/ ? i cannot get any info about it

[glenn-jocher]

@kusa-dowsing yes, the correct documentation link is https://docs.ultralytics.com/models/sam-2/. It provides detailed information on SAM 2's features and usage. If you have further questions, feel free to ask!

[spaceman-cb]

@glenn-jocher
I don't see much amount the memory mechanism within the documentation. Could you please clarify the following?

• a) Is the memory mechanism implicitly used when running inference on videos? (i.e. is there no need to explicitly call something like propagate_in_video or set an argument to track segments across frames in a video)?

• b) What about when running inference on single images one by one? Is the memory mechanism enabled or disabled in this case?

• c) If the memory mechanism is enabled in the case of (b), how can I reset the state of the model when running inference on unrelated images?

Thanks!

[glenn-jocher]

a) The memory mechanism is implicitly used during video inference, so no need to call a specific function.
b) For single images, the memory mechanism is not active.
c) Since it's inactive for single images, there's no need to reset the state.

[hoho2312]

Hi, may I know if the current version of SAM2 support multi-point prompting for one object? Or it can only treat each point prompt as individual object?

[glenn-jocher]

@hoho2312 sAM 2 supports multi-point prompting for a single object, allowing you to refine segmentation results interactively. For more details, please refer to the SAM 2 documentation.

[kusa-dowsing]

first of all, thanks for the great job. I want to get the detail of the "label" of sam2. is it same as coco, or where can i find the whole label list.

[glenn-jocher]

@kusa-dowsing thank you for your interest! SAM 2 doesn't use a predefined label set like COCO. It supports zero-shot generalization, allowing segmentation of objects without specific labels. For more details, please refer to the SAM 2 documentation.

[RuellePaul]

Hello ! Is it planned to integrate EfficientSAM ? https://github.com/yformer/EfficientSAM
It seems better than MobileSAM

[glenn-jocher]

@RuellePaul thank you for your interest! Currently, there are no plans to integrate EfficientSAM. We appreciate your suggestion and will consider it for future updates. For now, you can explore our existing models like SAM 2 for advanced segmentation capabilities.

[renyechao]

If I need to use sam2 to segment the objects I want in an image, do I need to train it in yolo for my custom dataset?

[glenn-jocher]

@renyechao you don't need to train SAM 2 with YOLO for custom datasets. SAM 2 supports zero-shot generalization, allowing it to segment objects without prior training on specific categories. For more details, visit the SAM 2 documentation.

[Hangsiin]

Is there a good way to use SAM 2 as an object tracker while performing object detection with YOLO? I know YOLO has built-in tracking functionality, but I'm having issues because in the video I'm working with, when objects overlap or disappear and reappear, their IDs get changed. Thanks in advance!

[glenn-jocher]

@Hangsiin for tracking stability in overlapping or disappearing objects, consider using SAM 2's advanced memory mechanism to maintain consistent tracking. You can integrate SAM 2 with YOLO for enhanced segmentation and tracking. For more details, visit the SAM 2 documentation.

[iuiu34]

hi,
how do you specify the frame index in a video?

in the following example. I want to track point 900,370. s.t. point coordinates are for frame 34.
what's the argument for that?
thanks

from ultralytics import SAM

model = SAM("sam2.1_b.pt")

model("path/to/video.mp4", points=[900, 370], labels=[1],
frame_id=[34] # error, not defined
)

[glenn-jocher]

@iuiu34 to specify a frame index in a video with SAM, you need to handle frame extraction separately before passing the frame to the model. SAM doesn't directly support frame indexing within its API. You can use a library like OpenCV to extract the desired frame and then pass it to the model for processing.

[iuiu34]

hi @glenn-jocher ,
thanks for the quick response.

In [sam2|https://github.com/facebookresearch/sam2?tab=readme-ov-file] video_predictor_example there's the method add_new_points_or_box .
with args
inference_state,
frame_idx,
obj_id,
points,
labels,

i would expect same args, in ultralytics framework, no?

[glenn-jocher]

@iuiu34 hi, thanks for reaching out. In the Ultralytics framework, the method add_new_points_or_box isn't directly available. You might need to adapt your approach using the available SAM 2 functionalities. For more details, please refer to the SAM 2 documentation.

[iuiu34]

understood.

Any plans to add missing args? Any pr or issue open already?

Having sam2 in same framework as yolo is very nice.
But if it's missing features, it's kinda of a hard blocker.

[glenn-jocher]

Thank you for your feedback. Currently, there are no open issues or PRs for adding those specific arguments. We appreciate your input and encourage you to open a feature request on our GitHub to help us prioritize this enhancement.

[iuiu34]

feature request

#17185

[glenn-jocher]

Thank you for your feature request. Currently, there are no plans to add those specific arguments. We encourage you to open a feature request on GitHub to help us prioritize this enhancement. If you have further questions, feel free to ask!

[davidetedd9203]

Hi!
how can i segment specific objects in video or images?
for example:
i need to create volleyball ball dataset how can i do this?

[glenn-jocher]

@davidetedd9203 to segment specific objects like a volleyball in images or videos, you can use SAM 2's prompt-based interface to specify objects of interest. For creating a volleyball dataset, consider using bounding boxes or points as prompts to guide the segmentation. You can find more details on using SAM 2 for segmentation tasks in the documentation.

[enviroQL]

how to get the labels?
i'm getting

AttributeError: 'SAM2Model' object has no attribute 'names'

with model.model.names or

AttributeError: 'SAM' object has no attribute 'names'

with model.names

[glenn-jocher]

@enviroQL the SAM 2 model does not include a 'names' attribute as it is designed for segmentation tasks, which do not require class labels like detection models. For segmentation, focus on the mask outputs to identify segmented areas. If you have further questions, feel free to ask!

[erik945]

Hello!
The description of SAM says "It leverages a flexible prompt-based interface, enabling users to specify objects of interest through various prompt types, such as points, bounding boxes, or masks"

How can I set a mask (black and white image) as a prompt?

like
results = model("path/to/image.jpg", points=[[[400, 370], [900, 370]]], labels=[[1, 1]])
but
results = model("path/to/image.jpg", mask = "F:/image/test_mask.png")

Thank you.

[glenn-jocher]

@erik945 hello! Currently, SAM does not support using an entire mask image as a prompt directly. You can use other prompt types like points or bounding boxes to guide the segmentation. If you have further questions or need more help, feel free to ask!

[erik945]

I wrote a function that turns a mask into a 3000 point prompt. When I use it, SAM produces a very fragmented mask consisting of separate regions. In addition, it highlights nearby objects

`
sam_positive_points = 1000 #count of positive points (object)
sam_negative_points = 2000 #count of negative points (background)

cordinates = np.argwhere(tmp_mask > 0.5) #light area - positive
wc = []
lb = []
tmp = list(range(0, len(cordinates)))
random.shuffle(tmp) # select random points on light area

i = 0

while((i < sam_positive_points) and (i < len(tmp))):
wc.append([cordinates[tmp[i]][1], cordinates[tmp[i]][0]]) # PIL order - y,x , SAM - x,y
tmp_imgDemo[cordinates[tmp[i]][0], cordinates[tmp[i]][1], 1 ] = 1
lb.append(1)

  i = i + 1

cordinates = np.argwhere(tmp_mask < 0.5) #dark area - negative

tmp = list(range(0, len(cordinates)))
random.shuffle(tmp)

i = 0

while((i < sam_negative_points) and (i < len(tmp))):
wc.append([cordinates[tmp[i]][1], cordinates[tmp[i]][0]])
tmp_imgDemo[cordinates[tmp[i]][0], cordinates[tmp[i]][1], 0 ] = 1
lb.append(0)

  i = i + 1

save_image_array(tmp_imgDemo, "C:\2\charparse\_SAM_prompt.png") # test image with labels, green points - positive, red - negative

fpth = os.path.join(tmp_folder, "toSAMuniversal.png")
save_image_array(tmp_img, fpth)

resultsSAM = modelSAM(fpth, points=[wc], labels=[lb], stream = False)
`

If I use bbox, it creates a good, monolithic mask, but often misses the object, for example - I want a ball, but it segments the palm that holds it
P.S. Sorry, for some reason the code highlighting doesn't work.

[glenn-jocher]

It seems like creating a large number of random points might be causing segmentation inconsistencies. Consider refining your point selection criteria to focus more precisely on the object of interest. Additionally, using fewer but more strategically placed prompts might help improve the segmentation quality. Experimenting with SAM 2.1 might also offer improved results due to its enhanced accuracy.

[erik945]

I will experiment.
Thank you very much for the clarification!

[glenn-jocher]

You're welcome! If you have any more questions or need further assistance, feel free to ask. Good luck with your experiments!

[martinferianc]

Hello! SAM2 allows adding prompts at different frame indices, not just the first frame in the stream, how can this be done in SAM2VideoPredictor? More specifically, how to do this in the streaming version? Thank you!

[glenn-jocher]

@martinferianc thank you for your question! To add prompts at different frame indices in SAM2VideoPredictor, you can specify the points and labels parameters for each frame during inference. For streaming applications, ensure to manage prompt inputs dynamically based on the frame index. If you need further clarification, refer to the SAM 2 documentation for examples or share specific details about your setup.

[martinferianc]

Thank you @glenn-jocher for a quick reply. I have tried using https://github.com/ultralytics/ultralytics/blob/main/ultralytics/models/sam/predict.py#L976 before running the inference or during it but unfortunately it does not work. Reading also other comments, this feature, adding prompts at different frames except the first one, does not seem to be implemented in ultralytics. Could you please update the documentation stating that the features can only be added at the first frame or provide a working example? Thanks!

[glenn-jocher]

Thank you for your feedback! You are correct that adding prompts at frames other than the first is currently not implemented in SAM2VideoPredictor. We've noted this limitation and will update the documentation to reflect this for clarity. If you have further suggestions, feel free to share them here.