Tutorial

AI Summarization: Vision Instruct with HuggingFace on Droplets

Published on March 31, 2025

AI Summarization: Vision Instruct with HuggingFace on Droplets

Introduction

DigitalOcean has recently introduced the innovative Vision Instruct models in partnership with Hugging Face. This collaboration enables developers to effortlessly integrate advanced multi-modal AI capabilities into their projects. Vision Instruct models excel at processing both visual data and textual instructions, simplifying the integration of multi-modal AI into various applications. To further support these capabilities, DigitalOcean offers GPU Droplets specifically designed for Vision Instruct deployments via 1-click Models. This results in a streamlined and efficient environment for the rapid development and scaling of AI applications.

This tutorial is designed for developers, data scientists, and anyone interested in leveraging AI to automate tasks and improve workflows. You will learn how to apply Vision Instruct models, hosted remotely using Hugging Face’s InferenceClient, to generate concise presentation notes directly from your slides.

What are Vision Instruct Models and Who are They For?

Vision Instruct models are a type of AI model that can process both visual data and textual instructions. They are designed to simplify the integration of multi-modal AI capabilities into various applications, making them an ideal solution for developers, data scientists, and anyone looking to leverage AI to automate tasks and improve workflows. These models are particularly useful for tasks that require the analysis of visual data, such as images or videos, in conjunction with textual instructions or context.

Vision Instruct models are suitable for a wide range of applications, including but not limited to:

Image and video analysis
Text-to-image synthesis
Image captioning and description
Visual question answering
Multimodal chatbots and virtual assistants

What You’ll Learn

Convert a PDF presentation into individual slide images using ImageMagick, an open-source software suite for editing and manipulating digital images.
Use Python to interact with a Hugging Face Vision Instruct model hosted on DigitalOcean to generate concise, context-aware summaries for each slide automatically.
Streamline your workflow and enhance your presentation quality by ensuring your talking points align perfectly with your visual aids.

Prerequisites

Before we dive in, make sure you have:

A Linux or Mac-based Developer’s Laptop. Windows Users should use a VM or Cloud Instance.
Python 3.10 or newer installed. We recommended Using a miniconda or venv virtual environment.
Install prerequisite library: pip install huggingface_hub.
Install ImageMagick for converting PDFs to images.
An example slide deck in PDF format. Example: Crack the AI Black Box: Practical Techniques for Explainable AI [S74147]

Automation - An Awesome Use Case

Manually creating slide summaries can be tiresome, especially when you have a lot of content to review. Vision Instruct models streamline this process by quickly interpreting slide images and your presentation abstract, reducing manual labor and boosting efficiency. This makes it an ideal solution for busy educators, professionals, and anyone who wants to maintain high-quality presentations without the extra effort.

Beyond slide summarization, Vision Instruct can be used in various scenarios. Imagine generating descriptive alt-text for accessibility, automating content tagging for digital libraries, or even creating quick previews for image-heavy reports. Its flexibility means you can extend this technology to many parts of your workflow, ensuring that complex visual data is easily digestible.

By using Vision Instruct, you’re simplifying a repetitive task and laying the groundwork for more integrated, AI-driven processes across your projects. The ability to adapt these models to different data types and tasks opens up a world of possibilities for future innovations.

Step 1 - Deploying the Vision Instruct Model on DigitalOcean

Deploying the Vision Instruct model on DigitalOcean can be done in a few steps. This was covered in the joint announcement in this Introducing Llama Vision-Instruct Models with DigitalOcean 1-Click GPU Droplets article, but the TLDR version is:

Create a GPU Droplet.
Select the Vision Instruct model.
Done!

Deploy the Vision Instruct Model

Step 2 - Converting Slides to Images

If you haven’t selected a presentation or slide deck to generate notes for, a sample presentation used for this NVIDIA GTC session, Crack the AI Black Box: Practical Techniques for Explainable AI [S74147], can be downloaded here.

Use ImageMagick to convert your PDF slide deck to PNG images. Execute the following command in your terminal:

magick your_presentation.pdf slide_%03d.png

This command outputs images like slide_001.png, slide_002.png, and so on. You will then need to move these images into a subfolder called slides_images in this project’s working directory.

Upload the Folder to Spaces Bucket

Finally, upload the entire folder to a DigitalOcean Spaces bucket. Make sure you set the folder’s permissions to Public. This step is crucial so your Python application can access the images via direct URLs in the next step.

Step 3 - Generating Summaries with Vision Instruct

Now, utilize the provided Python script to interact with your DigitalOcean-hosted Vision Instruct model, generating summaries based on the uploaded images and your abstract. If you are using the example NVIDIA GTC session provided in the prerequisites section, you can use the following session abstract:

Artificial Intelligence often operates in ways that are challenging to interpret, creating a gap in trust and transparency. Explainable AI (XAI) bridges this gap by providing strategies to demystify complex models, enabling stakeholders to understand how decisions are made. We'll explore foundational XAI concepts and offer practical methods to bring interpretability into developing and deploying AI systems, ensuring better decision-making and accountability. You'll learn actionable techniques for explaining AI behavior, from feature attributions and decision-path analyses to scenario-based insights. Through a live demonstration, you'll see how to apply these methods to real-world problems, enabling you to effectively diagnose, debug, and optimize your models. In the end, you'll have a clear roadmap for integrating XAI practices into your workflows to build trust and confidence in AI-powered solutions.

Replace the following placeholders in the Python script below:

IP Address of your 1-click Model/Droplet.
Your Bearer Token, which you can access by logging into your Droplet.
Your FQDN for the Spaces Bucket you uploaded your folder.
The Session Abstract for your Slides.

Here’s the complete Python script to automate this:

#!/usr/bin/env python3

import os
from huggingface_hub import InferenceClient

# ------------------------------------------------------------------------------
# Configuration
# ------------------------------------------------------------------------------
# Change this to the IP address/URL where your inference server is running
BASE_URL = "http://<REPLACE WITH YOUR 1-CLICK MODEL IP>/v1"

# Provide your token via the environment variable BEARER_TOKEN, or hardcode here
API_KEY = "<REPLACE WITH YOUR BEARER_TOKEN>" # os.getenv("BEARER_TOKEN")

# Directory containing your local slides, but assume they're *already* uploaded somewhere
# so we won't directly read from here. Instead, we just build a URL for each slide name.
IMAGES_DIR = "./slides_images"

# Example URL prefix where your images are hosted
# In practice, you might dynamically generate or retrieve these URLs
IMAGE_URL_PREFIX = "<YOUR UNIQUE DIGITALOCEAN BUCKET NAME>/slides_images"

# Abstract text describing the overall presentation
ABSTRACT_TEXT = (
   "<REPLACE WITH A SESSION ABSTRACT FOR YOUR SLIDES>"
)

# Initialize the inference client
client = InferenceClient(
   base_url=BASE_URL,
   api_key=API_KEY
)

# ------------------------------------------------------------------------------
# Helper Function
# ------------------------------------------------------------------------------
def generate_slide_summary(slide_file: str, slide_number: int, abstract_text: str) -> str:
   """
   Sends the abstract text and an image URL to the InferenceClient's chat endpoint.
   The returned string is the summary generated by the remote model.
   """

   # Construct the final URL to the hosted slide image
   # e.g., https://my-image-bucket.example.com/slides/slide_1.png
   slide_url = f"{IMAGE_URL_PREFIX}/{slide_file}"

   # Build the chat messages. Instead of base64 data, we pass a hosted URL.
   messages = [
       {
           "role": "user",
           "content": [
               {
                   "type": "text",
                   "text": f"Presentation Abstract: {abstract_text}"
               }
           ],
       },
       {
           "role": "user",
           "content": [
               {
                   "type": "image_url",
                   "image_url": {
                       "url": slide_url
                   }
               },
               {
                   "type": "text",
                   "text": (
                       f"Slide number {slide_number}. "
                       "Please summarize this slide based on the context of the abstract."
                   ),
               }
           ],
       }
   ]

   # Request a completion from the inference endpoint
   response = client.chat.completions.create(
       messages=messages,
       temperature=0.7,
       top_p=0.95,
       max_tokens=150,
   )

   # Extract the model's reply
   return response["choices"][0]["message"]["content"]

# ------------------------------------------------------------------------------
# Main Routine
# ------------------------------------------------------------------------------
def main():
   # Look for any PNG slides in the local directory, but assume they're all uploaded
   # to your hosting location. The local directory listing is just so we can parse
   # the filenames and build URLs.
   slide_images = sorted(
       [f for f in os.listdir(IMAGES_DIR) if f.lower().endswith(".png")]
   )

   if not slide_images:
       print("No slide images found in the specified directory.")
       return

   # For each slide, generate a summary
   for idx, slide_file in enumerate(slide_images, start=1):
       print(f"\n--- Generating summary for {slide_file} ---")
       slide_summary = generate_slide_summary(slide_file, idx, ABSTRACT_TEXT)
       print(f"Summary:\n{slide_summary}")

if __name__ == "__main__":
   main()

Once you execute the script, you should get session notes on a slide-per-slide basis which describe what each slide is talking about in the context of the presentation. You will find a sample output below.

--- Generating summary for slide_004.png ---
Summary:
Slide 5 illustrates flawed data, which is a key challenge in AI/ML. The slide features two main bullet points:
1. AI/ML only as good as the data: This highlights the importance of data quality in AI/ML models, emphasizing that the accuracy and reliability of AI/ML outputs directly depend on the quality of the input data.
2. Real-world examples of flawed data: This section lists real-world examples of flawed data, including:
- Recruiter AI + male-skewed: This example illustrates how biased data can lead to discriminatory outcomes, such as in recruitment AI systems that favor male candidates due to biased data.
- Offensive AI Chatbot: This example highlights how flawed data can result in offensive or inappropriate responses from AI chat

--- Generating summary for slide_005.png ---
Summary:
This slide introduces the topic of Explainable AI (XAI) by highlighting the importance of trust, transparency, debugging, improvement, compliance, and ethics. It emphasizes that XAI helps bridge the gap between AI decision-making processes and stakeholders' understanding of those processes. The slide also lists key goals of XAI, which include interpretability, accountability, and fairness, as well as bias detection. 
The slide serves as an introduction to the course, setting the stage for the practical methods and techniques that will be covered in the subsequent slides. By emphasizing the importance of trust, transparency, and accountability in AI decision-making, the slide establishes the context for the course's focus on XAI and its applications.

FAQs

1. What is the purpose of Vision Instruct models?

Vision Instruct models are a type of AI model specifically designed to handle multi-modal tasks, which involve processing and integrating both visual and textual data. Their primary purpose is to enable the generation of summaries, descriptions, or captions from images, as well as other tasks that require the fusion of visual and linguistic information. This capability allows Vision Instruct models to excel in applications such as image captioning, visual question answering, and image-text retrieval, making them a powerful tool for a wide range of AI applications.

2. How do I convert a PDF presentation into individual slide images?

To convert a PDF presentation into individual slide images, you can utilize ImageMagick, an open-source software suite specifically designed for editing and manipulating digital images. ImageMagick offers a wide range of tools and features that enable the conversion of PDF files into various image formats, including PNG, JPEG, and GIF. For more information on how to use ImageMagick for PDF conversion, refer to the ImageMagick documentation.

3. What is the role of Hugging Face’s `InferenceClient` in this tutorial?

Hugging Face’s InferenceClient plays a crucial role in this tutorial by facilitating the interaction with the Vision Instruct model hosted remotely. This client enables the automatic generation of concise, context-aware summaries for each slide. By using the InferenceClient, you can seamlessly integrate the Vision Instruct model into your workflow, allowing for the efficient creation of summaries that accurately capture the essence of each slide. This integration is particularly useful for large presentations, as it saves time and effort that would be required to manually summarize each slide.

4. How can I ensure my talking points align perfectly with my visual aids?

To ensure that your talking points align perfectly with your visual aids, it’s crucial to have a clear understanding of the content and message you want to convey through each slide. Vision Instruct models can significantly aid in this process by generating concise and context-aware summaries for each slide. These summaries can serve as a guide for crafting your talking points, ensuring that they are relevant, accurate, and effectively complement your visual aids. By leveraging Vision Instruct models in this way, you can streamline your workflow, enhance your presentation quality, and deliver a more cohesive and engaging message to your audience.

5. Can I use Vision Instruct models for other applications beyond slide summarization?

Yes, Vision Instruct models can be applied to a wide range of applications beyond slide summarization. Their multi-modal capabilities make them suitable for tasks such as image captioning, visual question answering, image-text retrieval, and more. Additionally, Vision Instruct models can be used for generating descriptive alt-text for accessibility, automating content tagging for digital libraries, or even creating quick previews for image-heavy reports. The flexibility of these models allows you to extend this technology to many parts of your workflow, ensuring that complex visual data is easily digestible.

6. How do I deploy a Vision Instruct model on DigitalOcean?

Deploying a Vision Instruct model on DigitalOcean is a straightforward process. First, create a GPU Droplet on DigitalOcean. Then, select the Vision Instruct model from the DigitalOcean Marketplace. This will automatically set up the necessary environment for your Vision Instruct model, allowing you to start using it for your AI applications.

7. What are the benefits of using Vision Instruct models for slide summarization?

The benefits of using Vision Instruct models for slide summarization are numerous. They can significantly reduce the manual effort required to create summaries, saving time and increasing productivity. Additionally, Vision Instruct models can generate summaries that are more accurate and context-aware than manual summaries, ensuring that the essence of each slide is captured effectively. This technology also enables the creation of summaries for large presentations, making it an ideal solution for busy educators, professionals, and anyone who wants to maintain high-quality presentations without the extra effort.

Conclusion

Vision Instruct models reduce the manual effort in creating slide summaries and, more importantly, offer a deeper impact on your overall AI processes. By seamlessly integrating with platforms like DigitalOcean, these models are now more accessible and available than ever. Their ability to handle both text and image inputs opens up new opportunities for real-time data processing and content generation.

This technology marks a significant step forward for anyone working in AI-driven fields. With Vision Instruct, you have a robust tool that enhances clarity, boosts efficiency, and drives smarter decisions. Dive into this technology and explore how it can transform your workflows and elevate your presentations.