A Comprehensive Guide to Fine-Tuning Reasoning Models: Fine-Tuning DeepSeek-R1 on Medical CoT with DigitalOcean’s GPU Droplets

Introduction

Recent advances in Large Language Models (LLMs) have shown promise in systematic reasoning tasks, with open-source models like DeepSeek-R1 demonstrating impressive capabilities in breaking down complex problems into logical steps. By fine-tuning these reasoning-focused models for medical applications, we can create proof-of-concept AI assistants that could potentially support healthcare professionals in their clinical decision-making processes while maintaining transparent chains of reasoning. In this tutorial, we’ll explore how to leverage DigitalOcean’s GPU Droplets to fine-tune a distilled quantized version of DeepSeek-R1, transforming it into a specialized reasoning assistant that can help analyze patient cases, suggest potential diagnoses, and provide verified structured explanations for its recommendations.

Shoutout to this great DataCamp tutorial and the paper, HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs, for inspiring this tutorial.

Prerequisites

Knowledge of these prerequisites will be helpful with following along with this tutorial:

• Python and PyTorch
• Deep Learning fundamentals (ex: neural networks, hyperparameters, etc.)
• Experience working with Hugging Face models and the Transformers library

When should we use Fine-Tuning?

Fine-tuning adapts a pre-trained model’s existing knowledge to perform specific tasks by training it further on a curated dataset. Fine-tuning shines in scenarios where consistent formatting, specific tone requirements, or complex instruction following are needed, as it can optimize the model’s behaviour for these particular use cases. This approach typically requires fewer computational resources and less time than training a model from scratch. Before proceeding with fine-tuning, however, it is good practice for developers to first consider the advantages of alternatives such as prompt engineering, Retrieval Augmented Generation (RAG), and even training a model from scratch.

One can do combinations of different approaches such as fine-tuning and RAG. By combining fine-tuning to establish a robust baseline with RAG to handle dynamic updates, the system achieves both adaptability and efficiency without requiring constant re-training. It really all comes down to organizational resource constraints and desired performance.

Monitoring whether outputs deliver to the standards of the intended utility and iterating/pivoting if not is absolutely critical.

Once we know that fine-tuning is the approach we want to take, we need to assemble the necessary components.

What do we need to Fine-Tune a Model?

A pre-trained model

A pre-trained model is a neural network that has already been trained on a large general-purpose corpus of data. Hugging Face has a plethora of open-source models available for you to use.

In this tutorial, we will be using a very popular reasoning model, DeepSeek-R1. Reasoning models excel at intricate tasks like advanced problems in math or coding. We chose “unsloth/DeepSeek-R1-Distill-Llama-8B-bnb-4bit” because it is distilled and pre-quantized, making it a more memory efficient and cost-effective model to perform experiments with. We were especially curious about its potential for complex tasks such as medical analysis. Note that using them for simpler tasks such as summarization or translation would be overkill due to the tendency reasoning models have towards being computationally expensive and verbose.

Dataset

Hugging Face has a great selection of datasets. We will be using the Medical O1 Reasoning Dataset. This dataset was generated with GPT-4o by searching for solutions to verifiable medical problems and validating them through a medical verifier.

This dataset will be used to perform supervised fine-tuning (SFT), where models are trained on a dataset of instructions and responses. To minimize the difference between the generated answers and ground-truth responses, SFT adjusts the weights in the LLM.

GPUs

GPUs aren’t always necessary to fine-tune a model. However, using a GPU (or multiple GPUs) can speed up the process significantly, especially for larger models or datasets like the ones used in this tutorial. In this article, we will show you how you can make use of DigitalOcean GPU Droplets.

Tools and Frameworks

Before starting this tutorial, it is recommended to familiarize yourself with the following libraries and tools:

Unsloth

Unsloth is all about making LLM training faster, with a particular focus on fine-tuning. The FastLanguageModel class, part of the Unsloth library, provides a simplified abstraction for fine-tuning LLMs. This class can handle loading the trained model weights, preprocessing input text, and executing inference to generate outputs.

Transformer Reinforcement Learning (TRL)

The HuggingFace Library, TRL, is used to train transformer language models with Reinforcement Learning. This tutorial will utilize the SFTTrainer Class.

Transformers

Transformers is also a HuggingFace Library. We will be using the TrainingArguments class to specify our desired arguments in SFTTrainer.

Weights and Biases

The W&B platform will be used for experiment tracking. Specifically, loss curves will be monitored.

Part 2: Implementation

Step 1: Set up a GPU Droplet and Launch Jupyter Labs

Follow this tutorial, “Setting Up the GPU Droplet Environment for AI/ML Coding”, to set up a GPU Droplet environment for our Jupyter Notebook.

Step 2: Install unsloth

%%capture
!pip install unsloth
!pip install --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git

Step 3: Configure Access Tokens

HuggingFace Tokens can be obtained from the Hugging Face Access Token page. Note that you may need to create Hugging Face account.

from huggingface_hub import login 
hf_token = "Replace with your actual token"
login(hf_token) 

Similarly, you will also need a Weights & Biases account to get a token for this step.

import wandb
wb_token = "Replace with your actual token"

wandb.login(key=wb_token)
run = wandb.init(
    project='Medical Assistant', 
    job_type="training", 
    anonymous="allow"
)

from unsloth import FastLanguageModel

Step 4: Loading the model and tokenizer

max_seq_length = 2048

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/DeepSeek-R1-Distill-Llama-8B-bnb-4bit",
    max_seq_length = max_seq_length,
    load_in_4bit = True,
    dtype = None, #if using H100, will automatically default to bfloat16
    token = hf_token,
)

Step 5: Testing Model Outputs Before Fine-Tuning

Creating a System Prompt

It is good practice to verify whether model outputs match your standards for format, quality, accuracy, etc. to assess if fine-tuning is necessary. Since we are interested in reasoning, we will formulate a system prompt that elicits a chain of thought.

Instead of writing the prompt directly in our input, let’s start by writing up a prompt template that incorporates place holders.

In this prompt template, we will specify precisely what we are looking for.

prompt_template= """### Role:
You are a medical expert specializing in clinical reasoning, diagnostics, and treatment planning. Your responses should:
- Be evidence-based and clinically relevant
- Include differential diagnoses when appropriate
- Consider patient safety and standard of care
- Note any important limitations or uncertainties

### Question:
{question}

### Thinking Process:
{thinking}

### Clinical Assessment:
{response}

"""

Notice the {thinking} placeholder. The primary goal of this step is to instruct the LLM to explicitly articulate its reasoning process before providing the final answer. This is often what is referred to as "chain-of-thought prompting”.

Inference with our System Prompt (Before Fine-tuning)

Here, we format the question using the structured prompt (prompt_template) to ensure the model follows a logical reasoning process. We will tokenize the input, return them as PyTorch tensors, and move it to the GPU (cuda) for faster inference.

FastLanguageModel.forinference(model) #model defined in step 4
inputs = tokenizer([prompt_template.format(question, "")], return_tensors="pt").to("cuda")

After, we will generate a response using the model, specifying key parameters like max_new_tokens=1200 (limits response length).

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    use_cache=True,
)

To obtain the final readable answer, we will decode the output tokens back into text.

response = tokenizer.batch_decode(outputs)
print(response[0].split("### Response:")[1])

Feel free to experiment with different prompt formulations and see how they affect your outputs.

Step 6: Load the Dataset

The dataset, FreedomIntelligence/medical-o1-reasoning-SFT, that we’re using has three columns: Question, Complex_CoT, and Response.

We will create a function (formatting_prompts_func) to format the input prompts in the dataset.

def formatting_prompts_func(examples):
    inputs = examples["Question"]
    cots = examples["Complex_CoT"]
    outputs = examples["Response"]
    texts = []
    for input, cot, output in zip(inputs, cots, outputs):
        text = prompt_template.format(input, cot, output) + tokenizer.eos_token
        texts.append(text)
    return {
        "text": texts,
    }

from datasets import load_dataset
dataset = load_dataset("FreedomIntelligence/medical-o1-reasoning-SFT","en", split = "train[0:500]",trust_remote_code=True)
dataset = dataset.map(formatting_prompts_func, batched = True,)
dataset["text"][0]

Step 7: Prepare the Model for Parameter Efficient Fine-Tuning (PEFT)

Instead of updating all the parameters of the model during fine-tuning, PEFT methods typically only modify a small subset of parameters, resulting in savings in computational power and time.

Here is an overview of some of the parameters and arguments we will be using the .get_peft_model method of Unsloth’s FastLanguageModel class.

model = FastLanguageModel.get_peft_model(
    model,
    r=16,  
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
    lora_alpha=16,
    lora_dropout=0,  
    bias="none",  
    use_gradient_checkpointing="unsloth",  # True or "unsloth" for very long context
    random_state=522,
    use_rslora=False
    )

Now that we’ve evaluated model outputs, it is time to use our SFT dataset to fine-tune the pre-trained model.

Step 8: Model Training with SFTTrainer

Supervised Fine-tuning Trainer is a class to develop supervised fine-tuned models from TRL.
We will also be using

from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported

Training Arguments

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=10,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=522,
        output_dir="outputs", #saving in Response column
    ),
)

This command will start the training process.

trainer_stats = trainer.train()

Step 9: Monitoring Experiments

Experiment tracking can be done with Weights and Biases. Essentially, we want to ensure that the training loss decreases over time to ensure model performance is improving with fine-tuning.

If model performance is degrading, it may be worth experimenting with the hyperparameter values.

Step 10: Model Inference After Fine-Tuning

question = "A 58-year-old woman reports a 3-year history of urine leakage when laughing, exercising, or lifting heavy objects. She denies any nighttime incontinence or feelings of urgency. On physical exam, she demonstrates urine loss with Valsalva maneuver, and a Q-tip test shows hypermobility of the urethrovesical junction with a 45-degree excursion. What would urodynamic testing most likely show regarding her post-void residual volume and detrusor muscle activity?"


FastLanguageModel.for_inference(model)  
inputs = tokenizer([prompt_template.format(question, "")], return_tensors="pt").to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("### Response:")[1])

Step 11: Saving the Model Locally

new_model_local = "DeepSeek-R1-Medical-COT"
model.save_pretrained(new_model_local) 
tokenizer.save_pretrained(new_model_local)

model.save_pretrained_merged(new_model_local, tokenizer, save_method = "merged_16bit",)

Step 12: Pushing the Model to HuggingFace Hub

If it is desirable to make the model accessible and beneficial to the wider AI community, we can publish the adopter, tokenizer, and model on to the Hugging Face Hub. This will allow others to easily integrate our model into their own projects and systems.

new_model_online = "HuggingFaceUSERNAME/DeepSeek-R1-Medical-COT"
model.push_to_hub(new_model_online)
tokenizer.push_to_hub(new_model_online)

model.push_to_hub_merged(new_model_online, tokenizer, save_method = "merged_16bit")

Conclusion

Fine-tuning is how smart teams transform those pre-trained models into precise, targeted tools that solve real problems. Here, we’re not reinventing the wheel, but rather aligning these wheels so that they take us where we want to go. While pre-trained models are powerful, they can be generic with outputs that may lack the structure and substance characteristic of professional-grade work.

We hope that through this tutorial, you gained an intuition around when to use and fine-tune reasoning models as well as some inspiration to better refine this technology for your use-case.

References and Additional Resources

Fine-Tuning DeepSeek R1 (Reasoning Model) | DataCamp HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs Train your own R1 reasoning model locally (GRPO) Unslothai Llama3.1_(8B)-GRPO.ipynb Fine-Tuning Your Own Llama 3 Model