Question

Getting storage and transfer stats for a subdirectory?

Posted on February 15, 2024
DigitalOcean Spaces
Asked by JetForMe

I’m working on an app to streamline sharing assets online (images, video, etc.). Right now, the user has to have their own S3 account.

But I’m thinking about offering a version that handles the service aspect, too, essentially re-selling my Spaces storage. Is there any kind of built-in support for determining how much storage and transfer a particular user is using across a subset of objects? I hope to have thousands of users, so I can’t really create a bucket for each one.

I was thinking a subdirectory for each user would let me keep their content separate from others, making it easier to measure. Is there a single call to get the recursive size of a directory, and/or bytes transferred to/from it?

Or maybe there’s a way to tag each object they upload with an identifier, and then I can query Spaces based on that tag? Like, return all keys with tag “12345”? I want to be able to impose quotas on my users, and hide or delete objects if they stop paying.

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Bobby Iliev

Site Moderator

• February 17, 2024

Accepted Answer

Hey @jetforme,

The DigitalOcean Spaces offering does not provide built-in support for tracking storage and transfer statistics at the granularity of subdirectories or based on specific tags for objects. Spaces primarily provides usage metrics at the bucket level, which includes overall storage size. However, implementing a user-level tracking system for storage and bandwidth usage within a single Space requires a more customized approach.

The best thing to do to get your voice heard regarding this would be to head over to our Product Ideas board and post a new idea, including as much information as possible for what you’d like to see implemented.

https://ideas.digitalocean.com/

To track storage usage per user (assuming a user’s data is stored in a specific subdirectory), you would need to periodically list all objects under each user’s prefix (subdirectory) and calculate the total size. This can be automated using a script or a backend service that utilizes the DigitalOcean Spaces API or any S3-compatible SDK. For example:

import boto3

s3 = boto3.client('s3', region_name='nyc3', endpoint_url='https://nyc3.digitaloceanspaces.com',
                  aws_access_key_id='YOUR_ACCESS_KEY', aws_secret_access_key='YOUR_SECRET_KEY')

def calculate_storage_usage(bucket_name, prefix):
    total_size = 0
    paginator = s3.get_paginator('list_objects_v2')
    for page in paginator.paginate(Bucket=bucket_name, Prefix=prefix):
        for obj in page.get('Contents', []):
            total_size += obj['Size']
    return total_size

Tracking bytes transferred is more complex because Spaces does not directly provide metrics for this. One approach is to use a CDN in front of Spaces, like Cloudflare, which can provide a more detailed log of individual requests and the amount of data transferred:

https://www.digitalocean.com/community/questions/digitalocean-spaces-with-cloudflare

Hope that helps!

- Bobby.

alexdo

Site Moderator

• February 21, 2024

Heya, @jetforme

I will also vote for scripting this using boto3 it is the official AWS SDK tool for Python and the most commonly used tool for interacting with bucket services, there are alternative libraries available for working with AWS services in Python.

You can check our article on how to use AWS SDKs:

https://docs.digitalocean.com/products/spaces/how-to/use-aws-sdks/

Regards