By JetForMe
I’m working on an app to streamline sharing assets online (images, video, etc.). Right now, the user has to have their own S3 account.
But I’m thinking about offering a version that handles the service aspect, too, essentially re-selling my Spaces storage. Is there any kind of built-in support for determining how much storage and transfer a particular user is using across a subset of objects? I hope to have thousands of users, so I can’t really create a bucket for each one.
I was thinking a subdirectory for each user would let me keep their content separate from others, making it easier to measure. Is there a single call to get the recursive size of a directory, and/or bytes transferred to/from it?
Or maybe there’s a way to tag each object they upload with an identifier, and then I can query Spaces based on that tag? Like, return all keys with tag “12345”? I want to be able to impose quotas on my users, and hide or delete objects if they stop paying.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Accepted Answer
Hey @jetforme,
The DigitalOcean Spaces offering does not provide built-in support for tracking storage and transfer statistics at the granularity of subdirectories or based on specific tags for objects. Spaces primarily provides usage metrics at the bucket level, which includes overall storage size. However, implementing a user-level tracking system for storage and bandwidth usage within a single Space requires a more customized approach.
The best thing to do to get your voice heard regarding this would be to head over to our Product Ideas board and post a new idea, including as much information as possible for what you’d like to see implemented.
To track storage usage per user (assuming a user’s data is stored in a specific subdirectory), you would need to periodically list all objects under each user’s prefix (subdirectory) and calculate the total size. This can be automated using a script or a backend service that utilizes the DigitalOcean Spaces API or any S3-compatible SDK. For example:
import boto3
s3 = boto3.client('s3', region_name='nyc3', endpoint_url='https://nyc3.digitaloceanspaces.com',
aws_access_key_id='YOUR_ACCESS_KEY', aws_secret_access_key='YOUR_SECRET_KEY')
def calculate_storage_usage(bucket_name, prefix):
total_size = 0
paginator = s3.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket=bucket_name, Prefix=prefix):
for obj in page.get('Contents', []):
total_size += obj['Size']
return total_size
Tracking bytes transferred is more complex because Spaces does not directly provide metrics for this. One approach is to use a CDN in front of Spaces, like Cloudflare, which can provide a more detailed log of individual requests and the amount of data transferred:
https://www.digitalocean.com/community/questions/digitalocean-spaces-with-cloudflare
Hope that helps!
- Bobby.
Heya, @jetforme
I will also vote for scripting this using boto3 it is the official AWS SDK tool for Python and the most commonly used tool for interacting with bucket services, there are alternative libraries available for working with AWS services in Python.
You can check our article on how to use AWS SDKs:
https://docs.digitalocean.com/products/spaces/how-to/use-aws-sdks/
Regards
Become a contributor for community
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
DigitalOcean Documentation
Full documentation for every DigitalOcean product.
Resources for startups and SMBs
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
The developer cloud
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Get started for free
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.