How do I call a long-running script on DigitalOcean App Platform?

Question

(I’m using BitBucket so I have to containerize everything, rather an using git repos and Procfiles.) I’ve made a Docker image that runs a simple HTTP server (“api-server”) and another that has a long-running script (“test-long-script”). The api container looks like it’s running fine, but how do I get the API server to call the long-script container? To run it from the commandline, I’d use this command: docker run --rm test-long-script:latest I want people to be able to make HTTP requests to my api-server web service, like POST /encode-video or GET /do-something-slow?foo=bar, and have the request handler send the POSTdata / query params to the slow script: docker run --rm test-long-script:latest --foo=bar I can’t find any documentation on how to use workers, or how to communicate between components, besides “you can use the component name” and something about internal ports. What am I missing?

Jon Friesen · Accepted Answer

There’s no way for any part of the app to directly send information to the worker, so the worker process should continuously poll some kind of job queue from a database, and process jobs one by one. Does that sound correct?

Yup, that sounds correct.

It sounds like I can’t accomplish this with just a web service component and a worker component; at minimum, I need to add a database as well? And then I need to wrap my long-running-script in a service that runs continuously, and wrap that with some kind of job queue package, like “Bull” for Node.js or “Celery” for Python. Am I on the right track?

I think you might be able to do this without a database, it might make it easier to use one though. It really depends on the work you’re doing.

The worker process has to be a long-running script. In the past I’ve done this by having my code call an endpoint on my service every 30 seconds to see if any new jobs have been requested. If a job exists, it will process it and send the results back to the service.

If you want to to introduce a database you could have a table that has “tasks”, your worker would check for new “tasks” and if a new one exists, set it to “in-progress” and start doing the work, once complete saving the results back in the database and marking it as complete.

As a possible future feature, would you prefer to be able to call out to the worker privately? If so, what kind of behavior would you expect?

Jon Friesen · Answer

👋 @fsduncan

Great question! Workers are not routable, meaning they cannot be accessed remotely, however, they can access services.

This means that your worker (eg. test-long-script) can reach out to a service within the network. For example, if your api component is called api-server you can make HTTP request to it at http://api-server. You can also create an environment variable that automatically injects the private URL to a local service like: API-SERVICE=${api-service.PRIVATE_URL}. Here’s more bind variable information.

Services also support internal ports within the app spec (this isn’t included in the UI yet). If your api-server component starts two servers, one can be mapped for public usage, and another can be used for internal traffic with your workers.

Another common pattern is to share a database between your service and workers, where jobs to be done are put in the database by the service and the worker picks them up when it checks for new work.

In conclusion, workers best work with a pattern of checking for work by calling an exposed (public or private) api endpoint from a service or checking a database for new work.

Here’s how internal ports are set in the app spec (though this app does not use them):

name: internal-ports
region: nyc
services:
- github:
    branch: master
    repo: digitalocean/sample-golang
  instance_count: 1
  instance_size_slug: basic-xxs
  internal_ports:
  - 8082
  - 4001
  name: web

Here’s more information on app specs

Hope this helps!

Question

How do I call a long-running script on DigitalOcean App Platform?

Become a contributor for community

DigitalOcean Documentation

Resources for startups and SMBs

Get our newsletter

The developer cloud

Get started for free