Workers, log drains and rollbacks, now on velixir

A web server is only the start of running an app in production. Sooner or later you need something to work through a queue, a job that runs at three in the morning, a way to hear about crashes before your customers do, logs in the tool your team already uses, and services that talk to each other without going out over the internet.

Over the last week velixir shipped all of that, one piece at a time. Here's what each one does and where to start.

Wednesday: workers, cron jobs and migrations

Your Procfile now runs as it is. Every line other than web becomes a worker, release runs once per deploy before any traffic moves, and scheduled jobs live in velixir.json. One deploy ships the web process, the workers and the jobs on the same build.

The release command is the part we're proudest of. Migrations run once, on the new build, before anything about the running app changes. If one fails, the deploy stops and the previous release keeps serving. Two replicas never race to apply the same migration.

Cron jobs bill only for the seconds they run, so a 20-second job every 15 minutes costs around €0.15 a month instead of a worker sitting idle all month. Workers and cron jobs are on paid plans, and the release command runs on every plan. Read the post or the docs.

Thursday: crash alerts and webhooks

When an app or worker crashes, the Overview now says which process, how often, and why in plain words, with the last lines it printed underneath. If it keeps crashing, the team's Owners and Admins get one email, and nothing more until it recovers.

Webhooks send a signed POST when something happens, whether that's a crash, a deploy going live or failing, a failed cron job or release command, an approval waiting, or a low balance. Point one at a Slack channel and events arrive as readable lines, or at your own code, where any Standard Webhooks library verifies the signature. Crash events are grouped so a bad night doesn't become two hundred messages, and their payloads never carry log lines, which can hold secrets.

Free on every plan. Read the post or the docs.

Friday: log drains

If your team already lives in Grafana or Datadog, a hosting dashboard with its own log viewer is one more tab to check at 2am. Log drains send every line your apps write to Grafana Cloud, Datadog, Axiom, Better Stack, any OpenTelemetry endpoint or your own HTTPS endpoint, a few seconds after it's written, tagged with the app, process and deployment.

Set one up for the whole team or a single app and press Test to check it before you rely on it. If your log tool goes down, your apps don't notice. Free on every plan. Read the post or the docs.

Saturday: private networking

Every app on your team now has a private address, such as http://<app-slug>.internal. Calls go straight across velixir's network instead of out to the internet and back, so an internal API no longer needs a public URL to protect. The address always points at the live release, so deploys and rollbacks don't change it.

Only your team's apps can reach it. Other teams can't, and neither can forks or Rewind previews, which run with your live variables and are kept off the private network on purpose. Nothing to switch on, on every plan. Read the post or the docs.

Tuesday: undo a bad deploy

Thursday's alerts tell you something broke. At 3am the next question is whether it was the deploy you shipped that afternoon, and now the alert answers it. When a process starts crashing within half an hour of a deploy, and it was healthy before, the alert names the release, as in "It started 4 minutes after v14 went live. v13 ran before it." The crash card offers Roll back to v13 as one button, with no rebuild and your current settings kept.

It stays quiet when it isn't sure, because a wrong suggestion at 3am is worse than none. The crash webhooks carry the suspected deploy and the release to roll back to, so a script or an agent can do it for you. If the deploy changed your data as well, Rewind takes code, config and databases back to a moment you choose, with a preview first and an undo after. Crash alerts and rollbacks are free on every plan. Read the post or the docs.

Under the hood

Over the same days, we moved the network underneath our EU cluster from kube-router to Cilium, with apps running throughout. Every new app now has its network policy in place before it sends its first packet. We wrote up how we did it, including the one issue that reached production, in Migrating K3s from kube-router to Cilium, live.

Where to start

Private networking, log drains, crash alerts, webhooks and rollbacks are free on every plan. Workers and cron jobs are on paid plans, with cron billed per second, and the pricing page has a calculator.

Moving from Heroku, Render or Railway? The switch guides map worker dynos, schedulers and pre-deploy commands straight across.