Budget alerts that reach a human in time
A budget alert tells you that you crossed a line. It does not tell you how fast you were going, and it does not stop the car.
A Cloud Billing budget defaults to alerting at 50, 90 and 100 percent of the amount, by email, to Billing Account Administrators and Billing Account Users. Each of those defaults is wrong for an agency running client warehouses: the thresholds are percentages of a month rather than a rate, the recipients are usually finance rather than the person who can fix it, and the alert is lagging. Add a rate-based alert, route it to a named human, and put a hard quota underneath.
What a runaway actually looks like
Do the arithmetic before configuring anything. A dashboard query that scans 290 GB, accidentally scheduled every fifteen minutes, runs 96 times a day. That is roughly 27 TiB a day, and at $6.25 per TiB in the US multi-region it is about $170 a day.
Now put that against a $300 monthly budget with the default thresholds. The 50 percent alert fires part way through day one. The 90 and 100 percent alerts fire on day two. Then nothing more happens, because there are no thresholds above 100 percent, and the month ends somewhere north of $5,000.
Three emails, all in the first 48 hours, all saying a percentage of a number nobody remembers choosing. If they arrive in the inbox of someone who does not administer BigQuery, the meter runs for four weeks.
That is the design problem. Budgets are built to answer whether you will exceed a plan, and a runaway query is a question about rate, not about a plan.
The three defaults worth changing
When you create a budget, the default alert thresholds are 50, 90 and 100 percent of the budget amount, calculated against actual spend. Those are sensible defaults for a planned workload and poor ones for an anomaly.
Change the first one. Add a threshold well below 50 percent, at something like 20 percent, purely as an early tripwire. In a normal month it fires around the eighth day and everybody ignores it, which is fine. In an abnormal month it fires on day one, and the timing itself is the signal.
Add a forecast-based threshold too. A budget can evaluate against forecasted spend rather than actual, which is the only setting that reacts to a change in rate rather than a change in total. It is the closest thing to a rate alarm the budget system offers.
And set the budget amount from your own arithmetic rather than a round number. If your steady state is $40 a month, a $300 budget means every threshold is meaningless. A $60 budget makes the first threshold a real event.
Add thresholds above 100 percent as well. A budget with nothing above the full amount goes silent exactly when the situation is worst, and a threshold at 200 percent is the difference between finding out on day two and finding out when the invoice arrives four weeks later.
Where the alert lands
By default, budget alert emails go to Billing Account Administrators and Billing Account Users on the billing account. In a small company that is often one person in finance who has never opened BigQuery.
Budget alert email recipients can be customised through Cloud Monitoring notification channels, which is the mechanism to use rather than granting billing roles to engineers so that they receive email. Granting a role to fix a notification problem is how permission sprawl starts.
Route it to a channel a team watches, not to an individual's inbox. An email to one named person is an alert that stops working the week they are on holiday, and holidays are exactly when an unattended scheduled query has time to run.
For anything that should trigger a response rather than a read, programmatic notifications publish budget alerts to a Pub/Sub topic, which is what you build against if you want an automated reaction rather than a person.
What each control is actually for
| Control | Detects | Time to a human | Cannot do |
|---|---|---|---|
| Budget threshold on actual spend | Crossing a monthly total | Hours, after billing data lands | Tell you the rate |
| Budget threshold on forecast | A change in rate | Hours | Stop anything |
| Pub/Sub budget notification | The same events, machine readable | Immediate, if you built something | Exist without code |
| BigQuery custom query quota | Bytes processed per day | Not applicable, it blocks | Warn you before it bites |
| Job history query | Which query, whose, how often | Whenever you run it | Fire on its own |
Read down the last column and the design falls out. No single control both warns and stops, so you need at least two, and most teams have only the first row.
The fourth row is the one that ends the incident rather than reporting it. Custom query quotas cap the bytes processed per day at project level or per user, and a quota is the only item here that turns a $5,000 month into a failed query and an annoyed analyst.
The fifth row is the diagnosis rather than the alarm. Once something has fired, the question is which query and whose, which is the exercise in finding the query that cost you four hundred dollars.
The gap between the first row and the fourth is where most incidents live. The alert arrives after the money is spent, and the quota prevents it being spent at all, so a setup with alerts and no quota is a setup that will document its own worst month in careful detail.
One budget per client, not one per company
A single budget across an agency's whole cloud footprint is close to useless. Thirty clients averaging out means one client tripling their usage moves the total by a few percent and crosses nothing.
Scope a budget to each client's project. Then the threshold is measured against a number small enough for a change to be visible, and the alert names the client in its subject line, which is most of the triage done before anyone opens it.
That is only possible if each client already has their own project, which is one more argument for the structure in laying out BigQuery projects and datasets. Budgets, quotas and cost attribution all inherit from that one decision.
Scope the quota the same way. A per-project daily byte cap protects the client. A per-user cap, set lower, protects the project from one person exploring in the console at eleven at night.
Test the alert, then test the human
An untested alert is a belief. Set a temporary budget of a dollar or two on a real project, run a query, and confirm the email arrives, in whose inbox, and how long it took. That exercise routinely finds that the notification channel was never verified.
Then test the response. Whoever receives it should be able to say, without looking anything up, what the first three actions are: find the job, stop the schedule, check whether the quota held. If they cannot, the alert has a recipient and no owner.
Review the numbers quarterly. Budgets set during a build are wrong six months later, and a threshold that has never fired in a year is either a healthy account or a broken alert, and those look identical from the outside.
None of this replaces designing for a small bill in the first place. Alerts are the backstop; the actual control is layout and query shape, which is the argument in keeping a BigQuery bill boring, and the reason a commitment to reserved capacity is almost never the answer for marketing workloads, as set out in on-demand or reserved.
For a franchise operator running per-location reporting across more than fifteen clinics, the cost line should be uneventful every month. Making it uneventful, and alerting when it stops being, is part of what a marketing data warehouse build should hand over configured rather than documented.
Put the runbook where the alert lands rather than in a wiki. Three lines in the notification channel's description, naming the console page to open, the quota setting to check and the person to tell, means whoever is holding the phone at the weekend does not have to find anything first.