Automation and AI

Inference

Also called Model run, prediction, API call

Running a trained model to get an answer, and the per-run cost and delay that come with every one.

Quick facts: Inference

Category
Automation and AI
Also called
Model run, prediction, API call
Level
Intermediate
Affects
Running cost, response speed, model choice
Where to see it
Provider usage dashboards in OpenAI, Google Vertex AI, Anthropic; run logs in n8n, Make and Zapier
In this article4
  1. How inference works
  2. Why inference matters
  3. Where inference goes wrong
  4. How to act on it

How inference works

Building a model and running it are two separate activities. Training is the long, expensive job of creating the model in the first place, and for anything you are likely to use it happened once, on somebody else’s hardware. Inference is every occasion afterwards when the finished model is handed an input and produces an output: a caption drafted, an enquiry classified, a question answered in a chat window.

Each call reads your input, works through the model and returns the answer piece by piece. You are charged by the amount of text going in and coming out — the provider counts it in tokens — and you wait while it happens. Those two costs, money and delay, are what turn inference from a technical footnote into a planning problem.

Why inference matters

It is the point where an AI idea meets a budget. A workflow that feels free in a demo is billed on every run once it is live, and a workflow triggered by every page view or every incoming message runs a great deal. Knowing that your monthly bill is roughly the number of runs multiplied by the size of each one is enough to sketch a sensible plan before anyone writes code.

Speed matters just as much on anything a customer sees. A visitor waiting on a chat reply behaves very differently from a marketer waiting overnight on a batch of product descriptions. In Nepal, where most traffic arrives on a mobile connection, a slow model response lands on top of a slow network, and the visitor gives up sooner than the dashboard suggests.

Where inference goes wrong

The usual mistake is picking the largest available model for everything. Most marketing tasks — tagging an enquiry, tightening a subject line, pulling a phone number out of a message — are handled reliably by a small model at a fraction of the cost and the wait.

The second is running a model where a rule would do. Deciding which service an enquiry relates to, when the form itself already says so, is spending on a model to learn something you were told.

The third is having no ceiling. An automation caught in a loop, or a public chatbot with no limit on who may call it, will keep running and keep billing until somebody notices.

How to act on it

Measure before you scale. Run the workflow on a realistic sample, note what one run costs and how long it takes, then multiply by the volume you honestly expect rather than the volume you hope for. Cache anything repeated, batch anything that does not need an instant answer, and save the expensive model for the steps that genuinely need judgement.

Set a hard spend limit in the provider console and a proper failure path in the workflow, so a broken step stops instead of retrying forever. Then treat inference in your AI automation plan the way you treat hosting: a recurring cost that grows with use, not a one-off build item.

Do and do not

Do

  • Cost one run, then multiply by real volume
  • Use the smallest model that answers correctly
  • Set a hard spend limit at the provider

Do not

  • Call a model where a simple rule works
  • Leave a public chatbot with no usage limit
  • Retry a failing step without a stop condition

Questions people ask about this

Why does my AI tool cost more some months than others?

Because you are billed for use, not for a seat. The bill follows how many times the model was run and how much text went in and out of each run. A busier month, longer inputs, a chatbot left open to the public or an automation that retried after failures will all push it up without anything having changed in the setup.

Do I need my own server to run an AI model?

For almost every marketing use, no. You call a provider over the internet and pay per run, which avoids buying hardware and keeps you on current models. Running your own becomes worth discussing only when data cannot legally leave your systems, or when volume is so steady and so high that renting starts to look dearer than owning.

How can I make AI responses faster for visitors?

Use a smaller model for the step the visitor is waiting on, keep the prompt short, and stream the answer so words appear as they are produced rather than after a silence. Move anything that does not need to happen live, such as summarising or tagging, into a background job that runs after the visitor has gone.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.