Tech Explainer: What’s the difference between AI training and AI inference?

Artificial Intelligence (AI) training and inference are two sides of the same coin. Training is the process of teaching an AI model how to perform a given task. Inference is the AI model in action, drawing its own conclusions without human intervention. Take a theoretical machine learning (ML) model designed to detect counterfeit one-dollar bills. During the training process, AI engineers would feed the model large data sets containing thousands, or even millions, of pictures. And tell the training application which are real and which are counterfeit. Then inference could kick in. The AI model could be uploaded to retail locations, then run to detect bogus bills. During training, the pictures fed to the AI model would include annotations telling the AI how to think about each piece of data. AI training and inference also have different technology requirements. Training is far more resource-intensive. Inferencing workloads are both more concise and less demanding than those for training. The AMD Instinct MI300A is an accelerated processing unit (APU) that combines the facility of a standard AI accelerator with the efficiency of AMD EPYC processors. A single AMD MI300A APU packs 228 GPU compute units, 24 of AMD’s ‘Zen 4’ CPU cores, and 128GB of unified HBM3 memory. Compared with the previous-generation AMD MI250X accelerators, this translates to approximately 2.6x the workload performance per watt using FP32.