Guide to AI Hardware and Architecture

Artificial intelligence (AI) computer hardware and design principles enable AI developers to run machine learning algorithms and huge datasets in data centers. But what about running AI or machine learning in a factory? In an agricultural field? Or at a remote mining operation? AI hardware vendors and engineers are innovating specialized AI hardware components as well as architectural design principles to enable the use of AI/ML just about anywhere. 

Guide to AI Series Guide to AI Middleware Guide to AI Software

In this guide, part of a series from A3 that introduces AI software, AI middleware, and AI hardware, you learn about AI architecture and the types of hardware used in AI processing. You also learn about AI hardware for use at the network edge and in remote and extreme environments. Industrial AI experts from two companies that provide hardware for AI, Gidel and Neousys Technology, shared their expertise to help us explore this topic. 

What is AI architecture? 

AI architecture describes the structure of an AI system, end to end. An AI architect follows a framework of design principles to document details ranging from strategic vision and planning to the overall design of an AI system. Data acquisition, processing, machine learning models, decision support, connectivity, integration, and deployment are elements of an AI architecture.

Mix-and-match traditional and AI hardware

AI systems can compare two images or datasets and identify a difference, which has the potential to revolutionize the way manufacturing and industrial production automate visual inspection. Advancements in sensors used in AI inference and decision-making enable this capability but come with tradeoffs—larger image sizes and higher frame rates. In other words, much more data. Printed circuit board (PCB) visual inspection requires a huge amount of data processing, for example.

AI architects often mix and match hardware components to create heterogenous AI architectures that balance tradeoffs.

 “You can combine hardware technologies at the image-processing level as well as the machine-vision level,” Reuven Weintraub, founder and CTO of Gidel explains. “Combining old with new enables the development of next-generation applications that were impossible before, but with AI, produces both higher performance and lower cost,” he says.

For example, Gidel’s ultra-compact computers for high-bandwidth vision applications at the network edge combine NVIDIA Jetson with FPGA-based frame grabbing and acceleration technology and high-bandwidth camera interfaces. 

FPGA hardware makes many AI vision projects possible by pre-processing the image data. “Very high-speed smart camera applications, such as automatic sorting machines in the food industry, require FPGAs to first cut down the size of the images fed into the AI hardware,” says Hai Migdal, director of sales at Gidel. “By contrast, bringing all the available data from the cameras to the AI system would corrupt it—it’s simply too much data for real-time processing.” 

High-powered computers on premises and powerful hardware in the cloud can both process AI data.  

However, your operation situations might lack connectivity or bandwidth, demand real-time response, or require great care in data storage to comply with privacy regulations. In those scenarios, it is best to do as much as possible onsite, Weintraub says. Fortunately, much or all the AI processing can be performed locally without good network connections, which are lacking in remote areas. In addition, pre-processing data at the network edge reduces the cost and size of an AI system.

With ample preprocessing, AI hardware and architecture make cost-effective inference possible. “You get good results with fewer resources and reduce or eliminate bandwidth requirements for AI-specific computation,” Weintraub says. 

Although AI engineers are usually software engineers with experience using GPUs not FPGAs, companies such as Gidel offer FPGA expertise to help companies overcome image processing bottlenecks and reduce the amount of data transmitted using FPGA platforms.

Rugged embedded AI hardware for harsh environments

Remote and harsh environments—including outdoors—impose additional requirements on AI hardware and architecture. The best AI hardware for any project involves balancing tradeoffs between performance, power consumption, physical dimensions, and environmental factors, such as operating temperature, shock, vibration, dust, and water. Ruggedized AI hardware delivers reliable operation, even under high ambient temperatures. It also protects sensitive electronic components and cables from shock and vibration through mechanical design. Waterproof hardware further shields components from rain, snow, or other elements—and makes the computer washable for use in dirty environments. 

Fanless hardware helps overcome a number of environmental challenges by dissipating heat without a fan. It safeguards electronics from dust on a motherboard with conduction cooling. It reduces power consumption. It increases system reliability by eliminating ball bearings. And it enables the openings required by a fan to be sealed. 

In response to market demand, computing vendors, such as Neousys Technology, have created an ecosystem of low power, fanless, waterproof, and mobile network connectivity AI hardware. 

Traditionally, designers of industrial visual inspection systems required the equipment to be placed in controlled environments with fixed lighting, and the systems’ capabilities were simplistic—sorting products into good (passed) and reject (failed) categories, for example. By contrast, AI can grade and sort items into many categories or grades at once. Additionally, smart robotic arms flexibly pick up and pack objects of a variety of shapes and sizes, sorting them into multiple bins based on the results of the AI inspection. 

With the addition of ruggedized hardware, portable AI systems can grade and sort produce in the field—on a potato harvester, for example—explains Kaichu Wu, a product manager at Neousys Technology. The computer hardware manufacturer offers an edge AI GPU computing platform for system integrators who want to prototype next-generation AI hardware for use in industrial, rugged and extreme environments.  

Beyond protecting AI hardware components from demanding environmental conditions, power consumption looms as a big issue for portable AI hardware. In contrast to data center GPUs that require a 300-watt power supply to run fan-based cooling or a liquid-cooling pump, much less energy is available to AI hardware operating in remote places and on mobile platforms. 

“Any device powered by a battery needs to manage power consumption,” Wu says, which is why the NVIDIA Jetson modules are a good solution for embedded AI systems: they only use 10 to 30 watts and deliver AI processing comparable to a data-center GPU. Dedicated AI hardware often has better power consumption relative to performance and cost than general-purpose hardware. 

Size and weight of the AI hardware system may be important considerations as well. Many applications benefit from compact hardware, and more than half of the weight of a system can be aluminum in a computer case. 

Longevity of product life is another consideration for AI hardware designers, Wu says. When it is time to update hardware, software must be updated, too. Both increase total cost of ownership (TCO). In regions with high-cost labor, such as Europe and North America, an AI hardware designer may be willing to pay a premium for hardware that lasts longer. NVIDIA supports Jetson computers through a five- to 10-year operating lifetime while other AI hardware vendors may offer support for only two to three years, he says. 

Take your first step: Add AI to an existing process

Industrial automation engineers may be surprised to learn that sometimes adding AI to an existing process is trivial, because AI hardware is exceptionally good at doing some things more cost-effectively than traditional computers. A good place to start is to add an AI component when modernizing a system. Purpose-built AI hardware for industrial and extreme environments, as well as system integration services, are available to get your project underway and into production quickly so you can reap the benefits of artificial intelligence sooner.