The Ascend AI processor divides the network execution layers into basic execution units to obtain the granular computing engine (Engine). Each engine performs specific operations on data during process orchestration, such as classifying images, preprocessing input data, and outputting image data. In short, the computing engine is customized by developers to perform the required functions.

Generally, through the unified invoking of the process orchestrator, the entire deep neural network application includes four engines: data engine, preprocessing engine, model inference engine, and postprocessing engine. The data engine prepares the dataset (for example, MNIST dataset [4]) required by the neural network and processes the corresponding data (such as image filtering) as the data source of the subsequent computing engines. Generally, input media data needs to be preprocessed to meet the computing requirements of the Ascend AI processor. The preprocessing engine preprocesses the media data, performs operations such as image and video encoding and decoding, and format conversion. In addition, all functional modules of the DVPP need to be called by the process orchestrator. The model inference engine needs to be used when data flows are based on neural network inference. The model inference engine mainly uses the loaded model and the input data flow to complete the feed-forward calculation of the neural network. After the model inference engine outputs the result, the postprocessing engine performs subsequent processing on the output of the model inference engine, for example, adding a frame and adding a label to the image recognition. The four engines can be used to construct multiple types of neural network applications and implement service functions based on the neural network in a combination of computing engines. The computing engine integrates functions similar to operators, abstracts the functional structure of the neural network from a higher level, and brings concise basic functional modules for the specific application development of the neural network, accelerating the development process of the neural network application.

In addition to engine encapsulation of the execution layer, the Ascend AI processor performs offline model conversion on the neural network model, so that it can be executed on the Ascend AI processor in the form of an offline model. The neural network model can be converted to an offline model by using the Offline Model Generator. The offline model includes the dependency of operators on the network and the training weight information. These dependencies and weight information constitute a neural network calculation diagram in essence. The offline model can run independently on the hardware, so that the processor can completely separate from the basic neural network open-source framework when completing the specific inference task, thereby saving a lot of resource overhead. The offline model can also reduce the size of the model, save storage space, and adapt to lightweight applications by means of quantization and compression. In addition, the offline model is closely related to the hardware and is highly optimized based on the hardware of the Ascend AI processor. In this way, performance and efficiency are greatly improved.

In fact, the process of implementing a neural network offline model is achieved through the orchestration and design of the computing engine process. After the orchestration is complete, the computing engine flowchart is generated. Fig. 4.3 shows a typical example. In the computing engine flowchart, each data processing node is a computing engine. Data flows are processed and calculated based on the orchestrated paths, and the results are output. The final output of the entire flow chart is the result generated by the corresponding neural network. Two adjacent computing engine nodes establish a connection based on the configuration file in the flowchart of the computing engine. The actual data between nodes flow according to the node connection defined by a specific network model. After node attributes are configured, data is imported to the start node of the computing engine flowchart to start the running process of the entire computing engine.

As the process orchestrator of the entire neural network, which is between the L1 processor enabling layer and the L3 application enabling layer, it provides unified standard intermediate interfaces for multiple operating systems (such as Linux and Android), in addition, it is responsible for establishing, destroying, and reclaiming computing resources.

During the process of creating a computing engine flowchart, the process orchestrator completes the computing engine flowchart creation based on the configuration file of the computing engine. Before execution, the process orchestrator provides input data. If the form of video input data does not meet the processing requirement, the DVPP module may be invoked by using a corresponding programming interface to perform data preprocessing. If the data meets the processing requirements, the Offline Model Executor is invoked to perform inference calculation. During the execution, the process orchestration apparatus has multiple node scheduling and multiprocess management functions. It is responsible for computing the running of the process on the device, guarding the computing process, and collecting and summarizing related execution information. After the model is executed, it provides the application on the active node with the functionality of obtaining the output result.

The Ascend AI processor uses the computing engine and process orchestrator to orchestrate and execute the computing engine diagram. The process orchestrator provides four common computing engines, which effectively enables multiple neural network offline models to be object oriented and process based. In addition, the data flow control is enhanced, so that the execution of different models has a general process template for development and design. The concept of computing engine also allows data flows and computing flows to be centrally managed in the flowchart.

4.2.1.2: Application scenarios

The Ascend AI processor can set up a hardware platform with different dedicated hardware for different services. Therefore, based on the collaboration between the device and host, there are two common application scenarios: Accelerator and Atlas DK. The applications of the process orchestrator in these two typical scenarios are different.

Acceleration card form

In an acceleration card application scenario, as shown in Fig. 4.4, a PCIe [5] card based on an Ascend AI processor is mainly oriented to a data center or an edge server scenario. It is a dedicated acceleration card for accelerating neural network computing. The PCIe accelerator card supports multiple types of data precision. Compared with other similar acceleration cards, the PCIe accelerator card provides more powerful computing power for the calculation of the neural network. In the accelerator scenario, a host needs to be connected to the accelerator card, and the host is a server or a personal computer that supports the PCIe card. The host invokes the neural network computing capability of the acceleration card to perform corresponding processing.

As shown in Fig. 4.5, the process orchestrator function in the acceleration card scenario is implemented by three subprocesses: process orchestration agent subprocess (Matrix Agent), process orchestration daemon (Matrix Daemon), and process orchestration service subprocess (Matrix Service). The process orchestration agent subprocess runs on the host, controls and manages the data engine and postprocessing engine, processes data interaction with the host application, controls the application and communicates with the processing on the device. The process orchestration daemon and service subprocesses run on the device. The process orchestration daemon process creates processes on devices based on the configuration file, starts the process orchestration process on the device, and manages the process orchestration process. After the calculation is complete, the process orchestration process is released and resources are reclaimed. The process orchestration service subprocess starts and controls the preprocessing engine and model inference engine on the device. It controls the preprocessing engine to call the programming interface of the DVPP module to implement the video and image data preprocessing function. The process orchestration service subprocess can also call the AI model manager programming interface in the Offline Model Executor to load and inference offline models.

Fig. 4.5 shows the process of calculating the offline model of the neural network through the process orchestrator. The process consists of three steps: creating a computing engine flowchart, executing a computing engine flowchart, and destroying a computing engine. The process of creating a computing engine is to use the computing engine of different functions to orchestrate the execution process of the neural network. The computing engine flowchart is used to calculate and implement the neural network function based on the defined flowchart. The process of destroying the computing engine is used to release the system resources occupied by the computing engine after all computing is complete.

The process of executing an application in the system is as follows: first, the application program invokes a subprocess of the host’s process orchestration agent to orchestrate the computing engine flowchart according to a precompiled configuration file of the corresponding neural network, creates an execution process of it and defines a task of each computing engine. Then, the computing engine orchestration unit uploads the offline model file and the configuration file of the neural network to the process orchestration daemon process on the device, and then the process orchestration service subprocess of the device initializes the engine. The process orchestration service subprocess controls the model inference engine to invoke the initialization interface of the AI model manager to load the offline model of the neural network.

After the offline model is loaded, the process orchestration agent subprocess on the host is notified of the application data input. The application directly sends the data to the data engine for processing. If the input is media data and does not meet the calculation requirements of the Ascend AI processor, the preprocessing engine starts immediately and calls the interface of the DVPP module to preprocess media data, such as encoding, decoding, and scaling. After the preprocessing is complete, the preprocessing engine sends the data to the model inference engine. In addition, the model inference engine invokes the processing interface of the AI model manager to combine the data with the loaded offline model to complete inference calculation. After the output result is obtained, the model inference engine invokes the data sending interface of the process orchestration unit to return the inference result to the postprocessing engine, which completes the postprocessing operation of the data, and finally returns it to the application program by using the process orchestration unit, then completes the flowchart of executing the computing engine.

After all engine data is processed and returned, the application notifies the process orchestration agent subprocess to release the hardware resources of the data engine and postprocessing engine. The process orchestration agent subprocess instructs the service subprocess to release resources of the preprocessing engine and the model inference engine. After all, resources are released, the computing engine flowchart is destroyed. The process orchestration agent subprocess instructs the application program to perform the next neural network execution.

The three software units implement the computing engine process in an orderly manner and work together to implement the function application of a neural network offline model on the Ascend AI processor.

Developer board form

In the developer board scenario, Huawei launches the Atlas developer suite (Atlas200 Developer Kit, Atlas 200DK), as shown in Fig. 4.6, including the hardware of a developer board that uses the Ascend AI processor as the core. The developer board provides the kernel functions of the Ascend AI processor through the peripheral interfaces on the board. This facilitates the control and development of the processor from the external. It can easily and intuitively display the neural network processing capability of the Ascend AI processor. Therefore, the developer board based on the Ascend AI processor can be widely used in different AI fields and is also the main hardware on edge devices in the future.

In the developer board environment, the control function of the host is integrated into the developer board. Therefore, the process orchestrator has only one software unit to run the process, which includes the subprocess functions of the host process orchestration agent, device process orchestration service, and process orchestration daemon process in the acceleration card scenario, as shown in Fig. 4.7.

In this case, as the functional interface of the Ascend AI processor, the process orchestrator function implements data interaction and commands between the computing engine flowchart and applications. The process orchestrator creates a computing engine flowchart based on the configuration file of the computing engine process to orchestrate, control, and manage processes. After the calculation is complete, the process orchestrator releases the computing engine flowchart and reclaims resources. During preprocessing, the process orchestrator invokes the interface of the preprocessing engine to perform media preprocessing. During the inference process, the process orchestrator can also invoke the programming interface of the AI model manager to load and inference offline models. In the developer board application scenario, the process orchestrator orchestrates the implementation process of the entire computing engine flowchart and does not need to interact with other devices.

Similarly, in the developer board scenario, the offline model of the neural network uses the process orchestrator to perform inference calculation. The process is also divided into three main steps: creating, executing, and destroying a computing engine flowchart, as shown in Fig. 4.7.

Specifically, the application invokes the process orchestrator, creates a flowchart of the computing engine according to the network model, and initializes the computing engine. During initialization, the model inference engine loads the model through the initialization interface of the AI model manager to complete the creation of the computing engine flowchart. Then, the data is imported to the data engine. If the media data format does not meet the requirements, the preprocessing engine performs preprocessing. Then, the model inference engine invokes the AI model manager in the Offline Model Executor to perform inference calculation. After the calculation is complete, the model inference engine invokes the data output interface provided by the process orchestrator to return the inference result to the postprocessing engine, which returns the reasoning result to the application program through the callback function to complete the execution of the calculation engine flowchart. Finally, after the program calculation is complete, the process orchestrator destroys the computing engine flowchart and releases resources, then completes implementation of the neural network function in the developer board scenario.

4.2.2.1: Functional architecture

The DVPP module provides six modules: video decoding (VDEC) module, video encoding (VENC) module, JPEG decoding (JPEGD) module, JPEG encoding (JPEGE) module, PNG decoding (PNGD) module, and visual preprocessing (VPC) module. The video decoding module provides the video decoding function of the H.264/H.265, which decodes the input video streams and outputs images for scenarios such as video recognition. For the counterparts, the video encoding module provides the encoding function of the output video. For the output data of the vision preprocessing module or the raw input YUV [6] format data, the video encoding module encodes and outputs the H.264/H.265 video, so as to directly play and display the video. This function is commonly used in high-speed data transmission scenarios such as cloud games and simulated mobile phone service. For pictures in JPEG format, the corresponding encoding and decoding module is also available. The JPEG decoding module decodes the JPEG picture, converts the original JPEG picture into YUV data, and preprocesses input data of the neural network. After the image processing is complete, the JPEG encoding module needs to be used to restore the processed data in JPEG format for training of the neural network and postprocessing of the inference output data. When the input picture is in PNG format, the flow orchestrator needs to invoke the PNGD decoding module to decode the picture and output the PNG picture in RGB format to the Ascend AI processor for training or inference calculation. In addition to the encoding and decoding modules for video and image formats, the DVPP module also provides other functions such as format conversion (for example, conversion from YUV/RGB format to YUV420 format), resizing, and cropping.

The execution process of the digital visual processing module is shown in Fig. 4.8 and needs to be completed by a process orchestrator, a digital vision processing module, a DVPP driver, and a DVPP hardware module. The top layer of the framework is the process orchestrator, which is responsible for scheduling the functional modules in the DVPP for processing and managing data flows. The digital vision processing module is located at the upper layer of the functional architecture and provides a programming interface for the process orchestrator to invoke the video or image processing module. Through these interfaces, parameters related to the encoding/decoding or vision preprocessing module can be configured. The DVPP driver is located in the middle and lower layers of the functional architecture and is closest to the hardware module of the DVPP. It is responsible for device management, engine management, and engine module driver. The driver allocates the corresponding DVPP hardware engine according to the tasks delivered by the digital vision processing module and reads and writes the registers in the hardware module to complete other hardware initialization. The bottom layer is the real hardware computing resource DVPP module group. It is a dedicated accelerator independent of other modules in the Ascend AI processor. It is responsible for executing the encoding, decoding, and preprocessing tasks corresponding to images and videos.

4.2.2.2: Preprocessing mechanisms

When the input data enters the data engine, if the engine detects that the data format does not meet the processing requirements of the subsequent AI Core, the DVPP module can be enabled to perform data preprocessing. The data flow shown in Fig. 4.8 is taken as an example. First, the process orchestrator moves the data from the memory to the buffer of the DVPP module for caching. According to the format of the specific data, the preprocessing engine completes parameter configuration and data transmission through the programming interface provided by the digital vision processing module. After the programming interface is started, the digital vision processing module transfers the configuration parameters and the original data to the driver. The DVPP driver invokes the PNG or JPEG decoding module for initialization and task delivery. In this case, the PNG or JPEG decoding module starts the actual operation to decode the picture and obtain the YUV or RGB data to meet the subsequent processing requirements.

After the decoding is complete, the process orchestrator uses the same mechanism to continue to invoke the DVPP module to convert the image into the YUV420SP format, because the YUV420SP data storage efficiency is high and the occupied bandwidth is small. Therefore, more data can be transmitted under the same bandwidth to meet the powerful computing throughput requirement of the AI Core. In addition, the DVPP module can also crop and resize images.

A typical image crop and zero-padding operation are shown in Fig. 4.9. The vision preprocessing module extracts the part of the to-be-processed image from the original image, and then performs zero padding, so as to retain the edge feature information during the calculation of the convolutional neural network. The zero-padding operation requires four padding sizes namely, upper, lower, left, and right. The image edge is expanded in the zero areas, and finally, a zero-padded image that can be directly calculated is obtained.

After a series of preprocessing, the image data that meets the format requirements will enter the AI Core for the required neural network calculation under the control of the AI CPU. After that, the output image data is encoded by the JPEG encoding module. After encoding, the data is stored in the buffer of the DVPP module. The process orchestrator obtains the data for subsequent operations, releases DVPP of computing resources, and reclaims the cache.

During the preprocessing, the process orchestrator implements functions of different modules. As a customized data supply module, the DVPP module uses a heterogeneous and dedicated processing method to quickly transform image data and provides sufficient data sources for AI Core, thereby satisfying a requirement of a large amount of data and a large bandwidth in a neural network calculation.

4.2.3.1: Functional framework

The TBE provides the capability of developing customized operator based on TVM [7]. Through the TBE language and custom operator programming interface, the corresponding neural network operators can be developed. The structure of the TBE is shown in Fig. 4.10, which includes a domain-specific language (DSL) module, a scheduling (Schedule) module, an intermediate representation (IR) module, a compiler transfer (Pass) module, and a Code Generation (CodeGen) module.

TBE operator development is divided into computing logic writing and scheduling development. The domain-specific language module provides the programming interface of the operator calculation logic and writes the calculation and scheduling process of the operator directly based on the domain-specific language. The operator calculation process describes the calculation methods and steps of the operator, and the scheduling process describes the planning of the data block and data flow direction. Each calculation is processed according to a fixed data dimension. Therefore, data dimension splitting needs to be performed on the operators executed on different computing units in the Ascend AI processor in advance. For example, the Cube Unit, the Vector Unit, and the operator executed on the AI CPU have different requirements for the input data dimension.

After defining the basic implementation process of the operator, you need to start the tiling (Tiling) submodule in the scheduling module, divide the data among the operator according to the scheduling description, and specify the data transfer process to ensure that the execution on the hardware is optimal. In addition to data dimension segmentation, the operator fusion and optimization capabilities of the TBE are also provided by the fusion (Fusion) submodule in the scheduling module.

After the operator is written, an IR needs to be generated for further optimization, and the IR module generates an IR by using an IR format similar to TVM. After the generation, the module needs to compile and optimize the modules for various application scenarios. The optimization modes include double-buffer (Double Buffer), pipeline (Pipeline) synchronization, memory allocation management, instruction mapping, and block adaptation Cube Unit.

After the operator is processed by the Pass module, the code generation module generates a temporary file of the C-like code. The temporary code file may generate an implementation file of the operator by using the compiler and may be directly loaded and executed by the Offline Model Executor.

To sum up, a complete customized operator uses the submodule of the TBE to complete the entire development process and provides the primitive operator calculation logic and scheduling description from the domain-specific language module. After the operator prototype is formed, the scheduling module performs data segmentation and operator fusion and enters the IR module to generate the IR of the operator. The Pass module performs compilation optimization such as memory allocation by means of the IR. Finally, the code generation module generates a C-like code for compiler. In the definition process of the operator, the TBE not only completes the operator writing but also completes the related optimization, which improves the execution performance of the operator.

4.2.3.2: Application scenarios

Fig. 4.11 shows the three application scenarios of the TBE. Generally, a neural network model implemented by using a standard operator in the deep learning framework has been trained by using a GPU or another type of neural network processor. If the neural network model continues to run on the Ascend AI processor, it is naturally expected to benefit the portability of the model and perform optimally without changing the original code. Therefore, the TBE provides a complete TBE operator acceleration library. The operator function in the library has a one-to-one mapping to common standard operators in the neural network. Meanwhile, the software stack provides a programming interface for the operator to use. Thus, it provides acceleration for various frameworks or applications in the upper-layer deep learning, and avoids the development of adaptation code for the bottom-layer adaptation of the Ascend AI processor.

If a new operator appears in the neural network model construction, the standard operator library provided by the TBE may not meet the development requirements. In this case, you need to develop a customized operator in the TBE language. This development mode is similar to that of the CUDA C ++ on the GPU. In this way, more multifunctional operators can be implemented, and various network models can be flexibly compiled. The implemented operator is delivered to the compiler, and finally executes on the processor, revealing the acceleration capability of the AI Core or AI CPU.

In an appropriate scenario, the operator fusion capability provided by the TBE improves operator performance, and the neural network operator can perform multilevel cache fusion based on different levels of buffers so that the Ascend AI processor improves the on-chip resource utilization when the integrated operator is executed.

To sum up, because the TBE provides the operator development capability, the capability of standard operator invoking and operator fusion optimization, the Ascend AI processor can meet various function requirements in the actual neural network application, the network construction method is more convenient and flexible, and the fusion and optimization capabilities will also improve the running performance.

4.2.5.1: Functions

The TS runs on the task scheduling CPU on the device side to distribute specific tasks assigned by the Runtime to the AI CPU. It can also allocate tasks to the AI Core through the hardware task Block Scheduler (BS) and return the execution result to the Runtime after the execution is complete. Generally, the TS processes the following transactions: AI Core tasks, AI CPU tasks, memory copy tasks, event recording tasks, event waiting tasks, cleanup and maintenance (Maintenance) tasks, and performance analysis (Profiling) tasks.

Memory copy is mainly performed in asynchronous mode. The event recording task records the event information. If a task is waiting for the current event, it can continue execution after the event record being completed, so that the stream is not blocked due to the event record itself. The event-waiting task could refer to: (1) if the waiting event has occurred, then the task is accomplished directly, or (2) if the waiting event has not yet occurred, then waiting task is put into the to-be-processed list, and all subsequent tasks of the waiting task in the same execution stream are paused, then resumes to process when it occurs.

After the task execution is complete, the maintenance task performs cleaning based on the respective parameters of each task and reclaims the computing resources. During the execution, you may need to record and analyze the computing performance. In this case, you need to use the performance analysis task to control the start and pause of the performance analysis operation.

Fig. 4.14 shows the functional framework of the TS. The TS is usually located at the device end and has the task scheduling CPU. The task scheduling CPU consists of the scheduling interface (Interface), scheduling engine (Engine), scheduling logic processing module, AI CPU scheduler, task block scheduler, system control (SysCtrl) module, performance analysis (Profile), and log (Log) module.

The task scheduling CPU implements communication and interaction with the Runtime and the driver through the scheduling interface. The task scheduling engine sends the task execution result to the task scheduling engine. The task scheduling engine is responsible for task organization, task dependency, and task scheduling control. The task scheduling engine manages the execution process of the task scheduling CPU. Based on the task type, the task scheduling engine divides tasks into three types: computing, storage, and control. The task scheduling engine distributes the tasks to different scheduling logic processing modules and starts specific kernel function tasks, storage tasks, and interflow event dependency management and scheduling.

The logic processing module is divided into a kernel execute module (Kernel Execute), a direct memory access module (DMA Execute), and an event execute module (Event Execute). The kernel executes module schedules and processes the computing task, implements the task scheduling logic on the AI CPU and AI Core, and schedules the specific kernel function. The direct memory access executes module implements the scheduling logic of the storage task, and schedules and processes the memory copy. The event execute module implements the scheduling logic of synchronization control tasks and implements the logic processing of interflow event dependency. After the scheduling logic of different types of tasks is processed, the corresponding control units start to perform hardware execution.

For the AI CPU task execution, the AI CPU scheduler in the task scheduling CPU uses software to manage the status of the AI CPU and schedule tasks. For the task execution of the AI Core, the task scheduling CPU distributes the processed task to the AI Core by using a separate task block scheduler hardware, and the AI Core performs the specific calculation, and the result of the calculation is returned by the task block scheduler to the task scheduling CPU.

During task scheduling, the system controls the system configuration and processor function initialization. The performance analysis and log modules monitor the entire execution process and record key execution parameters and detailed execution details. When the execution ends or an error occurs, performance analysis or error locating can be performed to provide a basis for detailed efficiency evaluation and correctness analysis of the execution process.

4.2.5.2: Scheduling process

During the execution of the offline model of the neural network, the TS receives specific execution tasks from the Offline Model Executor. The tasks depend on each other. Therefore, the TS needs to release the dependency relationship, perform task scheduling, distribute the tasks to the AI Core or AI CPU based on the specific task type and completes the calculation or execution of specific hardware. During task scheduling, a task is composed of multiple execution instructions (CMD). The TS interacts with the Runtime to schedule the entire task instruction. The Runtime executes on the CPU of the host, the instruction queue is located in the memory of the device, and the TS delivers the specific task instruction.

Fig. 4.15 shows the scheduling process. First, the Runtime invokes the dvCommandOcuppy interface of the driver to enter the instruction queue, queries the available storage space in the instruction queue according to the tail information of the instruction, and returns the available instruction storage space address to the Runtime. After receiving the address, the Runtime fills the currently prepared task instruction into the storage space of the instruction queue and invokes the dvCommandSend interface of the driver to update the current tail information and the credit (Credit) information of the instruction queue. After receiving the new task instruction, the queue generates a doorbell interrupt and instructs the TS to add a task instruction in the instruction queue in the device memory. After receiving the notification, the TS enters the device memory, moves the task instruction to the buffer of the scheduler, and updates the header information of the instruction queue in the DDR memory of the device. Finally, the TS sends the instruction in the cache to the AI CPU or the AI Core for execution according to the condition.

The structure of the software stack is basically the same as that of most accelerators. The Runtime, driver, and TS in the Ascend AI processor works closely together to distribute tasks to corresponding hardware resources and executes the tasks in an orderly manner. This scheduling process carries out tasks closely and orderly in the process of deep neural network calculation, which ensures the continuity and efficiency of task execution.

4.2.6.1: Functional framework

The framework manager consists of three parts: Offline Model Generator (OMG), Offline Model Executor (OME), and AI Model Manager, as shown in Fig. 4.16. Developers use the Offline Model Generator to generate an offline model and save the file with the suffix of om. Then, the process orchestrator in the software stack invokes the AI model manager in the framework manager to start the Offline Model Executor, load the offline model to the Ascend AI processor, and complete the offline model execution through the entire software stack. From the birth of the offline model to the loading and entering to the Ascend AI processor hardware until the last function operation, the offline framework manager always plays a management role.

4.2.6.2: Offline model generation

Taking the convolutional neural network as an example, the network model is first constructed under the deep learning framework and trained on the original data. And then the Offline Model Generator converts the original model and generates the optimized offline model, by performing operator scheduling optimization, weight data rearrangement and compression, and memory optimization. The Offline Model Generator is used to generate an offline model that can be efficiently executed on the Ascend AI processor.

Fig. 4.17 shows the principle of the Offline Model Generator. After receiving the original model, the Offline Model Generator performs model parsing, quantization, compilation, and serialization on the convolutional neural network model. The steps are described as follows:

Quantification

As shown in Fig. 4.18, an intermediate graph is generated after the parsing is complete. If the model needs to perform quantization processing, it may be performed by using an automatic quantization tool based on the structure and weight of the intermediate graph. In the operator, the weight and the offset can be quantized. During the offline model generation, the quantized weight and offset are stored in the offline model. During inference calculation, the quantized weight and offset can be used to calculate the input data, and the calibration set is used to train the quantization parameter in the quantization process, so as to ensure the quantization precision. If no quantization is required, the offline model is directly compiled to generate an offline model.

The quantization mode is divided into data offset quantization and nonoffset quantization. The quantization scale (Scale) and the quantization offset (Offset) parameters need to be output. In the data quantization process, when no offset is specified, the quantization scale of the quantized data is calculated by using the nonoffset mode. If the specified data offset is quantized, the data uses the offset mode. In this case, the quantization scale and offset of the output data are calculated. In the weight quantization process, because the weight has a high requirement on the quantization precision, the nonoffset mode is always used. For example, INT8-type quantization is performed on the weight file according to the quantization algorithm, so that the INT8 weight and the quantization degree may be output. In the process of offset quantization, the FP32-type offset data can be quantized into INT32-type data output according to the quantization degree of weight and the quantization degree of the data.

You can perform quantization operations when there are higher requirements for the size and performance of the model. In the offline model generation process, the high-precision data is quantized to the low-bit data, so that the final offline model is lighter, thereby saving the network storage space, reducing the transmission delay, and improving the operation efficiency. In the quantization process, because the model storage size is greatly influenced by parameters, the Offline Model Generator focuses on the quantization of the convolution operator, full connection operator, and depth separable convolution (ConvolutionDepthwise).

4.2.6.3: Loading offline model

After the Offline Model Generator in the framework manager completes the offline model generation, the Offline Model Executor loads the model to the Runtime and integrates with the Ascend AI processor to perform inference calculation. In this process, the Offline Model Executor plays the main model execution function. Fig. 4.20 shows the process of loading an offline model. First, the process orchestrator, as the entry for interaction between applications and software stacks, provides the management capability for the execution process of inference tasks, divides the process that needs to be completed in the entire offline model into the engine of each execution phase and invokes the loading interface of the AI model manager to initialize the process of the device and load the offline model. Then, the Offline Model Executor is started to load the offline model, deserialize the offline model file, decode the executable file, invoke the storage interface of the execution environment to apply for memory, and copy the weight of the operator in the model to the memory. In addition, resources such as the model execution handle, execution flow, and event of the Runtime are applied for, and resources such as execution flows are bound to the corresponding models one by one. An execution handle executes a neural network computing graph. An execution handle can have multiple execution flows. Different execution flows contain AI Core or AI CPU computing tasks. A task is performed by a kernel function on AI CPU or AI Core, and events refer to the synchronization operations between different execution flows.

After a model is calculated, all operators in the offline model need to be traversed and the task information is updated. The Offline Model Executor invokes the running manager interface to deliver the task to the TS. Then, the Offline Model Executor returns the loading end information to the AI model manager, then, the process orchestrator sets the output result callback function to obtain the execution result. So far, the offline executor completes the loading process of the offline model, and the next step can directly perform inference calculation. This loading process is equivalent to adapting the model to the Ascend AI processor. The hardware resource and the operator in the offline model are planned in a coordinated manner. In this way, the offline model is executed in an orderly manner in subsequent execution, and the prespeed capability is provided for inference calculation.

4.2.6.4: Offline model inference

After the offline model is loaded, the inference function of the model can be implemented. Because the offline model is essentially a neural network model, corresponding functions, such as image recognition, are performed in the inference process. During the generation and loading of the offline model, no specific data to be processed is used. Only the software stack is used to construct, orchestrate, optimize, encapsulate, and perform hardware adaptation for the operator and calculation process in the model. In the specific inference execution process, the specific input data is read to drive the execution and output the result.

Fig. 4.21 shows the offline model inference process. When an application needs to process data, it prepares data to be processed. The process orchestrator invokes the processing interface of the AI model manager to inject data into the Offline Model Executor. Then, the Offline Model Executor invokes the execution flow (rtModelExecute) interface of the Runtime to deliver multiple inference tasks in the execution flow to the TS. The TS splits the task into task blocks and delivers them to the AI Core or AI CPU for execution. After the tasks are completed, the TS returns the result. The task scheduler traverses the tasks in the execution flow, cyclically transmits the task blocks, and returns the execution result. After all the tasks are complete, the TS returns the result to the memory of the Runtime. When the task of multiple executions flows is calculated, the operator synchronization between multiple streams needs to be performed through the event record and event waiting interface in the Runtime, and the operator calculation is completed in an orderly manner. After all operators in a stream are invoked, the Offline Model Executor synchronizes the execution completion of all execution flows through the execution flow synchronization (rtStreamSychronize) interface of the Runtime. After all related execution flow tasks are completed, the final results of all result generation models are integrated. In this case, the Offline Model Executor notifies the AI model manager that the stream has been executed. Finally, the AI model manager invokes the preset output callback function to return the result of the offline model reasoning execution to the process orchestrator, which then sends the result to the application.