Huawei has launched a new AI storage model, the OceanStor A310, designed for the era of large AI models at GITEX GLOBAL 2023.
The solution provides optimal storage capability for basic model training, industry model training, and inference in segmented scenario models, unleashing new AI capabilities.
The launch demonstrates Huawei’s alignment with the latest trends of AI development in the era of large models. The OceanStor A310 deep learning data lake storage caters to different industries and scenarios in large AI model applications.
Peter Zhou President Data Storage Product Line Huawei
“With the OceanStor A310, we are bringing our customers and partners a cutting-edge AI storage solution to unleash new AI capabilities and create value for various industries. Visitors to our GITEX stand can witness how Huawei is committed to accelerating the intelligent era, leveraging a trusted, secure, and reliable OceanStor platform used by some of the leading businesses in the world.”
Enterprises face four significant challenges in the development and implementation of large model applications. First, data preparation takes a long time, with dispersed data sources and slow aggregation. Currently, preprocessing about a hundred terabytes of data takes around ten days.
Secondly, for multi-modal large models using massive text and images as training sets, the current loading speed for a large number of small files is less than 100MB/s, resulting in low efficiency in loading training sets. Third, frequent tuning of large model parameters and instability in the training platform leads to training interruptions approximately every two days on average. The Checkpoint mechanism is needed to recover training, and the fault recovery process takes more than a day.
Finally, the high threshold for large model implementation, complex system construction, difficult resource scheduling, and GPU resource utilization is usually below 40 percent. Huawei OceanStor A310’s deep learning data lake storage caters to different industries and scenarios in large model applications.
In the context of basic/industry large model scenarios, OceanStor A310 is a deep learning data lake storage solution that offers unlimited horizontal scalability and high performance for mixed workloads. It enables end-to-end massive data storage management for AI processes, including data collection, preprocessing, training, and inference. It possesses data analysis capabilities consistent with HPC and big data, allowing for the processing of data from similar sources.
Huawei is a Gold Sponsor of GITEX GLOBAL 2023 and the lead sponsor for the event’s cybersecurity program, Cyber Valley.
