Performance test

Updated at：2025-11-03

Test data: 107 GB dataset, 1,000,000 images, with an average of 110 KB per image

Test environment: Baidu AI Cloud bcc.c5.c8m16, 8 CPUs, 16 GB memory, 3 Gbps intranet bandwidth

Dataset type: Example of building BosIterableDataset via from_prefix

Test results:

	batch_size	num_workers	Dataset type	Construction method	Result
bostorchconnector	256	8	BosIterableDataset	from_prefix	2785 img/s
bosfs	256	8	BosIterableDataset	from_prefix	48 img/s

Test code:

                Bash
                
            

                def transform(data):
    data.read()
    return data.key
    
@time_it
def test_bos():
    config = BosClientConfig()
    BOS_URI = "bos://bos-torch/img_1M/"
    
    dataset = BosIterableDataset.from_prefix(BOS_URI, endpoint="http://su.bcebos.com", transform=transform, bos_client_config=config, enable_sharding=True)
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=256, num_workers=8)
    for step, key in enumerate(dataloader):
        print(key)
        pass
            

Configuring BOS Connector for Pytorch

BOS-CLI Command-Line Tool (Old Version)