Std deviation for dataset is too low

[meganset]

The method for calculating a running standard deviation is not giving the correct answer (in utils.py)

e.g. for one big batch:

>>> import torch
>>> import torchvision
>>> import torchvision.transforms as transforms
>>> t=transforms.Compose([transforms.ToTensor()])
>>> s=torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=t)

>>> d=torch.utils.data.DataLoader(s, batch_size=50000, shuffle=False)
>>> for x,y in d:
...  for i in range(3):
...   print(i, x[:,i,:,:].mean(), x[:,i,:,:].std())
... 
0 tensor(0.4914) tensor(0.2470)
1 tensor(0.4822) tensor(0.2435)
2 tensor(0.4465) tensor(0.2616)

This means the std deviations used to normalize the data set are a little too low:

transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

Thanks for the great set of models for CIFAR10