The number of CPUs assigned to VM does not improve network performance and in some cases may hurt it due to increased concurrency in the guest.Here is the short summary of things to check in order to improve network performance:

1. Whenever possible use virtio network adapter, otherwise use one of Intel PRO/1000 adapters;
2. Use bridged attachment instead of NAT;
3. Make sure segmentation offloading is enabled in the guest OS. Usually it will be enabled by default. You can check and modify offloading settings using ethtoolcommand in Linux guests.

https://www.virtualbox.org/manual/ch06.html