AWS bandwidth fibs

I'm currently running some benchmarks at work to test message throughput and latency for a specific set of scenarios. The basic architecture is producer → processor → consumer with a server dedicated to each.

The throughput is likely to be pretty high and we need reasonable spec machines (8 cores minimum) with a 10Gbit network hooking them together. Since I don't have anything like that available at home or in the office, I'm using the company AWS account to rent some.

AWS has plenty of machine sizes available, but 8 or 16 cores only promises network bandwidth "up to" 10Gbits, so I have to step up to 32 cores to get the guarantee. Not ideal but I guess we'll have to pay for cores that we're not going to use.

After a day and a half of running these tests, something seems off, and I don't think it's the test setup itself. We should be getting better throughput than this. On a hunch I decide to test the raw network performance directly with the wonderfully useful iperf3. It appears that producer → processor and processor → consumer are only operating at 5Gbits. Hmm. But somehow the producer and consumer can quite happily talk to each other at 10Gbits.

I can't find any reason for this; I end up creating new instances for the processor VM until I get one that has a 10Gbit network connection.

Bad AWS.