100GB/s vs 25GB/s, make a difference?

drumcircle
Rising Star

Based on this chart it appears that the peak transfer rate for the latest DDR4 memory to CPU is 25gb/s.

On a multi-node Exasol cluster, recommended networking bandwidth is 25gb/s which matches DDR4/3200 speed.

I presume that shuffling data between servers should be as fast as possible, 200GB/s is better than 100 GB/s is better than 25GB/s.  Is this true in practice?  For queries with lots of data shuffling, are we going to see 1:1 improvements in performance as intra-cluster communications trend toward 0ms?

Practically, are 100gb/s networking cards which can be picked up for $450 each (used) worth installing?  These are typically InfiniBand cards that require special switches.

What are the data transfer rates for DDR, DDR2, DDR3 and DDR4? - Transcend Information, Inc. (transc...

RAM Type

Speed

Peak Transfer Rate*

DDR

266

2.1 GB/s

333

2.7GB/s

400

3.2 GB/s

DDR2

533

4.27 GB/s

667

5.33 GB/s

800

6.4 GB/s

DDR3

1066

8.5 GB/s

1333

10.6 GB/s

1600

12.8 GB/s

1866

14.9 GB/s

DDR4

2133

17 GB/s

2400

19.2 GB/s

2666

21.3 GB/s

3200

25.6 GB/s

 

 

2 REPLIES 2

mwellbro
Xpert

Hi again,

actually I wanted to post a few tidbits more on a very rudimentary test I did but the details sadly didn´t survive the crash of my sql client - however, I can at least report that we can´t use g4dn.8xlarge for testing:

mwellbro_0-1628018955677.png

For the actual attempt at putting the NICs under some pressure I went with the following Statements:

CREATE TABLE FACT_1(
my_pk decimal(15,0),
unique_text CHAR(32),
some_val decimal(18,2)
);

CREATE TABLE FACT_2(
my_pk decimal(15,0),
unique_text CHAR(32),
some_val decimal(18,2)
);

INSERT INTO FACT_1
WITH base(rid) AS (SELECT ROW_NUMBER() OVER() AS rid FROM MULTIPLIER a CROSS JOIN MULTIPLIER CROSS JOIN MULTIPLIER CROSS JOIN MULTIPLIER )
 SELECT rid, HASH_MD5(rid),CAST(rand(0,1000000000) AS decimal(18,2)) FROM base;

INSERT INTO FACT_2
WITH base(rid) AS (SELECT ROW_NUMBER() OVER() AS rid FROM MULTIPLIER a CROSS JOIN MULTIPLIER CROSS JOIN MULTIPLIER CROSS JOIN MULTIPLIER )
 SELECT rid, HASH_MD5(rid),CAST(rand(0,1000000000) AS decimal(18,2)) FROM base;

Both FACT_* tables then contained 100.000.000 rows each which roughly translated to a MEM_SIZE of 3GiB ( not tiny but not large enough to go over the 12 GiB DBRAM available ).

Query used for checking:

SELECT sum(a.some_val),sum(b.some_val) FROM FACT_1 a 
INNER JOIN FACT_2 b ON a.my_pk=b.my_pk;

Due to the table size and the absence of a distribution key we get a GLOBAL JOIN - tested this with a 2 node m5.8xlarge and m5.16xlarge cluster and didn´t see much difference in exec times or NET value in the corresponding PROFILE.

This is all still very crude, will post something more sophisticated once I´ll get around to it 😉

Cheers,
Malte

mwellbro
Xpert

interesting question...I think the most reliable answer could be provided with an actual test, and in order for you to not pile up on NICs you otherwise won´t be using, maybe hacking it out in the cloud might be a way to test this ?

m5.8xlarge => 10Gbs
m5.16xlarge => 20Gbs
g4dn.8xlarge => 50Gbs
src: https://aws.amazon.com/ec2/instance-types/

Might be a challenge to get this "ceteris paribus", but there might be options as far as I remember ( and come to think of it, g4dn.8xlarge isn´t actually supported as it isn´t listed in the exa cloud tools - haven´t tried it...even more interesting 😊  )

There´s probably also the topology of your cluster to be considered and just what kind of size we are "data shuffling" - I think next month my personal budget might allow for some testing in this area, just not sure how "real world" my setup is going to end up 😉


If you try hard to keep your processing on the local side of things ( as normally would be advised ) then your gains from the faster inter-node speeds is probably not as great as when your model requires (for whatever reason) a large amount of global operations.

Just my 2 cents, looking forward to all additional thoughts and opinions !

Cheers,
Malte