Fully managed cloud. 30-day FREE full featured trial. Start Now
cancel
Showing results for 
Search instead for 
Did you mean: 

100GB/s vs 25GB/s, make a difference?

drumcircle
Rising Star

Based on this chart it appears that the peak transfer rate for the latest DDR4 memory to CPU is 25gb/s.

On a multi-node Exasol cluster, recommended networking bandwidth is 25gb/s which matches DDR4/3200 speed.

I presume that shuffling data between servers should be as fast as possible, 200GB/s is better than 100 GB/s is better than 25GB/s.  Is this true in practice?  For queries with lots of data shuffling, are we going to see 1:1 improvements in performance as intra-cluster communications trend toward 0ms?

Practically, are 100gb/s networking cards which can be picked up for $450 each (used) worth installing?  These are typically InfiniBand cards that require special switches.

What are the data transfer rates for DDR, DDR2, DDR3 and DDR4? - Transcend Information, Inc. (transc...

RAM Type

Speed

Peak Transfer Rate*

DDR

266

2.1 GB/s

333

2.7GB/s

400

3.2 GB/s

DDR2

533

4.27 GB/s

667

5.33 GB/s

800

6.4 GB/s

DDR3

1066

8.5 GB/s

1333

10.6 GB/s

1600

12.8 GB/s

1866

14.9 GB/s

DDR4

2133

17 GB/s

2400

19.2 GB/s

2666

21.3 GB/s

3200

25.6 GB/s

 

 

2 REPLIES 2

mwellbro
Xpert

interesting question...I think the most reliable answer could be provided with an actual test, and in order for you to not pile up on NICs you otherwise won´t be using, maybe hacking it out in the cloud might be a way to test this ?

m5.8xlarge => 10Gbs
m5.16xlarge => 20Gbs
g4dn.8xlarge => 50Gbs
src: https://aws.amazon.com/ec2/instance-types/

Might be a challenge to get this "ceteris paribus", but there might be options as far as I remember ( and come to think of it, g4dn.8xlarge isn´t actually supported as it isn´t listed in the exa cloud tools - haven´t tried it...even more interesting 😊  )

There´s probably also the topology of your cluster to be considered and just what kind of size we are "data shuffling" - I think next month my personal budget might allow for some testing in this area, just not sure how "real world" my setup is going to end up 😉


If you try hard to keep your processing on the local side of things ( as normally would be advised ) then your gains from the faster inter-node speeds is probably not as great as when your model requires (for whatever reason) a large amount of global operations.

Just my 2 cents, looking forward to all additional thoughts and opinions !

Cheers,
Malte

mwellbro
Xpert

Hi again,

actually I wanted to post a few tidbits more on a very rudimentary test I did but the details sadly didn´t survive the crash of my sql client - however, I can at least report that we can´t use g4dn.8xlarge for testing:

mwellbro_0-1628018955677.png

For the actual attempt at putting the NICs under some pressure I went with the following Statements:

CREATE TABLE FACT_1(
my_pk decimal(15,0),
unique_text CHAR(32),
some_val decimal(18,2)
);

CREATE TABLE FACT_2(
my_pk decimal(15,0),
unique_text CHAR(32),
some_val decimal(18,2)
);

INSERT INTO FACT_1
WITH base(rid) AS (SELECT ROW_NUMBER() OVER() AS rid FROM MULTIPLIER a CROSS JOIN MULTIPLIER CROSS JOIN MULTIPLIER CROSS JOIN MULTIPLIER )
 SELECT rid, HASH_MD5(rid),CAST(rand(0,1000000000) AS decimal(18,2)) FROM base;

INSERT INTO FACT_2
WITH base(rid) AS (SELECT ROW_NUMBER() OVER() AS rid FROM MULTIPLIER a CROSS JOIN MULTIPLIER CROSS JOIN MULTIPLIER CROSS JOIN MULTIPLIER )
 SELECT rid, HASH_MD5(rid),CAST(rand(0,1000000000) AS decimal(18,2)) FROM base;

Both FACT_* tables then contained 100.000.000 rows each which roughly translated to a MEM_SIZE of 3GiB ( not tiny but not large enough to go over the 12 GiB DBRAM available ).

Query used for checking:

SELECT sum(a.some_val),sum(b.some_val) FROM FACT_1 a 
INNER JOIN FACT_2 b ON a.my_pk=b.my_pk;

Due to the table size and the absence of a distribution key we get a GLOBAL JOIN - tested this with a 2 node m5.8xlarge and m5.16xlarge cluster and didn´t see much difference in exec times or NET value in the corresponding PROFILE.

This is all still very crude, will post something more sophisticated once I´ll get around to it 😉

Cheers,
Malte