Controlla NVLink in Linux

Si prega di installare i driver Nvidia seguendo la nostra guida Installare il driver Nvidia su Linux, prima di verificare il supporto NVLink nel sistema operativo. Inoltre, è necessario installare il toolkit CUDA per compilare i campioni di applicazioni. In questa piccola guida, abbiamo raccolto alcuni comandi utili che è possibile utilizzare.

Comandi base

Controlla la topologia fisica del tuo sistema. Questo comando mostra tutte le GPU e i loro interconnessioni:

nvidia-smi topo -m

Se vuoi visualizzare lo stato dei collegamenti, esegui il seguente comando:

nvidia-smi nvlink -s

Il comando visualizza la velocità di ogni collegamento o. È possibile visualizzare informazioni su una singola connessione GPU. Ogni GPU ha un ID, che può essere specificato con l'opzione -i. Ad esempio, visualizziamo le informazioni di connessione sulla prima GPU con ID 0:

nvidia-smi nvlink -i 0 -c

Senza questa opzione, verranno visualizzate informazioni su tutte le connessioni delle GPU :

nvidia-smi nvlink -c

Installa CUDA-samples

Un buon modo per testare la larghezza di banda è utilizzare i campioni di applicazioni di Nvidia. Il codice sorgente di questi campioni è postato su GitHub ed è disponibile per tutti. Procedere con il cloning del repository sul server:

git clone https://github.com/NVIDIA/cuda-samples.git

Cambia directory per il repository scaricato:

cd cuda-samples

Seleziona il ramo appropriato tramite il tag in base alla versione CUDA installata. Ad esempio, se hai CUDA 12.2:

git checkout tags/v12.2

Installa alcuni prerequisiti che saranno utilizzati nel processo di compilazione:

sudo apt -y install freeglut3-dev build-essential libx11-dev libxmu-dev libxi-dev libgl1-mesa-glx libglu1-mesa libglu1-mesa-dev libglfw3-dev libgles2-mesa-dev

Ora, puoi compilare qualsiasi campione. Vai alla directory Samples:

cd Samples

Dare una occhiata veloce al contenuto:

ls -la

total 40
    drwxrwxr-x 10 usergpu usergpu 4096 Sep 13 14:54 .
    drwxrwxr-x  6 usergpu usergpu 4096 Sep 13 14:54 ..
    drwxrwxr-x 55 usergpu usergpu 4096 Sep 13 14:54 0_Introduction
    drwxrwxr-x  6 usergpu usergpu 4096 Sep 13 14:54 1_Utilities
    drwxrwxr-x 36 usergpu usergpu 4096 Sep 13 14:54 2_Concepts_and_Techniques
    drwxrwxr-x 25 usergpu usergpu 4096 Sep 13 14:54 3_CUDA_Features
    drwxrwxr-x 41 usergpu usergpu 4096 Sep 13 14:54 4_CUDA_Libraries
    drwxrwxr-x 52 usergpu usergpu 4096 Sep 13 14:54 5_Domain_Specific
    drwxrwxr-x  6 usergpu usergpu 4096 Sep 13 14:54 6_Performance
    drwxrwxr-x 11 usergpu usergpu 4096 Sep 13 14:54 7_libNVVM

Proviamo la larghezza di banda della GPU. Cambia la directory:

cd 1_Utilities/bandwidthTest

Compila l'app:

make

Esegui i test

Inizia i test eseguendo l'applicazione usando il suo nome:

./bandwidthTest

L'output potrebbe apparire così:

[CUDA Bandwidth Test] - Starting...
    Running on...
    
     Device 0: NVIDIA RTX A6000
     Quick Mode
    
     Host to Device Bandwidth, 1 Device(s)
     PINNED Memory Transfers
       Transfer Size (Bytes)        Bandwidth(GB/s)
       32000000                     6.0
    
     Device to Host Bandwidth, 1 Device(s)
     PINNED Memory Transfers
       Transfer Size (Bytes)        Bandwidth(GB/s)
       32000000                     6.6
    
     Device to Device Bandwidth, 1 Device(s)
     PINNED Memory Transfers
       Transfer Size (Bytes)        Bandwidth(GB/s)
       32000000                     569.2
    
    Result = PASS
    
    NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

In alternativa, puoi compilare e avviare il p2pBandwidthLatencyTest:

cd 5_Domain_Specific/p2pBandwidthLatencyTest

make

./p2pBandwidthLatencyTest

Questa applicazione ti mostrerà informazioni dettagliate sulla larghezza di banda della tua GPU in modalità P2P. Esempio di output:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
    Device: 0, NVIDIA RTX A6000, pciBusID: 3, pciDeviceID: 0, pciDomainID:0
    Device: 1, NVIDIA RTX A6000, pciBusID: 4, pciDeviceID: 0, pciDomainID:0
    Device=0 CAN Access Peer Device=1
    Device=1 CAN Access Peer Device=0
    
    ***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
    So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
    
    P2P Connectivity Matrix
         D\D     0     1
         0       1     1
         1       1     1
    Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
       D\D     0      1 
         0 590.51   6.04 
         1   6.02 590.51 
    Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
       D\D     0      1 
         0 589.40  52.75 
         1  52.88 592.53 
    Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
       D\D     0      1 
         0 593.88   8.55 
         1   8.55 595.32 
    Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
       D\D     0      1 
         0 595.69 101.68 
         1 101.97 595.69 
    P2P=Disabled Latency Matrix (us)
       GPU     0      1 
         0   1.61  28.66 
         1  18.49   1.53 
    
       CPU     0      1 
         0   2.27   6.06 
         1   6.12   2.23 
    P2P=Enabled Latency (P2P Writes) Matrix (us)
       GPU     0      1 
         0   1.62   1.27 
         1   1.17   1.55 
    
       CPU     0      1 
         0   2.27   1.91 
         1   1.90   2.34 
    
    NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

In caso di una configurazione con più GPU, potrebbe apparire così:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
    Device: 0, NVIDIA H100 PCIe, pciBusID: 30, pciDeviceID: 0, pciDomainID:0
    Device: 1, NVIDIA H100 PCIe, pciBusID: 3f, pciDeviceID: 0, pciDomainID:0
    Device: 2, NVIDIA H100 PCIe, pciBusID: 40, pciDeviceID: 0, pciDomainID:0
    Device: 3, NVIDIA H100 PCIe, pciBusID: 41, pciDeviceID: 0, pciDomainID:0
    Device: 4, NVIDIA H100 PCIe, pciBusID: b0, pciDeviceID: 0, pciDomainID:0
    Device: 5, NVIDIA H100 PCIe, pciBusID: b1, pciDeviceID: 0, pciDomainID:0
    Device: 6, NVIDIA H100 PCIe, pciBusID: c2, pciDeviceID: 0, pciDomainID:0
    Device: 7, NVIDIA H100 PCIe, pciBusID: c3, pciDeviceID: 0, pciDomainID:0
    Device=0 CAN Access Peer Device=1
    Device=0 CAN Access Peer Device=2
    Device=0 CAN Access Peer Device=3
    Device=0 CAN Access Peer Device=4
    Device=0 CAN Access Peer Device=5
    Device=0 CAN Access Peer Device=6
    Device=0 CAN Access Peer Device=7
    Device=1 CAN Access Peer Device=0
    Device=1 CAN Access Peer Device=2
    Device=1 CAN Access Peer Device=3
    Device=1 CAN Access Peer Device=4
    Device=1 CAN Access Peer Device=5
    Device=1 CAN Access Peer Device=6
    Device=1 CAN Access Peer Device=7
    Device=2 CAN Access Peer Device=0
    Device=2 CAN Access Peer Device=1
    Device=2 CAN Access Peer Device=3
    Device=2 CAN Access Peer Device=4
    Device=2 CAN Access Peer Device=5
    Device=2 CAN Access Peer Device=6
    Device=2 CAN Access Peer Device=7
    Device=3 CAN Access Peer Device=0
    Device=3 CAN Access Peer Device=1
    Device=3 CAN Access Peer Device=2
    Device=3 CAN Access Peer Device=4
    Device=3 CAN Access Peer Device=5
    Device=3 CAN Access Peer Device=6
    Device=3 CAN Access Peer Device=7
    Device=4 CAN Access Peer Device=0
    Device=4 CAN Access Peer Device=1
    Device=4 CAN Access Peer Device=2
    Device=4 CAN Access Peer Device=3
    Device=4 CAN Access Peer Device=5
    Device=4 CAN Access Peer Device=6
    Device=4 CAN Access Peer Device=7
    Device=5 CAN Access Peer Device=0
    Device=5 CAN Access Peer Device=1
    Device=5 CAN Access Peer Device=2
    Device=5 CAN Access Peer Device=3
    Device=5 CAN Access Peer Device=4
    Device=5 CAN Access Peer Device=6
    Device=5 CAN Access Peer Device=7
    Device=6 CAN Access Peer Device=0
    Device=6 CAN Access Peer Device=1
    Device=6 CAN Access Peer Device=2
    Device=6 CAN Access Peer Device=3
    Device=6 CAN Access Peer Device=4
    Device=6 CAN Access Peer Device=5
    Device=6 CAN Access Peer Device=7
    Device=7 CAN Access Peer Device=0
    Device=7 CAN Access Peer Device=1
    Device=7 CAN Access Peer Device=2
    Device=7 CAN Access Peer Device=3
    Device=7 CAN Access Peer Device=4
    Device=7 CAN Access Peer Device=5
    Device=7 CAN Access Peer Device=6
    
    ***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
    So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
    
    P2P Connectivity Matrix
         D\D     0     1     2     3     4     5     6     7
         0       1     1     1     1     1     1     1     1
         1       1     1     1     1     1     1     1     1
         2       1     1     1     1     1     1     1     1
         3       1     1     1     1     1     1     1     1
         4       1     1     1     1     1     1     1     1
         5       1     1     1     1     1     1     1     1
         6       1     1     1     1     1     1     1     1
         7       1     1     1     1     1     1     1     1
    Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
       D\D     0      1      2      3      4      5      6      7
         0 1629.83  38.43  38.39  37.66  38.51  38.19  38.09  37.92
         1  38.22 1637.04  35.52  35.59  38.15  38.38  38.08  37.55
         2  37.76  35.62 1635.32  35.45  38.59  38.21  38.77  37.94
         3  37.88  35.50  35.60 1639.45  38.49  37.43  38.72  38.49
         4  36.87  37.03  37.00  36.90 1635.86  34.48  38.06  37.22
         5  37.27  37.06  36.92  37.06  34.51 1636.18  37.80  37.50
         6  37.05  36.95  37.45  37.15  37.51  37.96 1630.79  34.94
         7  36.98  36.91  36.95  36.87  37.83  38.02  34.73 1633.35
    
    Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
       D\D     0      1      2      3      4      5      6      7
         0 1635.22  34.42  33.84 256.54  27.74  28.68  28.00  28.41
         1  34.66 1636.93 256.16  17.97  71.58  71.64  71.65  71.61
         2  34.78 256.81 1655.79  30.29  70.34  70.42  70.37  70.33
         3 256.65  30.65  70.67 1654.53  70.66  70.69  70.70  70.73
         4  28.26  30.80  69.99  70.04 1630.36 256.45  69.97  70.02
         5  28.10  31.08  71.60  71.59 256.47 1654.31  71.62  71.54
         6  28.37  30.96  70.99  70.93  70.91  70.96 1632.12 257.11
         7  27.66  30.87  70.30  70.40  70.30  70.39 256.72 1649.57
    
    Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
       D\D     0      1      2      3      4      5      6      7
         0 1673.16  51.88  51.95  51.76  51.61  51.44  52.07  51.30
         1  52.04 1676.28  39.06  39.21  51.62  51.62  51.98  51.36
         2  52.11  39.27 1674.62  39.16  51.42  51.21  51.72  51.71
         3  51.74  39.70  39.22 1672.77  51.50  51.27  51.70  51.24
         4  52.14  52.41  51.38  52.14 1671.54  38.81  46.76  45.72
         5  51.82  52.65  52.30  51.67  38.57 1676.33  46.90  45.96
         6  52.92  52.66  53.02  52.68  46.23  46.31 1672.74  38.91
         7  52.61  52.74  52.79  52.64  45.90  46.35  39.07 1673.16
    
    Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
       D\D     0      1      2      3      4      5      6      7
         0 1670.31  52.41 140.69 508.68 139.85 141.88 141.71 140.55
         1 141.69 1673.30 509.23 141.22 139.91 143.28 141.71 140.61
         2 140.64 508.90 1669.67 140.68 139.93 140.61 140.67 140.50
         3 509.14 141.36 140.61 1682.65 139.93 141.45 141.45 140.67
         4 140.01 140.03 140.07 139.94 1670.68 508.37 140.01 139.90
         5 141.92 143.17 140.50 141.19 508.92 1670.73 141.72 140.52
         6 141.72 141.72 140.60 141.31 139.66 141.85 1671.51 510.03
         7 140.62 140.71 140.66 140.63 140.02 140.72 509.77 1668.28
    
    P2P=Disabled Latency Matrix (us)
       GPU     0      1      2      3      4      5      6      7
         0   2.35  17.23  17.13  13.38  12.86  21.15  21.39  21.12
         1  17.54   2.32  12.95  13.78  21.05  21.23  21.31  21.37
         2  16.85  14.83   2.35  16.07  12.71  12.80  21.23  12.79
         3  14.98  16.06  14.64   2.41  13.35  12.81  13.60  21.36
         4  21.31  21.31  20.49  21.32   2.62  12.33  12.66  12.98
         5  20.36  21.22  20.17  12.79  16.74   2.58  12.41  12.93
         6  17.51  12.84  12.79  12.70  17.63  18.78   2.36  13.69
         7  21.23  12.71  19.41  21.09  14.69  13.79  15.52   2.59
    
    CPU     0      1      2      3      4      5      6      7
         0   1.73   4.99   4.88   4.85   5.17   5.18   5.18   5.33
         1   5.04   1.71   4.74   4.82   5.04   5.14   5.10   5.19
         2   4.86   4.75   1.66   4.78   5.08   5.09   5.11   5.17
         3   4.80   4.72   4.73   1.63   5.09   5.11   5.06   5.10
         4   5.07   5.00   5.03   4.96   1.77   5.33   5.34   5.38
         5   5.12   4.94   5.00   4.96   5.31   1.77   5.38   5.41
         6   5.09   4.97   5.09   5.01   5.35   5.39   1.80   5.42
         7   5.18   5.09   5.02   5.00   5.39   5.40   5.40   1.76
    
    P2P=Enabled Latency (P2P Writes) Matrix (us)
       GPU     0      1      2      3      4      5      6      7
         0   2.33   2.15   2.11   2.76   2.07   2.11   2.07   2.12
         1   2.07   2.30   2.77   2.07   2.12   2.06   2.06   2.10
         2   2.09   2.75   2.34   2.12   2.09   2.08   2.08   2.12
         3   2.78   2.10   2.13   2.40   2.13   2.14   2.14   2.13
         4   2.18   2.23   2.23   2.17   2.59   2.82   2.15   2.16
         5   2.15   2.17   2.15   2.20   2.82   2.56   2.17   2.16
         6   2.13   2.18   2.21   2.17   2.15   2.17   2.36   2.85
         7   2.19   2.21   2.19   2.22   2.19   2.19   2.86   2.61
    
       CPU     0      1      2      3      4      5      6      7
         0   1.78   1.32   1.29   1.40   1.33   1.34   1.34   1.33
         1   1.32   1.69   1.34   1.35   1.35   1.34   1.40   1.33
         2   1.38   1.37   1.73   1.36   1.36   1.35   1.35   1.34
         3   1.34   1.42   1.35   1.66   1.34   1.34   1.35   1.33
         4   1.53   1.41   1.40   1.40   1.77   1.43   1.48   1.47
         5   1.46   1.43   1.43   1.42   1.47   1.84   1.51   1.56
         6   1.53   1.45   1.45   1.45   1.45   1.44   1.85   1.47
         7   1.54   1.47   1.47   1.47   1.45   1.44   1.50   1.84
    
    NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.