Ross,
Thanks, I will look for bios updates. Firmware aside, is there any
configuration in the bios that would affect this?
Thanks,
Steve
On Aug 12, 2016 12:29 AM, "Ross Walker" <ross.rosswalker.co.uk> wrote:
> Hi Steven,
>
> Ah I thought you meant you had 4 GPUs as in 2 K80s rather than a single
> K80 card that contains 2 GPUs.
>
> Either way this shows your hardware is incorrectly configured / has a
> buggy bios. Who makes it? You probably need to go back to them and get an
> updated bios that properly handles peer to peer communication.
>
> You could also check the motherboard manufacturer and see if they have an
> up to date bios that fixes this bug.
>
> All those entries reported by lspci should have a minus after them if
> things are correct in the bios.
>
> All the best
> Ross
>
> On Aug 11, 2016, at 9:21 PM, Steven Ford <sford123.ibbr.umd.edu> wrote:
>
> Ross,
>
> The output of lspci -d "10b5:*" -vvv | grep ACSCtl is:
>
> ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+
> EgressCtrl- DirectTrans-
> ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+
> EgressCtrl- DirectTrans-
>
>
> With CUDA_VISIBLE_DEVICES unset:
>
> [./simpleP2P] - Starting...
> Checking for multiple GPUs...
> CUDA-capable device count: 2
> > GPU0 = " Tesla K80" IS capable of Peer-to-Peer (P2P)
> > GPU1 = " Tesla K80" IS capable of Peer-to-Peer (P2P)
>
> Checking GPU(s) for support of peer to peer memory access...
> > Peer access from Tesla K80 (GPU0) -> Tesla K80 (GPU1) : Yes
> > Peer access from Tesla K80 (GPU1) -> Tesla K80 (GPU0) : Yes
> Enabling peer access between GPU0 and GPU1...
> Checking GPU0 and GPU1 for UVA capabilities...
> > Tesla K80 (GPU0) supports UVA: Yes
> > Tesla K80 (GPU1) supports UVA: Yes
> Both GPUs can support UVA, enabling...
> Allocating buffers (64MB on GPU0, GPU1 and CPU Host)...
> Creating event handles...
> cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 1.11GB/s
> Preparing host buffer and memcpy to GPU0...
> Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
> Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
> Copy data back to host from GPU0 and verify results...
> Verification error . element 0: val = nan, ref = 0.000000
> Verification error . element 1: val = nan, ref = 4.000000
> Verification error . element 2: val = nan, ref = 8.000000
> Verification error . element 3: val = nan, ref = 12.000000
> Verification error . element 4: val = nan, ref = 16.000000
> Verification error . element 5: val = nan, ref = 20.000000
> Verification error . element 6: val = nan, ref = 24.000000
> Verification error . element 7: val = nan, ref = 28.000000
> Verification error . element 8: val = nan, ref = 32.000000
> Verification error . element 9: val = nan, ref = 36.000000
> Verification error . element 10: val = nan, ref = 40.000000
> Verification error . element 11: val = nan, ref = 44.000000
> Disabling peer access...
> Shutting down...
> Test failed!
>
> With CUDA_VISIBLE_DEVICES=0,1
>
> [./simpleP2P] - Starting...
> Checking for multiple GPUs...
> CUDA-capable device count: 2
> > GPU0 = " Tesla K80" IS capable of Peer-to-Peer (P2P)
> > GPU1 = " Tesla K80" IS capable of Peer-to-Peer (P2P)
>
> Checking GPU(s) for support of peer to peer memory access...
> > Peer access from Tesla K80 (GPU0) -> Tesla K80 (GPU1) : Yes
> > Peer access from Tesla K80 (GPU1) -> Tesla K80 (GPU0) : Yes
> Enabling peer access between GPU0 and GPU1...
> Checking GPU0 and GPU1 for UVA capabilities...
> > Tesla K80 (GPU0) supports UVA: Yes
> > Tesla K80 (GPU1) supports UVA: Yes
> Both GPUs can support UVA, enabling...
> Allocating buffers (64MB on GPU0, GPU1 and CPU Host)...
> Creating event handles...
> cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 1.11GB/s
> Preparing host buffer and memcpy to GPU0...
> Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
> Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
> Copy data back to host from GPU0 and verify results...
> Verification error . element 0: val = nan, ref = 0.000000
> Verification error . element 1: val = nan, ref = 4.000000
> Verification error . element 2: val = nan, ref = 8.000000
> Verification error . element 3: val = nan, ref = 12.000000
> Verification error . element 4: val = nan, ref = 16.000000
> Verification error . element 5: val = nan, ref = 20.000000
> Verification error . element 6: val = nan, ref = 24.000000
> Verification error . element 7: val = nan, ref = 28.000000
> Verification error . element 8: val = nan, ref = 32.000000
> Verification error . element 9: val = nan, ref = 36.000000
> Verification error . element 10: val = nan, ref = 40.000000
> Verification error . element 11: val = nan, ref = 44.000000
> Disabling peer access...
> Shutting down...
> Test failed!
>
>
> With CUDA_VISIBLE_DEVICES=2,3
>
> [./simpleP2P] - Starting...
> Checking for multiple GPUs...
> CUDA error at simpleP2P.cu:63 code=38(cudaErrorNoDevice)
> "cudaGetDeviceCount(&gpu_n)"
>
>
> and with CUDA_VISIBLE_DEVICES=0,2
>
> CUDA-capable device count: 1
> Two or more GPUs with SM 2.0 or higher capability are required for
> ./simpleP2P.
> Waiving test.
>
>
> I'm guessing the last two test fail because I have only one card with two
> K80 GPUs on it, so no devices 2 or 3. Seems like something's awry with the
> peer to peer communication between 0 and 1. Is it possible for them to be
> on different PCIe domains even though they are on the same physical card?
>
> This makes me wonder: If each PCIe slot is connected to one CPU, should
> this system either use only one CPU or have another K80 in the other PCIe
> slot that's connected to the other CPU?
>
> If it helps, nvidia-smi topo -m shows:
>
> GPU0 GPU1 CPU Affinity
> GPU0 X PIX 0-7,16-23
> GPU1 PIX X 0-7,16-23
>
>
> Thanks again,
>
> Steve
>
> On Thu, Aug 11, 2016 at 11:17 PM, Ross Walker <ross.rosswalker.co.uk>
> wrote:
>
>> Hi Steve,
>>
>> I suspect your hardware is misconfigured. Can you run a couple of tests
>> please.
>>
>> With CUDA_VISIBLE_DEVICES unset
>>
>> 1) As root run: lspci -d "10b5:*" -vvv | grep ACSCtl
>>
>> and post the output here.
>>
>> 2) Compile the CUDA samples installed as part of CUDA 7.5 and then run
>> the following:
>>
>> unset CUDA_VISIBLE_DEVICES
>> ./simpleP2P
>>
>> export CUDA_VISIBLE_DEVICES=0,1
>> ./simpleP2P
>>
>> export CUDA_VISIBLE_DEVICES=2,3
>> ./simpleP2P
>>
>> export CUDA_VISIBLE_DEVICES=0,2
>> ./simpleP2P
>>
>> And post the results here.
>>
>> My suspicion is that your two K80s are on different PCI-E domains
>> connected to different CPU sockets BUT your bios is misconfigured such that
>> it is incorrectly reporting that the two K80s can talk to each other via
>> P2P. Thus the first two simpleP2P runs above should pass. The last one will
>> likely report that P2P is possible but then the bandwidth will be very low
>> and it will ultimately fail the test because the array received by GPU 2
>> will be garbage.
>>
>> If my suspicions are correct you would find the following behavior with
>> AMBER
>>
>> 4 x 1 GPU runs, one on each GPU would be fine.
>> (1 or 2) x 2 GPU runs will be fine if you use GPUS 0,1 and 2,3 but will
>> fail if you were to use 0,2 - 0,3 - 1,2 or 1,3
>> 1 x 4 GPU runs will fail unless you restrict it to GPUs 0,1 or 2,3 and
>> thus overload the GPUs.
>>
>> Ps. nvidia-smi reporting 2 threads per mpi task is not an issue - it to
>> be expected.
>>
>> All the best
>> Ross
>>
>> On Aug 11, 2016, at 7:54 PM, Steven Ford <sford123.ibbr.umd.edu> wrote:
>>
>> Hello,
>>
>> I'm still trying to figure out why the MPI CUDA tests are failing.
>>
>> If I run tests with DO_PARALLEL="mpirun -np 4" and limit
>> CUDA_VISIBLE_DEVICES to only 0 or 1, all tests pass. I get the same
>> behavior with OpenMPI 1.8, 1.10, 2.0 and mpich 3.1.
>>
>> I ran gpuP2PCheck just in case communication between the GPUs was the
>> problem. It confirms that communication is working:
>>
>> CUDA-capable device count: 2
>> GPU0 " Tesla K80"
>> GPU1 " Tesla K80"
>>
>> Two way peer access between:
>> GPU0 and GPU1: YES
>>
>> If it's of any use, here is the output of nvidia-smi -q:
>>
>> ==============NVSMI LOG==============
>>
>> Timestamp : Thu Aug 11 22:42:34 2016
>> Driver Version : 352.93
>>
>> Attached GPUs : 2
>> GPU 0000:05:00.0
>> Product Name : Tesla K80
>> Product Brand : Tesla
>> Display Mode : Disabled
>> Display Active : Disabled
>> Persistence Mode : Disabled
>> Accounting Mode : Disabled
>> Accounting Mode Buffer Size : 1920
>> Driver Model
>> Current : N/A
>> Pending : N/A
>> Serial Number : 0325015055313
>> GPU UUID : GPU-a65eaa77-8871-ded5-b6ee-52
>> 68404192f1
>> Minor Number : 0
>> VBIOS Version : 80.21.1B.00.01
>> MultiGPU Board : Yes
>> Board ID : 0x300
>> Inforom Version
>> Image Version : 2080.0200.00.04
>> OEM Object : 1.1
>> ECC Object : 3.0
>> Power Management Object : N/A
>> GPU Operation Mode
>> Current : N/A
>> Pending : N/A
>> PCI
>> Bus : 0x05
>> Device : 0x00
>> Domain : 0x0000
>> Device Id : 0x102D10DE
>> Bus Id : 0000:05:00.0
>> Sub System Id : 0x106C10DE
>> GPU Link Info
>> PCIe Generation
>> Max : 3
>> Current : 3
>> Link Width
>> Max : 16x
>> Current : 16x
>> Bridge Chip
>> Type : PLX
>> Firmware : 0xF0472900
>> Replays since reset : 0
>> Tx Throughput : N/A
>> Rx Throughput : N/A
>> Fan Speed : N/A
>> Performance State : P0
>> Clocks Throttle Reasons
>> Idle : Not Active
>> Applications Clocks Setting : Active
>> SW Power Cap : Not Active
>> HW Slowdown : Not Active
>> Unknown : Not Active
>> FB Memory Usage
>> Total : 12287 MiB
>> Used : 56 MiB
>> Free : 12231 MiB
>> BAR1 Memory Usage
>> Total : 16384 MiB
>> Used : 2 MiB
>> Free : 16382 MiB
>> Compute Mode : Default
>> Utilization
>> Gpu : 0 %
>> Memory : 0 %
>> Encoder : 0 %
>> Decoder : 0 %
>> Ecc Mode
>> Current : Disabled
>> Pending : Disabled
>> ECC Errors
>> Volatile
>> Single Bit
>> Device Memory : N/A
>> Register File : N/A
>> L1 Cache : N/A
>> L2 Cache : N/A
>> Texture Memory : N/A
>> Total : N/A
>> Double Bit
>> Device Memory : N/A
>> Register File : N/A
>> L1 Cache : N/A
>> L2 Cache : N/A
>> Texture Memory : N/A
>> Total : N/A
>> Aggregate
>> Single Bit
>> Device Memory : N/A
>> Register File : N/A
>> L1 Cache : N/A
>> L2 Cache : N/A
>> Texture Memory : N/A
>> Total : N/A
>> Double Bit
>> Device Memory : N/A
>> Register File : N/A
>> L1 Cache : N/A
>> L2 Cache : N/A
>> Texture Memory : N/A
>> Total : N/A
>> Retired Pages
>> Single Bit ECC : 0
>> Double Bit ECC : 0
>> Pending : No
>> Temperature
>> GPU Current Temp : 31 C
>> GPU Shutdown Temp : 93 C
>> GPU Slowdown Temp : 88 C
>> Power Readings
>> Power Management : Supported
>> Power Draw : 59.20 W
>> Power Limit : 149.00 W
>> Default Power Limit : 149.00 W
>> Enforced Power Limit : 149.00 W
>> Min Power Limit : 100.00 W
>> Max Power Limit : 175.00 W
>> Clocks
>> Graphics : 562 MHz
>> SM : 562 MHz
>> Memory : 2505 MHz
>> Applications Clocks
>> Graphics : 562 MHz
>> Memory : 2505 MHz
>> Default Applications Clocks
>> Graphics : 562 MHz
>> Memory : 2505 MHz
>> Max Clocks
>> Graphics : 875 MHz
>> SM : 875 MHz
>> Memory : 2505 MHz
>> Clock Policy
>> Auto Boost : On
>> Auto Boost Default : On
>> Processes : None
>>
>> GPU 0000:06:00.0
>> Product Name : Tesla K80
>> Product Brand : Tesla
>> Display Mode : Disabled
>> Display Active : Disabled
>> Persistence Mode : Disabled
>> Accounting Mode : Disabled
>> Accounting Mode Buffer Size : 1920
>> Driver Model
>> Current : N/A
>> Pending : N/A
>> Serial Number : 0325015055313
>> GPU UUID : GPU-21c2be1c-72a9-1b68-adab-45
>> 9d05dd7adc
>> Minor Number : 1
>> VBIOS Version : 80.21.1B.00.02
>> MultiGPU Board : Yes
>> Board ID : 0x300
>> Inforom Version
>> Image Version : 2080.0200.00.04
>> OEM Object : 1.1
>> ECC Object : 3.0
>> Power Management Object : N/A
>> GPU Operation Mode
>> Current : N/A
>> Pending : N/A
>> PCI
>> Bus : 0x06
>> Device : 0x00
>> Domain : 0x0000
>> Device Id : 0x102D10DE
>> Bus Id : 0000:06:00.0
>> Sub System Id : 0x106C10DE
>> GPU Link Info
>> PCIe Generation
>> Max : 3
>> Current : 3
>> Link Width
>> Max : 16x
>> Current : 16x
>> Bridge Chip
>> Type : PLX
>> Firmware : 0xF0472900
>> Replays since reset : 0
>> Tx Throughput : N/A
>> Rx Throughput : N/A
>> Fan Speed : N/A
>> Performance State : P0
>> Clocks Throttle Reasons
>> Idle : Not Active
>> Applications Clocks Setting : Active
>> SW Power Cap : Not Active
>> HW Slowdown : Not Active
>> Unknown : Not Active
>> FB Memory Usage
>> Total : 12287 MiB
>> Used : 56 MiB
>> Free : 12231 MiB
>> BAR1 Memory Usage
>> Total : 16384 MiB
>> Used : 2 MiB
>> Free : 16382 MiB
>> Compute Mode : Default
>> Utilization
>> Gpu : 0 %
>> Memory : 0 %
>> Encoder : 0 %
>> Decoder : 0 %
>> Ecc Mode
>> Current : Disabled
>> Pending : Disabled
>> ECC Errors
>> Volatile
>> Single Bit
>> Device Memory : N/A
>> Register File : N/A
>> L1 Cache : N/A
>> L2 Cache : N/A
>> Texture Memory : N/A
>> Total : N/A
>> Double Bit
>> Device Memory : N/A
>> Register File : N/A
>> L1 Cache : N/A
>> L2 Cache : N/A
>> Texture Memory : N/A
>> Total : N/A
>> Aggregate
>> Single Bit
>> Device Memory : N/A
>> Register File : N/A
>> L1 Cache : N/A
>> L2 Cache : N/A
>> Texture Memory : N/A
>> Total : N/A
>> Double Bit
>> Device Memory : N/A
>> Register File : N/A
>> L1 Cache : N/A
>> L2 Cache : N/A
>> Texture Memory : N/A
>> Total : N/A
>> Retired Pages
>> Single Bit ECC : 0
>> Double Bit ECC : 0
>> Pending : No
>> Temperature
>> GPU Current Temp : 24 C
>> GPU Shutdown Temp : 93 C
>> GPU Slowdown Temp : 88 C
>> Power Readings
>> Power Management : Supported
>> Power Draw : 70.89 W
>> Power Limit : 149.00 W
>> Default Power Limit : 149.00 W
>> Enforced Power Limit : 149.00 W
>> Min Power Limit : 100.00 W
>> Max Power Limit : 175.00 W
>> Clocks
>> Graphics : 562 MHz
>> SM : 562 MHz
>> Memory : 2505 MHz
>> Applications Clocks
>> Graphics : 562 MHz
>> Memory : 2505 MHz
>> Default Applications Clocks
>> Graphics : 562 MHz
>> Memory : 2505 MHz
>> Max Clocks
>> Graphics : 875 MHz
>> SM : 875 MHz
>> Memory : 2505 MHz
>> Clock Policy
>> Auto Boost : On
>> Auto Boost Default : On
>> Processes : None
>>
>>
>> If it matters, when I do the tests with DO_PARALLEL="mpirun -np 4", I see
>> that each process is running a thread on both GPUs. For example:
>>
>> # gpu pid type sm mem enc dec command
>> # Idx # C/G % % % % name
>> 0 30599 C 24 0 0 0 pmemd.cuda_DPFP
>> 0 30600 C 0 0 0 0 pmemd.cuda_DPFP
>> 0 30601 C 11 0 0 0 pmemd.cuda_DPFP
>> 0 30602 C 0 0 0 0 pmemd.cuda_DPFP
>> 1 30599 C 0 0 0 0 pmemd.cuda_DPFP
>> 1 30600 C 36 0 0 0 pmemd.cuda_DPFP
>> 1 30601 C 0 0 0 0 pmemd.cuda_DPFP
>> 1 30602 C 6 0 0 0 pmemd.cuda_DPFP
>>
>> Is that expected behavior?
>>
>> Has anybody else had any problems using K80s with MPI and CUDA? Or using
>> CentOS/RHEL 6?
>>
>> This machine does have dual CPUs, could that be a factor?
>>
>> I'm currently using AmberTools version 16.12 and Amber version 16.05.
>>
>> Any insight would be greatly appreciated.
>>
>> Thanks,
>>
>> Steve
>>
>>
>>
>> On Mon, Jul 25, 2016 at 3:06 PM, Steven Ford <sford123.ibbr.umd.edu>
>> wrote:
>>
>>> Ross,
>>>
>>> This is CentOS version 6.7 with kernel version
>>> 2.6.32-573.22.1.el6.x86_64.
>>>
>>> The output of nvidia-smi is:
>>>
>>> +------------------------------------------------------+
>>>
>>> | NVIDIA-SMI 352.79 Driver Version: 352.79 |
>>>
>>> |-------------------------------+----------------------+----
>>> ------------------+
>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>> Uncorr. ECC |
>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>> Compute M. |
>>> |===============================+======================+====
>>> ==================|
>>> | 0 Tesla K80 Off | 0000:05:00.0 Off |
>>> Off |
>>> | N/A 34C P0 59W / 149W | 56MiB / 12287MiB | 0%
>>> Default |
>>> +-------------------------------+----------------------+----
>>> ------------------+
>>> | 1 Tesla K80 Off | 0000:06:00.0 Off |
>>> Off |
>>> | N/A 27C P0 48W / 149W | 56MiB / 12287MiB | 0%
>>> Default |
>>> +-------------------------------+----------------------+----
>>> ------------------+
>>>
>>>
>>> +-----------------------------------------------------------
>>> ------------------+
>>> | Processes: GPU
>>> Memory |
>>> | GPU PID Type Process name Usage
>>> |
>>> |===========================================================
>>> ==================|
>>> | No running processes found
>>> |
>>> +-----------------------------------------------------------
>>> ------------------+
>>>
>>> The version of nvcc:
>>>
>>> nvcc: NVIDIA (R) Cuda compiler driver
>>> Copyright (c) 2005-2015 NVIDIA Corporation
>>> Built on Tue_Aug_11_14:27:32_CDT_2015
>>> Cuda compilation tools, release 7.5, V7.5.17
>>>
>>> I used the GNU compilers, version 4.4.7.
>>>
>>> I am using OpenMPI version 1.8.1-5.el6 from the CentOS repository. I
>>> have not tried any other MPI installation.
>>>
>>> Output of mpif90 --showme:
>>>
>>> gfortran -I/usr/include/openmpi-x86_64 -pthread -I/usr/lib64/openmpi/lib
>>> -Wl,-rpath -Wl,/usr/lib64/openmpi/lib -Wl,--enable-new-dtags
>>> -L/usr/lib64/openmpi/lib -lmpi_usempi -lmpi_mpifh -lmpi
>>>
>>>
>>> I set DO_PARALLEL to "mpirun -np 2"
>>>
>>> The parallel tests for the CPU were all successful.
>>>
>>> I had not run 'make clean' in between each step. I tried the tests again
>>> this morning after running 'make clean' and got the same result.
>>>
>>> I applied all patches this morning before testing again. I am using
>>> AmberTools 16.10 and Amber 16.04
>>>
>>>
>>> Thanks,
>>>
>>> Steve
>>>
>>> On Sat, Jul 23, 2016 at 6:32 PM, Ross Walker <ross.rosswalker.co.uk>
>>> wrote:
>>>
>>>> Hi Steven,
>>>>
>>>> This is a large number of very worrying failures. Something is
>>>> definitely very wrong here and I'd like to investigate further. Can you
>>>> give me some more details about your system please. This includes:
>>>>
>>>> The specifics of what version of Linux you are using.
>>>>
>>>> The output of nvidia-smi
>>>>
>>>> nvcc -V (might be lower case v to get version info).
>>>>
>>>> Did you use the GNU compilers or the Intel compilers and in either case
>>>> which version?
>>>>
>>>> OpenMPI - can you confirm the version again and also send me the output
>>>> of mpif90 --showme (it might be --show or -show or something similar) -
>>>> essentially I want to see what the underlying compilation line is.
>>>>
>>>> Can you confirm what you had $DO_PARALLEL set to when you ran make test
>>>> for the parallel GPU build. Also can you confirm if the regular (CPU)
>>>> parallel build passed the tests please?
>>>>
>>>> Also did you run 'make clean' before each build step? E.g.
>>>>
>>>> ./configure -cuda gnu
>>>> make -j8 install
>>>> make test
>>>> *make clean*
>>>>
>>>> ./configure -cuda -mpi gnu
>>>> make -j8 install
>>>> make test
>>>>
>>>> Have you tried any other MPI installations? - E.g. MPICH?
>>>>
>>>> And finally can you please confirm which version of Amber (and
>>>> AmberTools) this is and which patches have been applied?
>>>>
>>>> Thanks.
>>>>
>>>> All the best
>>>> Ross
>>>>
>>>> On Jul 21, 2016, at 14:20, Steven Ford <sford123.ibbr.umd.edu> wrote:
>>>>
>>>> Ross,
>>>>
>>>> Attached are the log and diff files. Thank you for taking a look.
>>>>
>>>> Regards,
>>>>
>>>> Steve
>>>>
>>>> On Thu, Jul 21, 2016 at 5:34 AM, Ross Walker <ross.rosswalker.co.uk>
>>>> wrote:
>>>>
>>>>> Hi Steve,
>>>>>
>>>>> Indeed that is too big a difference to just be rounding error -
>>>>> although if those tests are using Langevin or Anderson for the thermostat
>>>>> that would explain it (different random number streams) - although those
>>>>> tests are supposed to be skipped in parallel.
>>>>>
>>>>> Can you send me a copy directly of your .log and .dif files for the 2
>>>>> GPU run and I'll take a closer look at it.
>>>>>
>>>>> All the best
>>>>> Ross
>>>>>
>>>>> > On Jul 20, 2016, at 21:19, Steven Ford <sford123.ibbr.umd.edu>
>>>>> wrote:
>>>>> >
>>>>> > Hello All,
>>>>> >
>>>>> > I currently trying to get Amber16 installed and running on our
>>>>> computing
>>>>> > cluster. Our researchers are primarily interested in running the GPU
>>>>> > accelerated programs. For GPU computing jobs, we have one CentOS 6.7
>>>>> node
>>>>> > with a Tesla K80.
>>>>> >
>>>>> > I was able to build Amber16 and run the Serial/Parallel CPU plus the
>>>>> Serial
>>>>> > GPU tests with all file comparisons passing. However, only 5
>>>>> parallel GPU
>>>>> > tests succeeded, while the other 100 comparisons failed.
>>>>> >
>>>>> > Examining the diff file shows that some of the numbers are not off
>>>>> by much
>>>>> > like the documentation said could happen. For example:
>>>>> >
>>>>> > 66c66
>>>>> > < NSTEP = 1 TIME(PS) = 50.002 TEMP(K) = 351.27
>>>>> PRESS =
>>>>> > 0.
>>>>> >> NSTEP = 1 TIME(PS) = 50.002 TEMP(K) = 353.29
>>>>> PRESS =
>>>>> > 0.
>>>>> >
>>>>> > This may also be too large to attribute to a rounding error, but it
>>>>> is a
>>>>> > small difference compared to others:
>>>>> >
>>>>> > 85c85
>>>>> > < Etot = -217.1552 EKtot = 238.6655 EPtot =
>>>>> > -455.8207
>>>>> >> Etot = -1014.2562 EKtot = 244.6242 EPtot =
>>>>> > -1258.8804
>>>>> >
>>>>> > This was build with CUDA 7.5, OpenMPI 1.8, and run with
>>>>> DO_PARALLEL="mpirun
>>>>> > -np 2"
>>>>> >
>>>>> > Any idea what else could be affecting the output?
>>>>> >
>>>>> > Thanks,
>>>>> >
>>>>> > Steve
>>>>> >
>>>>> > --
>>>>> > Steven Ford
>>>>> > IT Infrastructure Specialist
>>>>> > Institute for Bioscience and Biotechnology Research
>>>>> > University of Maryland
>>>>> > (240)314-6405
>>>>> > _______________________________________________
>>>>> > AMBER mailing list
>>>>> > AMBER.ambermd.org
>>>>> > http://lists.ambermd.org/mailman/listinfo/amber
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> AMBER mailing list
>>>>> AMBER.ambermd.org
>>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Steven Ford
>>>> IT Infrastructure Specialist
>>>> Institute for Bioscience and Biotechnology Research
>>>> University of Maryland
>>>> (240)314-6405
>>>> <2016-07-20_11-17-52.diff><2016-07-20_11-17-52.log>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Steven Ford
>>> IT Infrastructure Specialist
>>> Institute for Bioscience and Biotechnology Research
>>> University of Maryland
>>> (240)314-6405
>>>
>>
>>
>>
>> --
>> Steven Ford
>> IT Infrastructure Specialist
>> Institute for Bioscience and Biotechnology Research
>> University of Maryland
>> (240)314-6405
>>
>>
>>
>
>
> --
> Steven Ford
> IT Infrastructure Specialist
> Institute for Bioscience and Biotechnology Research
> University of Maryland
> (240)314-6405
>
>
>
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Thu Aug 11 2016 - 22:00:02 PDT
