Hi Kenneth,
No idea on the Underflow error - I've not seen it before - do you know where that signal is coming from? From the compiler side of things or from the MPI side of things? - as in does it do it for serial GPU runs as well as multi-GPU runs?
If the test cases are all good I wouldn't worry about it.
All the best
Ross
> On Oct 27, 2015, at 9:56 AM, Kenneth Huang <kennethneltharion.gmail.com> wrote:
>
> Hi Ross,
>
> Mahidhar actually went ahead and recompiled amber under gnu instead of
> Intel, and all of the tests I've done with it seem to be working fine now.
>
> That said, I am consistently seeing a message in the job log files for the
> runs- the outputs from Amber don't seem to have any issues, and there's
> nothing so far to suggest anything is actually wrong, but I wanted to check
> if it's an actual issue.
>
> Note: The following floating-point exceptions are signalling:
> IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
> Note: The following floating-point exceptions are signalling: IEEE_DENORMAL
>
> Best,
>
> Kenneth
>
> On Fri, Oct 23, 2015 at 12:37 AM, Ross Walker <ross.rosswalker.co.uk> wrote:
>
>> Hi Kenneth,
>>
>> A few things to try.
>>
>> 1) Right after the modules are loaded add: nvidia-smi -pm 1 (this will
>> force loading of the nvidia driver)
>>
>> 2) Are you certain that the CUDA_VISIBLE_DEVICES you are specifying in the
>> mpi command is getting propogated? - What does your two mdout files report
>> for the value of CUDA_VISIBLE_DEVICES? If this isn't getting propogated
>> that would explain it.
>>
>> 3) Ditch all this fancy mpi crap: MV2_USE_CUDA=1
>> MV2_USE_GPUDIRECT_GDRCOPY=0 MV2_CPU_MAPPING=0:2
>>
>> I have no idea what these options are doing and they are probably just
>> breaking things. I'd also ditch all the fancy module loads:
>>
>>> module load amber/14
>>
>>> module load intel/2015.2.164
>>> module load cuda/6.5
>>> module load mvapich2_gdr
>>
>> Load plain old vanilla GCC and Gfortran and vanilla mpich3 or mpich2 - and
>> compile your own copy of amber 14 with:
>>
>> ./configure -cuda gnu
>> make
>> make clean
>> ./configure -cuda -mpi gnu
>> make
>> make clean
>>
>> And you'll probably find all you problems go away. Amber GPU was written
>> deliberately not to need ANY fancy compilers or mpi libraries, or fancy
>> interconnects or fancy GPU direct options etc. They all just make things
>> fragile. I don't know why the SDSC folk compiled the GPU code for the
>> modules listed above in the first place.
>>
>> Ultimately you want it nice and simple. I don't have a login on Comet so I
>> can't give you the exact options but something like
>>
>> module load gnu/4.4.7
>> module load mpich3
>> module load cuda/6.5
>>
>> copy in your own amber 14 and AmberTools15 tar files. Untar them in ~/ and
>>
>> export AMBERHOME=~/amber14
>> ./update_amber --update
>> ./configure -cuda gnu
>> make -j8 install
>> make clean
>> ./configure -cuda -mpi gnu
>> make -j8 install
>> make clean
>>
>>
>> Then have your runscript be real simple like:
>>
>> #!/bin/bash
>> #SBATCH --job-name="testB1"
>> #SBATCH --output="comet.%j.%N.out"
>> #SBATCH --partition=gpu
>> #SBATCH --nodes=1
>> #SBATCH --ntasks-per-node=4
>> #SBATCH --no-requeue
>> #SBATCH --gres=gpu:4
>> #SBATCH --export=ALL
>> #SBATCH -t 00:10:000
>> #SBATCH -A TG-TRA130030
>> #SBATCH --mail-type=begin
>> #SBATCH --mail-type=end
>>
>> module load gnu/4.4.7
>> module load mpich3
>> module load cuda/6.5
>>
>> hostname
>> nvidia-smi (-pm 1 if you can - may need root in which case leave it out)
>>
>> export AMBERHOME=~/amber14
>>
>> cd job1
>> export CUDA_VISIBLE_DEVICES=0,1
>> mpirun -np 2 $AMBERHOME/bin/pmemd.cuda.MPI -O -i ... >& job1.log &
>>
>> cd ../job2
>> export CUDA_VISIBLE_DEVICES=2,3
>> mpirun -np 2 $AMBERHOME/bin/pmemd.cuda.MPI -O -i ... >& job2.log &
>>
>> wait
>>
>> Hope that helps.
>>
>> All the best
>> Ross
>>
>>
>>> On Oct 22, 2015, at 20:39, Kenneth Huang <kennethneltharion.gmail.com>
>> wrote:
>>>
>>> Hi Ross,
>>>
>>> Right, I suppose functionally identical would be a better description.
>>>
>>> First thing first is to run jobs with 'IDENTICAL' input on both sets of
>>>> GPUs. If you see it fail on one set but not the other then it means it
>> is a
>>>> machine configuration issue / bios / etc and I can escalate it to SDSC
>>>> support.
>>>>
>>>
>>> That's what happens when it fails- I haven't been able to see it fail on
>>> both jobs yet, which might be an issue of running more tests. Whatever is
>>> the first job in the script seems to be the one to fail when the error
>> pops
>>> up.
>>>
>>> With identical inputs for both GPUs using 06B, the first job in the
>> script
>>> failed while the second was able to run. But resubmitting the exact same
>>> job, both jobs ran without any issues. Doing the same thing for 06A
>> didn't
>>> see any issues on two tries, even though that job was the one originally
>>> failing in the past (possibly because it was first).
>>>
>>> If I try with different inputs, ie swapping 06B on GPUs 0,1 and 06A on
>> 2,3,
>>> then when it's the one on top (06B) that fails. Likewise, the bottom one
>>> that previously kept hitting the MPI error (06A) no longer has any
>> issues.
>>>
>>> I actually opened a ticket with SDSC support through the XSEDE help desk
>>> earlier this week about this and some bizarre performance drops on one of
>>> the GPU nodes, but we couldn't figure out if this problem was a bug or a
>>> resource issue, so I figured to check and see.
>>>
>>>
>>> If it fails on both (or runs fine on both) then it says it is something
>>>> with your job and we can attempt to find if there is a bug in the GPU
>> code
>>>> or something weird about your input. To do this though I need input that
>>>> fails on any combination of 2 GPUs.
>>>
>>>
>>> That's the part I can't get my head around is that the error doesn't seem
>>> to be consistent? The 06B job mentioned above used the below script and
>>> failed with first job, but work fine on both when I resubmitted it
>> without
>>> changing anything.
>>>
>>> #!/bin/bash
>>> #SBATCH --job-name="testB1"
>>> #SBATCH --output="comet.%j.%N.out"
>>> #SBATCH --partition=gpu
>>> #SBATCH --nodes=1
>>> #SBATCH --ntasks-per-node=24
>>> #SBATCH --no-requeue
>>> #SBATCH --gres=gpu:4
>>> #SBATCH --export=ALL
>>> #SBATCH -t 00:10:000
>>> #SBATCH -A TG-TRA130030
>>> #SBATCH --mail-type=begin
>>> #SBATCH --mail-type=end
>>>
>>> module load amber/14
>>> module load intel/2015.2.164
>>> module load cuda/6.5
>>> module load mvapich2_gdr
>>>
>>> export SLURM_NODEFILE=`generate_pbs_nodefile`
>>> mpirun_rsh -hostfile $SLURM_NODEFILE -np 2 MV2_USE_CUDA=1
>>> MV2_USE_GPUDIRECT_GDRCOPY=0 MV2_CPU_MAPPING=0:2 CUDA_VISIBLE_DEVICES=0,1
>>> /share/apps/gpu/amber/pmemd.cuda.MPI -O -i 06B_prod.in -o testB1.out -p
>>> B.prmtop -c 06B_preprod2.rst -r testB1.rst -x testB1.nc -inf
>> testB1.mdinfo
>>> -l testB1.log &
>>>
>>> mpirun_rsh -hostfile $SLURM_NODEFILE -np 2 MV2_USE_CUDA=1
>>> MV2_USE_GPUDIRECT_GDRCOPY=0 MV2_CPU_MAPPING=1:3 CUDA_VISIBLE_DEVICES=2,3
>>> /share/apps/gpu/amber/pmemd.cuda.MPI -O -i 06B_prod.in -o testB2.out -p
>>> B.prmtop -c 06B_preprod2.rst -r testB2.rst -x testB2.nc -inf
>> testB2.mdinfo
>>> -l testB2.log &
>>>
>>> wait
>>>
>>>
>>>
>>> Best,
>>>
>>> Kenneth
>>>
>>> On Thu, Oct 22, 2015 at 12:16 PM, Ross Walker <rosscwalker.gmail.com>
>> wrote:
>>>
>>>> Hi Kenneth,
>>>>
>>>>> Yes, both the inputs and systems themselves are almost identical- 06B
>>>> has a
>>>>> ligand that 06A doesn't have, so the only difference in the inputs is
>> the
>>>>> nmr restraint file that they refer to.
>>>>>
>>>>
>>>> So they are not the same. There is no such thing as 'almost' identical.
>>>> Same as there is no such thing as 'almost' unique. The terms identical
>> and
>>>> unique are absolute adjectives. They can be true or false but nothing in
>>>> between. The same is true of the word 'perfect' - although I note that
>> even
>>>> the US constitution gets this wrong with the phrase "..in order to form
>> a
>>>> more perfect union..."
>>>>
>>>> First thing first is to run jobs with 'IDENTICAL' input on both sets of
>>>> GPUs. If you see it fail on one set but not the other then it means it
>> is a
>>>> machine configuration issue / bios / etc and I can escalate it to SDSC
>>>> support.
>>>>
>>>> If it fails on both (or runs fine on both) then it says it is something
>>>> with your job and we can attempt to find if there is a bug in the GPU
>> code
>>>> or something weird about your input. To do this though I need input that
>>>> fails on any combination of 2 GPUs.
>>>>
>>>> All the best
>>>> Ross
>>>>
>>>> /\
>>>> \/
>>>> |\oss Walker
>>>>
>>>> ---------------------------------------------------------
>>>> | Associate Research Professor |
>>>> | San Diego Supercomputer Center |
>>>> | Adjunct Associate Professor |
>>>> | Dept. of Chemistry and Biochemistry |
>>>> | University of California San Diego |
>>>> | NVIDIA Fellow |
>>>> | http://www.rosswalker.co.uk | http://www.wmd-lab.org |
>>>> | Tel: +1 858 822 0854 | EMail:- ross.rosswalker.co.uk |
>>>> ---------------------------------------------------------
>>>>
>>>> Note: Electronic Mail is not secure, has no guarantee of delivery, may
>> not
>>>> be read every day, and should not be used for urgent or sensitive
>> issues.
>>>>
>>>>
>>>> _______________________________________________
>>>> AMBER mailing list
>>>> AMBER.ambermd.org
>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>>
>>>
>>>
>>>
>>> --
>>> Ask yourselves, all of you, what power would hell have if those
>> imprisoned
>>> here could not dream of heaven?
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>
>>
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>>
>
>
>
> --
> Ask yourselves, all of you, what power would hell have if those imprisoned
> here could not dream of heaven?
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Tue Oct 27 2015 - 10:30:03 PDT
