Which ones will those be? There are some other failures on 4 GPUs but I am not sure.
I am attaching the log files for the tests for 2 and 4 GPUs.
Marcela
> On Jul 3, 2019, at 3:55 PM, koushik kasavajhala <koushik.sbiiit.gmail.com> wrote:
>
> I do not have any other issues except REMD on GPUs. REMD on CPUs runs
> fine.
>
> Sorry, it wasn't clear from the above line. Do other parallel GPU tests
> pass?
>
>
>
> On Wed, Jul 3, 2019 at 3:45 PM Marcela Madrid <mmadrid.psc.edu> wrote:
>
>> hi, thanks Koushik.
>>
>> I do not think it is just the test case as the user is having the same
>> problem.
>> I do not have any other issues except REMD on GPUs. REMD on CPUs runs fine.
>> That is why I am fearing that I may be assigning the GPUs wrong?
>> I wonder if you have an account at the PSC and would like to try, or we
>> can give you one?
>>
>> I asked for an interactive session with 2 GPUs and 2 tasks per node:
>>
>> interact -p GPU --gres=gpu:p100:2 -n 2
>> export DO_PARALLEL=“mpirun -np 2"
>> ./Run.rem.sh
>>
>> I diff the file that you sent me and they are identical. I commented out
>> the last lines in the Run script,
>> but no other output is produced, only:
>>
>>> rem_2rep_gb]$ ./Run.rem.sh
>>> No precision model specified. Defaulting to DPFP.
>>>
>> --------------------------------------------------------------------------------
>>> Two replica GB REMD test.
>>>
>>> Running multipmemd version of pmemd Amber18
>>> Total processors = 2
>>> Number of groups = 2
>>>
>>>
>>> Unit 5 Error on OPEN: rem.in.001
>>
>>
>>
>>>
>>> Unit 5 Error on OPEN: rem.in.001
>>
>>
>>
>>> Abort(1) on node 1 (rank 1 in comm 0): application called
>> MPI_Abort(MPI_COMM_WORLD, 1) - process 1
>>> ./Run.rem.sh: Program error
>>
>> and the two input files that are not deleted because I commented out the
>> line to delete them:
>>
>> more rem.in.001
>> Ala3 GB REMD
>> &cntrl
>> imin = 0, nstlim = 100, dt = 0.002,
>> ntx = 5, irest = 1, ig = -71277,
>> ntwx = 500, ntwe = 0, ntwr = 500, ntpr = 100,
>> ioutfm = 0, ntxo = 1,
>> ntt = 1, tautp = 5.0, tempi = 0.0, temp0 = 350.0,
>> ntc = 2, tol = 0.000001, ntf = 2, ntb = 0,
>> cut = 9999.0, nscm = 500,
>> igb = 5, offset = 0.09,
>> numexchg = 5,
>> &end
>>
>> and rem.in.000
>>
>> thanks, Marcela
>>
>>> On Jul 2, 2019, at 4:23 PM, koushik kasavajhala <
>> koushik.sbiiit.gmail.com> wrote:
>>>
>>> Interesting. I have attached my rem_2rep_gb test directory as a
>> reference.
>>> I just want to make sure you do not have a corrupted version of AMBER.
>>> Check if there are differences between your files and the attached files.
>>> After that, can you comment out lines 52-57 and run the test and let us
>>> know the output of rem.out.000 file? There is usually more information in
>>> that file if a program fails.
>>>
>>> Also, do you have any issues running other tests besides REMD tests?
>>>
>>> On Tue, Jul 2, 2019 at 2:59 PM Marcela Madrid <mmadrid.psc.edu> wrote:
>>>
>>>> Thanks Koushik,
>>>>
>>>> The user is getting this same error about not finding the input files.
>>>> That is why I am doing these tests for her.
>>>> I run from the directory $AMBERHOME/test/cuda/remd/rem_2rep_gb
>>>> export DO_PARALLEL=“mpirun -np 2"
>>>> with 2 GPUs and number of tasks =2
>>>> And this is the error that I am getting:
>>>>
>>>>> ./Run.rem.sh
>>>>> No precision model specified. Defaulting to DPFP.
>>>>>
>>>>
>> --------------------------------------------------------------------------------
>>>>> Two replica GB REMD test.
>>>>>
>>>>> Running multipmemd version of pmemd Amber18
>>>>> Total processors = 2
>>>>> Number of groups = 2
>>>>>
>>>>>
>>>>> Unit 5 Error on OPEN: rem.in.001
>>>>
>>>>
>>>>
>>>>>
>>>>> Unit 5 Error on OPEN: rem.in.001
>>>>
>>>>
>>>>
>>>>> Abort(1) on node 1 (rank 1 in comm 0): application called
>>>> MPI_Abort(MPI_COMM_WORLD, 1) - process 1
>>>>> ./Run.rem.sh: Program error
>>>>>
>>>> I commented out the line at the end of Run.rem.sh so that it does not
>>>> erase the input files and they are there:
>>>> more rem.in.001
>>>>
>>>> Ala3 GB REMD
>>>> &cntrl
>>>> imin = 0, nstlim = 100, dt = 0.002,
>>>> ntx = 5, irest = 1, ig = -71277,
>>>> ntwx = 500, ntwe = 0, ntwr = 500, ntpr = 100,
>>>> ioutfm = 0, ntxo = 1,
>>>> ntt = 1, tautp = 5.0, tempi = 0.0, temp0 = 350.0,
>>>> ntc = 2, tol = 0.000001, ntf = 2, ntb = 0,
>>>> cut = 9999.0, nscm = 500,
>>>> igb = 5, offset = 0.09,
>>>> numexchg = 5,
>>>> &end
>>>>
>>>> So at least I know that your are running the same way we are, and not
>>>> getting error message.
>>>> It is quite puzzling.
>>>>
>>>> Marcela
>>>>
>>>>
>>>>
>>>>> On Jul 2, 2019, at 11:56 AM, koushik kasavajhala <
>>>> koushik.sbiiit.gmail.com> wrote:
>>>>>
>>>>> Hi Marcela,
>>>>>
>>>>> Our lab also uses a slurm queuing system. We use the below script,
>> which
>>>> is
>>>>> similar to your script, to submit 2 replica REMD jobs to one node.
>>>>>
>>>>> #!/bin/bash
>>>>> #SBATCH -N 1
>>>>> #SBATCH --tasks-per-node 2
>>>>> #SBATCH --gres=gpu:2
>>>>>
>>>>> mpirun -np 2 /opt/amber/bin/pmemd.cuda.MPI -O -ng 2 -groupfile
>>>> groupremd
>>>>>
>>>>> So, I do not see anything wrong with your submission script. Since your
>>>> CPU
>>>>> jobs run fine, I think there might be some issue with the way the GPUs
>>>> are
>>>>> configured on your cluster. Note: CPU REMD jobs require 2 cpus per
>>>> replica
>>>>> whereas the GPU REMD jobs require only 1 gpu per replica.
>>>>>
>>>>> I just ran the test cases with 2 and 4 replicas, they all pass for me.
>> If
>>>>> you are having issues with the test cases, I think something might be
>>>> wrong
>>>>> with the way files are being sourced. I don't think it is a compiler
>>>> issue
>>>>> either. We use gnu compilers on our cluster and all tests pass for us.
>>>>>
>>>>> Can you run the test cases inside the directory that David Case pointed
>>>>> out? There is a Run.rem.sh file inside
>>>> AMBERHOME/test/cuda/remd/rem_2rep_gb
>>>>> directory. Executing this file should not give the error message that
>>>> input
>>>>> files were not found. If this doesn't work, then can you post the error
>>>> the
>>>>> user had? They might have had a different error instead of input files
>>>> not
>>>>> being found.
>>>>>
>>>>> .David Case: I looked at the files in test/cuda/remd folder.
>> rem_gb_2rep,
>>>>> rem_gb_4rep, rem_wat are not used at all. Deleting those folders did
>> not
>>>>> affect any of the test cases; they all passed.
>>>>>
>>>>> Best,
>>>>> Koushik
>>>>> Carlos Simmerling Lab
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jul 2, 2019 at 9:57 AM Marcela Madrid <mmadrid.psc.edu> wrote:
>>>>>
>>>>>> hi Dave,
>>>>>>
>>>>>> thanks for your answer. It is not just a problem with the test
>> examples.
>>>>>> It is a problem whenever we try to run REMD on the GPUs on Bridges at
>>>> the
>>>>>> PSC.
>>>>>> The reason why I am looking at it is a user wants to run it. REMD on
>> the
>>>>>> CPUs works fine (with the corresponding executable of course),
>>>>>> it is just a problem with the GPUs. So it occurred to me to see if it
>>>>>> passes the tests and we have the same error
>>>>>> messages. The user has her input files in the directory where she
>> runs.
>>>>>>
>>>>>> I think it is either a problem with the configuration of the GPU nodes
>>>> on
>>>>>> Bridges or a bug.
>>>>>> Each Bridges node has 24 cores and 2 P100 GPUs. I have asked for 1
>> node,
>>>>>> ntasks-per-node=2 and the 2 GPUs
>>>>>> but I get the error message about not finding the input files.
>>>>>> Amber on GPUs was compiled with
>>>>>> ./configure -cuda -mpi gnu
>>>>>> Attempting to compile with intel compilers instead of gnu gave error
>>>>>> messages.
>>>>>>
>>>>>> O3 -ccbin icpc -o cuda_mg_wrapper.o -c cuda_mg_wrapper.cu
>>>>>> In file included from
>>>>>>
>>>>
>> /opt/packages/cuda/9.2/bin/../targets/x86_64-linux/include/host_config.h(50),
>>>>>> from
>>>>>>
>>>>
>> /opt/packages/cuda/9.2/bin/../targets/x86_64-linux/include/cuda_runtime.h(78),
>>>>>> from cuda_mg_wrapper.cu(0):
>>>>>>
>>>>
>> /opt/packages/cuda/9.2/bin/../targets/x86_64-linux/include/crt/host_config.h(79):
>>>>>> error: #error directive: -- unsupported ICC configuration! Only
>>>>>> ICC 15.0, ICC 16.0, and ICC 17.0 on Linux x86_64 are supported!
>>>>>> #error -- unsupported ICC configuration! Only ICC 15.0, ICC 16.0, and
>>>>>> ICC 17.0 on Linux x86_64 are supported!
>>>>>>
>>>>>> We do not have such old versions of the compilers. Any hints will be
>>>>>> appreciated
>>>>>> as to how to run REMD on the GPUS. Thanks so much,
>>>>>>
>>>>>> Marcela
>>>>>>
>>>>>>
>>>>>>> On Jul 2, 2019, at 9:01 AM, David A Case <david.case.rutgers.edu>
>>>> wrote:
>>>>>>>
>>>>>>> On Mon, Jul 01, 2019, Marcela Madrid wrote:
>>>>>>>
>>>>>>>>> Two replica GB REMD test.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Unit 5 Error on OPEN: rem.in.001
>>>>>>>
>>>>>>> OK: query for the REMD experts: in AMBERHOME/test/cuda/remd there are
>>>>>>> two directories: rem_2rep_gb and rem_gb_2rep. The rem.in.00? files
>> are
>>>>>>> in the former, but the tests actually get run in the latter
>> directory.
>>>>>>>
>>>>>>> Same general problem for rem_2rep_pme: the needed rem.in.00? files
>> are
>>>>>>> in rem_wat_2 (or maybe in rem_wat).
>>>>>>>
>>>>>>> I'm probably missing something here, but cleaning up (or at least
>>>>>>> commenting) the cuda/remd test folder seems worthwhile: there are
>>>>>>> folders that seem never to be used, and input files that seem to be
>> in
>>>>>>> the wrong place.
>>>>>>>
>>>>>>> Marcela: I'd ignore these failures for now; something should get
>> posted
>>>>>>> here that either fixes the problem, or figures out a problem with
>> your
>>>>>>> inputs. (My money is on the former.)
>>>>>>>
>>>>>>> ...dac
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> AMBER mailing list
>>>>>>> AMBER.ambermd.org
>>>>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>>>>
>>>>>> _______________________________________________
>>>>>> AMBER mailing list
>>>>>> AMBER.ambermd.org
>>>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>>>>
>>>>> _______________________________________________
>>>>> AMBER mailing list
>>>>> AMBER.ambermd.org
>>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>>
>>>> _______________________________________________
>>>> AMBER mailing list
>>>> AMBER.ambermd.org
>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>>
>>> <rem_2rep_gb.tar.gz>_______________________________________________
>>> AMBER mailing list
>>> AMBER.ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>
>> _______________________________________________
>> AMBER mailing list
>> AMBER.ambermd.org
>> http://lists.ambermd.org/mailman/listinfo/amber
>>
> _______________________________________________
> AMBER mailing list
> AMBER.ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber
_______________________________________________
AMBER mailing list
AMBER.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber
Received on Wed Jul 03 2019 - 13:30:02 PDT
