Aim of this Hirlam technical contribution is purely technical. Without introducing any new functionality and preserving binary reproducibility of output data. Most of in depth testing was done on V8_FOR_UNIFIED_ASSIMILATION branch. Changes where ported to work to a newer V8_ASSIM branch, some of new MEB part problems not fixed. Tested with gfortran v4.8.2, v4.9.2, v5.0.0 and ifort v12.1.6 Since gfortran v4.8.2 has wrong FPE trapping in OpenMP compilation it's results here is not provided. ################################## Platform 1. DragonFly BSD pre4.2.0 gcc version 5.0.0 20150215 (experimental) (FreeBSD Ports Collection) Since dragonfly bsd kernel has low hardcoded stacksize max limit(only 330MB) it was recompiled to have 1GB. But that was not enough for most PGD and some PREP tests when compiled with -fopenmp flag. Problem happens in latlonmask_conf_proj() due to excesive static array allocation in execution stack. Used versions: OLD: LXgfortran, but without -fopenmp flag. LXgfortran-SFX-V8-0-0-MPIAUTO-O2 PGD,PREP,OFFLINE,SODA VER_GRIBAPI=GRIB_API_AUTO VER_CDF=CDFEXT(version 4.1.3) NEW: bgfortran, with FPE trapping -ffpe-trap=overflow,zero,invalid bgfortran-SFX-V8-0-0-MPIAUTO-NOOMP-O2 PGD,PREP bgfortran-SFX-V8-0-0-MPIAUTO-OMP-O2 OFFLINE,SODA VER_GRIBAPI=GRIB_API_EXT VER_CDF=CDFEXT(version 4.1.3) Failing SERIAL tests: PLANTE_CAS_PARTS.txt: TOPD_TOPD_EXP_IGN: OLD_OFFLINE_SERIAL TOPD_TOPD_EXP_IGN: NEW_OFFLINE_SERIAL (see PROBLEM_1) PLANTE_OFFLINE.txt: SHADOWS_SLOPE: OLD_OFFLINE_SERIAL (fixed by changing default NBLOCK=1 => 0 in modd_surfex_omp.F90) SHADOWS_OTHER: OLD_OFFLINE_SERIAL (fixed by changing default NBLOCK=1 => 0 in modd_surfex_omp.F90) Failing PAR tests: PANTE_SEA.txt: (see PROBLEM_2) SEAFLUX_SST_UNIF_271.15_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_OMP4 SEAFLUX_SST_UNIF_271.15_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_MPI2OMP2 SEAFLUX_SST_DATA_ECUME_CANOPY_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_OMP4 SEAFLUX_SST_DATA_ECUME_PRECIP_PWEBB_ICHCE0.75_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_MPI2OMP2 SEAFLUX_SST_DATA_ECUME6_PRECIP_PWEBB_ICHCE0.75_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_OMP4 SEAFLUX_SST_DATA_ECUME6_INTERPOL_SSS_MONTH_CANOPY_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_OMP4 SEAFLUX_SST_DATA_ECUME6_Z01_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_MPI2OMP2 SEAFLUX_SST_DATA_ECUME6_Z02_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_MPI2OMP2 SEAFLUX_SST_DATA_ITERAT_CANOPY_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_OMP4 SEAFLUX_SST_DATA_COARE30_NGRVWAVES1_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_OMP4 SEAFLUX_SST_DATA_SEA_ALBMK10_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_OMP4 PLANTE_CSTS.txt: (OMP4,MPI2OMP2 deadlocks) only SERIAL+MPI tested and passed NAM_SURF_ATM_RRGUST_ARP_VALEURS_CANOPY and others PROBLEM_1: (happens only with gfortran v5.0.0 in TOPD_TOPD_EXP_IGN test) Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation. Backtrace for this error: #0 0x8024EFB37 #1 0x8024EED80 #2 0x7FFFFFFFFFC2 #3 0xC831D5 in topodyn_lat_ at spll_topodyn_lat.F90:179 #4 0xBA57F8 in coupl_topd_ at spll_coupl_topd.F90:296 #5 0xBA3659 in coupling_surf_topd_ at spll_coupling_surf_topd.F90:87 #6 0xA00160 in coupling_isba_n_ at spll_coupling_isba_n.F90:682 #7 0x8CA4C9 in coupling_isba_canopy_n_ at spll_coupling_isba_canopy_n.F90:361 #8 0x726D74 in coupling_isba_orography_n_ at spll_coupling_isba_orography_n.F90:279 #9 0x7288CF in coupling_isba_svat_n_ at spll_coupling_isba_svat_n.F90:223 #10 0x456966 in treat_surf at spll_coupling_surf_atm_n.F90:623 #11 0x457C56 in coupling_surf_atm_n_ at spll_coupling_surf_atm_n.F90:335 #12 0x41079A in MAIN__._omp_fn.2 at spll_offline.F90:941 #13 0x802E438EE #14 0x418D55 in offline at spll_offline.F90:891 PROBLEM_2: (randomly often crashes(4/5), finishes correctly(1/5) per invocation during OMP4,MPI2OMP2 ) Sometimes work, sometimes not. Happen only for NC output, during write of SSS or XX or SBL variables. Something causes openmp data race in SEA MERCATOR PROGSST tests. aborted with text:HANFLE_ERR: ABORTING PROGRAM TO WRITE A NETCDF FILE: SSS| Program aborted. Backtrace: #0 0x8024EFB37 #1 0x8024F0942 #2 0x8025BD9F8 #3 0x454EDC in abor1_sfx_ at spll_abor1_sfx.F90:99 #4 0x6FC6D6 in handle_err_ at spll_handle_err.F90:23 #5 0x6593ED in write_datax1_nc at spll_mode_write_surf_nc.F90:549 #6 0x65DB7C in __mode_write_surf_nc_MOD_write_surfx1_nc at spll_mode_write_surf_nc.F90:484 #7 0x46E973 in write_surfx1_ at spll_write_surfx1.F90:114 (discriminator 2) #8 0x633669 in writesurf_seaflux_n_ at spll_writesurf_seaflux_n.F90:143 #9 0x631A2F in write_seaflux_n_ at spll_write_seaflux_n.F90:73 #10 0x45397C in write_surf_atm_n_ at spll_write_surf_atm_n.F90:138 (discriminator 1) #11 0x410FB3 in MAIN__._omp_fn.5 at spll_offline.F90:1353 OR Program received signal SIGBUS: Access to an undefined portion of a memory object. Backtrace for this error: #5 0xF64C7D in opal_backtrace_buffer #6 0xDD32A7 in ompi_mpi_abort #7 0xDC59C3 in ompi_mpi_errors_are_fatal_comm_handler #8 0xDC5154 in ompi_errhandler_invoke #9 0xD69491 in MPI_Bcast #10 0xD60193 in mpi_bcast__ #11 0x65DBFA in __mode_write_surf_nc_MOD_write_surfx1_nc at spll_mode_write_surf_nc.F90:469 #12 0x46E973 in write_surfx1_ at spll_write_surfx1.F90:114 (discriminator 2) #13 0x637B93 in writesurf_watflux_n_ at spll_writesurf_watflux_n.F90:99 #14 0x631B2F in write_watflux_n_ at spll_write_watflux_n.F90:73 #15 0x4539AA in write_surf_atm_n_ at spll_write_surf_atm_n.F90:144 (discriminator 1) #16 0x410FB3 in MAIN__._omp_fn.5 at spll_offline.F90:1353 OR Program aborted. Backtrace: #0 0x8024EFB37 #1 0x8024F0942 #2 0x8025BD9F8 #3 0x454EDC in abor1_sfx_ at spll_abor1_sfx.F90:99 #4 0x6FC6D6 in handle_err_ at spll_handle_err.F90:23 #5 0x65ED98 in __mode_write_surf_nc_MOD_write_surfl0_nc at spll_mode_write_surf_nc.F90:313 #6 0x46CEEC in write_surfl0_ at spll_write_surfl0.F90:128 #7 0x63672A in writesurf_watflux_sbl_n_ at spll_writesurf_watflux_sbl_n.F90:76 #8 0x631B6C in write_watflux_n_ at spll_write_watflux_n.F90:75 (discriminator 3) #9 0x4539AA in write_surf_atm_n_ at spll_write_surf_atm_n.F90:144 (discriminator 1) #10 0x410FB3 in MAIN__._omp_fn.5 at spll_offline.F90:1353 MD5_status check between OLD and NEW versions: ./PRUN_SEA/MD5_STATUS.txt:SEAFLUX_SST_DATA_ECUME_PRECIP_PWEBB_ICHCE0.75_CANOPY_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_MPI2OMP2 MD5: FAIL ./PRUN_SEA/MD5_STATUS.txt:SEAFLUX_SST_DATA_ECUME6_Z02_CANOPY_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_MPI2OMP2 MD5: FAIL ./PRUN_SEA/MD5_STATUS.txt:SEAFLUX_SST_DATA_COARE30_CANOPY_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_MPI2OMP2 MD5: FAIL ./PRUN_SEA/MD5_STATUS.txt:SEAFLUX_SST_DATA_COARE30_PWG_CANOPY_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_OMP4 MD5: FAIL ./PRUN_SEA/MD5_STATUS.txt:SEAFLUX_SST_DATA_COARE30_NGRVWAVES1_CANOPY_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_MPI2OMP2 MD5: FAIL ./PRUN_SEA/MD5_STATUS.txt:SEAFLUX_SST_DATA_SEA_ALBTA96_CANOPY_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_MPI2OMP2 MD5: FAIL ./PRUN_SEA/MD5_STATUS.txt:SEAFLUX_SST_DATA_SEA_ALBMK10_CANOPY_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_MPI2OMP2 MD5: FAIL ./PRUN_FLAKE/MD5_STATUS.txt:FLAKE_INTERPOL_TS_ANNUAL: NEW_OFFLINE_OMP4 MD5: FAIL ./PRUN_FLAKE/MD5_STATUS.txt:FLAKE_TS_UNIF271_CANOPY_SEDIMENTS: NEW_OFFLINE_OMP4 MD5: FAIL ./PRUN_FLAKE/MD5_STATUS.txt:FLAKE_FLAKE_CANOPY_SEDIMENTS: NEW_OFFLINE_MPI2OMP2 MD5: FAIL ./PRUN_FLAKE/MD5_STATUS.txt:FLAKE_FLK_ALBTA96: NEW_OFFLINE_OMP4 MD5: FAIL Remark Parallel flake tests fail only for OMP4,MPI2OMP2 execution, slight numerical disturbance. No NaNs or XUNDEF mask changes in output files. Other *.nc *.txt *.lfi(with -DLFI_BFD define) *.fa(with -DLFI_BFB define) outputs are binary identical. ############################### Platform 2. Old SLES linux, sgi cluster with ifort version 12.1.6 Special code changes where done to ensure it compiles with -fpic -openmp -openmp-threadprivate=compat in: modd_teb_irrign.F90 (init pointers to NULL() modd_seafluxn.F90 (init pointers to NULL()) mode_flake.F90 (init threadprivate variables to REAL,PARAMETER,PRIVATE :: Z_=-HUGE(0.0)) mode_sfcflx.F90 (init threadprivate variables to REAL,PARAMETER,PRIVATE :: Z_=-HUGE(0.0)) While this helped to compile, it still crashed during runtime with segfaults, thus -fpic was not used. Also to help with reproducibility READ_SURFX2COV/WRITE_SURFX2COV were moved out from read_surf.F90/write_surf.F90 OLD: LXifort, but without -fpic flag. LXifort-SFX-V8-0-0-MPIAUTO-O2 PGD,PREP,OFFLINE,SODA VER_GRIBAPI=GRIB_API_AUTO VER_CDF=CDFEXT(version 4.3.2) NEW: bgfortran, with -assume ieee_fpe_flags flag zifort-SFX-V8-0-0-MPIAUTO-OMP-O2 PGD,PREP,OFFLINE,SODA VER_GRIBAPI=GRIB_API_EXT VER_CDF=CDFEXT(version 4.3.2) Failing SERIAL tests: RUN_CAS/PLANTE_CAS_PARTS.txt TOPD_TOPD_EXP_12P: OLD_OFFLINE_SERIAL (see PROBLEM_3) RUN_ISBA_2lpho/PLANTE_ISBA.txt (many OLD test fails with PROBLEM_3) ISBA_2L_PHOTO_12P: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_12P_CANOPY: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_12P_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_AGS: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_AGS_PRM: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_AGS_PRM_TR_ML: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_AST: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_AST_TR_ML: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_AST_TR_ML_CANOPY: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_AST_TR_ML_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_AST_TR_ML_PRM: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LAI: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LAI_TR_ML: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LAI_TR_ML_PRM: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LAI_TR_ML_PRM_CANOPY: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LAI_TR_ML_PRM_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LAI_TR_ML_PRM_AGRIP: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LAI_TR_ML_PRM_AGRIP_IRRIG_FILE: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LAI_TR_ML_PRM_AGRIP_IRRIG_FILE_LCLIM_LAIF: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LST: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LST_AGRIP: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LST_AGRIP_TR_ML: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_LST_AGRIP_TR_ML_PRM: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_D95: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_D95_CANOPY: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_D95_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_EBA: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_EBA_CANOPY: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_EBA_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SN3L: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SN3L_CANOPY: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SN3L_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SN3L_GLACIER: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SN3L_GLACIER_RIL: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SNCRO8: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SNCRO8: NEW_OFFLINE_SERIAL (see PROBLEM_4) ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER: NEW_OFFLINE_SERIAL (see PROBLEM_4) ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER_RIL: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER_RIL: NEW_OFFLINE_SERIAL (see PROBLEM_4) ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER_RIL_CANOPY: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER_RIL_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: NEW_OFFLINE_SERIAL (see PROBLEM_4) ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM_NOSNOWDRIFT: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM_NOSNOWDRIFT: NEW_OFFLINE_SERIAL (see PROBLEM_4) ISBA_2L_PHOTO_NIT_SNCRO10: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SNCRO15: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_NITRO_DILU: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_NITRO_DILU_PRM: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_NITRO_DILU_PRM_AGRIP: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_NITRO_DILU_PRM_AGRIP_TR_ML: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB_PRM: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB_PRM_TR_ML: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB_CNT: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB_CNT_TR_ML: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB_CNT_TR_ML_NITRO_DILU: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB_CNT_TR_ML_SPINUPCARBS: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB_CNT_TR_ML_SPINUPCARBS_CANOPY: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB_CNT_TR_ML_SPINUPCARBS_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB_AGRIP: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB_AGRIP_CNT: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NCB_AGRIP_CNT_SPINUPCARBW: OLD_OFFLINE_SERIAL RUN_ISBA_3lpho/PLANTE_ISBA.txt (many just OLD test fails with PROBLEM_3) ISBA_3L_PHOTO_12P: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_12P_CANOPY: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_12P_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_AGS: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_AGS_PRM: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_AGS_PRM_TR_ML: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_AST: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_AST_TR_ML: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_AST_TR_ML_CANOPY: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_AST_TR_ML_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_AST_TR_ML_PRM: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LAI: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LAI_TR_ML: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LAI_TR_ML_PRM: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LAI_TR_ML_PRM_CANOPY: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LAI_TR_ML_PRM_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LAI_TR_ML_PRM_AGRIP: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LAI_TR_ML_PRM_AGRIP_IRRIG_FILE: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LAI_TR_ML_PRM_AGRIP_IRRIG_FILE_LCLIM_LAIF: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LST: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LST_AGRIP: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LST_AGRIP_TR_ML: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_LST_AGRIP_TR_ML_PRM: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_D95: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_D95_CANOPY: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_D95_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_EBA: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_EBA_CANOPY: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_EBA_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SN3L: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SN3L_CANOPY: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SN3L_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SN3L_GLACIER: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SN3L_GLACIER_RIL: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SNCRO8: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SNCRO8_GLACIER: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SNCRO8_GLACIER_RIL: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_CANOPY: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM_NOSNOWDRIFT: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SNCRO10: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SNCRO15: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_NITRO_DILU: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_NITRO_DILU_PRM: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_NITRO_DILU_PRM_AGRIP: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_NITRO_DILU_PRM_AGRIP_TR_ML: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB_PRM: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB_PRM_TR_ML: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB_CNT: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB_CNT_TR_ML: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB_CNT_TR_ML_NITRO_DILU: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB_CNT_TR_ML_SPINUPCARBS: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB_CNT_TR_ML_SPINUPCARBS_CANOPY: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB_CNT_TR_ML_SPINUPCARBS_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB_AGRIP: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB_AGRIP_CNT: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NCB_AGRIP_CNT_SPINUPCARBW: OLD_OFFLINE_SERIAL RUN_ISBA_difpho/PLANTE_ISBA.txt (many OLD test fails with PROBLEM_3) ISBA_DIF8_SN3L_PHOTO_SNCRO8_C13_SNOWRAD_TA2: NEW_OFFLINE_SERIAL (see PROBLEM_5) FIXED by rev 2921 ISBA_DIF8_SN3L_PHOTO_12P: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_12P_CANOPY: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_12P_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_AGS: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_AGS_PRM: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_AGS_PRM_TR_ML: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_AST: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_AST_TR_ML: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_AST_TR_ML_CANOPY: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_AST_TR_ML_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_AST_TR_ML_PRM: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LAI: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LAI_TR_ML: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LAI_TR_ML_PRM: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LAI_TR_ML_PRM_CANOPY: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LAI_TR_ML_PRM_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LAI_TR_ML_PRM_AGRIP: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LAI_TR_ML_PRM_AGRIP_IRRIG_FILE: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LAI_TR_ML_PRM_AGRIP_IRRIG_FILE_LCLIM_LAIF: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LST: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LST_AGRIP: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LST_AGRIP_TR_ML: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_LST_AGRIP_TR_ML_PRM: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SN3L: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SN3L_CANOPY: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SN3L_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SN3L_GLACIER: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SN3L_GLACIER_RIL: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_GLACIER: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_GLACIER_RIL: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_CANOPY: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM_NOSNOWDRIFT: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO10: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO15: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_SNOWMETAMO_C13_B92: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_SNOWMETAMO_T07_B92: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_SNOWMETAMO_F06_B92: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_B92_SNOWRAD_TAR: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_C13_SNOWRAD_TAR: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_C13_SNOWRAD_TAR: NEW_OFFLINE_SERIAL (see PROBLEM_5) FIXED by rev 2921 ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_C13_SNOWRAD_TA1: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_C13_SNOWRAD_TA1: NEW_OFFLINE_SERIAL (see PROBLEM_5) FIXED by rev 2921 ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_C13_SNOWRAD_TA2: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_SNCRO8_C13_SNOWRAD_TA2: NEW_OFFLINE_SERIAL (see PROBLEM_5) FIXED by rev 2921 ISBA_DIF8_SN3L_PHOTO_NIT_NITRO_DILU: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_NITRO_DILU_PRM: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_NITRO_DILU_PRM_AGRIP: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NIT_NITRO_DILU_PRM_AGRIP_TR_ML: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB_PRM: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB_PRM_TR_ML: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB_PRM_TR_ML_NITRO_DILU: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB_CNT: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB_CNT_TR_ML: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB_CNT_TR_ML_SPINUPCARBS: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB_CNT_TR_ML_SPINUPCARBS_CANOPY: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB_CNT_TR_ML_SPINUPCARBS_CANOPY_DRAG: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB_AGRIP: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB_AGRIP_CNT: OLD_OFFLINE_SERIAL ISBA_DIF8_SN3L_PHOTO_NCB_AGRIP_CNT_SPINUPCARBW: OLD_OFFLINE_SERIAL RUN_OFFLINE/PLANTE_OFFLINE.txt (many OLD test fails with PROBLEM_3) SURF_FILETYPE_NC_RESTART: OLD_OFFLINE_SERIAL SURF_FILETYPE_NC_RESTART_OLD: OLD_OFFLINE_SERIAL SURF_FILETYPE_ASCII_RESTART: OLD_OFFLINE_SERIAL SURF_FILETYPE_ASCII_RESTART_OLD: OLD_OFFLINE_SERIAL SURF_FILETYPE_LFI: OLD_OFFLINE_SERIAL SURF_FILETYPE_LFI_RESTART: OLD_OFFLINE_SERIAL SURF_FILETYPE_LFI_RESTART_OLD: OLD_OFFLINE_SERIAL SURF_FILETYPE_LFI_RESTART_RESET: OLD_OFFLINE_SERIAL SURF_FILETYPE_LFI_RESTART_RESET_OLD: OLD_OFFLINE_SERIAL SURF_FILETYPE_LFI_RESTART_NO_DIAG: OLD_OFFLINE_SERIAL SURF_FILETYPE_LFI_RESTART_NO_DIAG_OLD: OLD_OFFLINE_SERIAL SURF_FILETYPE_FA: OLD_OFFLINE_SERIAL SURF_FILETYPE_FA_RESTART: OLD_OFFLINE_SERIAL SURF_FILETYPE_FA_RESTART_OLD: OLD_OFFLINE_SERIAL NAM_WRITE_SURF_ATM: OLD_OFFLINE_SERIAL NAM_WRITE_COVER_TEX_EN: OLD_OFFLINE_SERIAL RUN_PREP/PLANTE_PREP.txt (OLD test fails with PROBLEM_3) PREP_EXTE_FILE_ASCII_ISBA_1P12P_NEW: OLD_OFFLINE_SERIAL RUN_SODA/PLANTE_SODA_PARTS.txt (OLD test fails with PROBLEM_3) SODA_EKF1_2007071106_0: OLD_OFFLINE_SERIAL SODA_EKF1_2007071106_1: OLD_OFFLINE_SERIAL SODA_EKF1_2007071106_2: OLD_OFFLINE_SERIAL SODA_EKF1_2007071106: OLD_SODA_SERIAL SODA_EKF1_2007071206_0: OLD_OFFLINE_SERIAL SODA_EKF1_2007071206_1: OLD_OFFLINE_SERIAL SODA_EKF1_2007071206_2: OLD_OFFLINE_SERIAL SODA_EKF1_2007071206: OLD_SODA_SERIAL SODA_EKF1_2007071306_0: OLD_OFFLINE_SERIAL SODA_EKF1_2007071306_1: OLD_OFFLINE_SERIAL SODA_EKF1_2007071306_2: OLD_OFFLINE_SERIAL SODA_EKF1_2007071306: OLD_SODA_SERIAL SODA_EKF1_2007071406_0: OLD_OFFLINE_SERIAL SODA_EKF1_2007071406_1: OLD_OFFLINE_SERIAL SODA_EKF1_2007071406_2: OLD_OFFLINE_SERIAL SODA_EKF1_2007071406: OLD_SODA_SERIAL SODA_EKF1_2007071506_0: OLD_OFFLINE_SERIAL SODA_EKF1_2007071506_1: OLD_OFFLINE_SERIAL SODA_EKF1_2007071506_2: OLD_OFFLINE_SERIAL SODA_EKF1_2007071506: OLD_SODA_SERIAL SODA_EKF1_2007071606_0: OLD_OFFLINE_SERIAL SODA_EKF1_2007071606_1: OLD_OFFLINE_SERIAL SODA_EKF1_2007071606_2: OLD_OFFLINE_SERIAL SODA_EKF1_2007071606: OLD_SODA_SERIAL RUN_TEB_gbem_3l/PLANTE_TEB.txt (many OLD test fails with PROBLEM_3) TEB_GARDEN_GREENROOF_BEM_3L_SOLAR_PANEL: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_3L_SOLAR_PANEL_TWO_WALLS: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_3L_SOLAR_PANEL_CANOPY: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_3L_SOLAR_PANEL_TWO_WALLS_CANOPY: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_3L_SOLAR_PANEL_DATA: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_3L_SOLAR_PANEL_DATA_TWO_WALLS: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_3L_SOLAR_PANEL_DATA_CANOPY: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_3L_SOLAR_PANEL_DATA_TWO_WALLS_CANOPY: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_3L_IRRIG: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_3L_IRRIG_TWO_WALLS: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_3L_IRRIG_CANOPY: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_3L_IRRIG_TWO_WALLS_CANOPY: OLD_OFFLINE_SERIAL RUN_TEB_gbem_nit/PLANTE_TEB.txt (many OLD test fails with PROBLEM_3) TEB_GARDEN_GREENROOF_BEM_NIT: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_NIT_TWO_WALLS: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_NIT_CANOPY: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_NIT_TWO_WALLS_CANOPY: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_NIT_3L: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_NIT_3L_TWO_WALLS: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_NIT_3L_CANOPY: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_NIT_3L_TWO_WALLS_CANOPY: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_NIT_DIF: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_NIT_DIF_TWO_WALLS: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_NIT_DIF_CANOPY: OLD_OFFLINE_SERIAL TEB_GARDEN_GREENROOF_BEM_NIT_DIF_TWO_WALLS_CANOPY: OLD_OFFLINE_SERIAL Failing PAR tests: PRUN_ISBA/PLANTE_ISBA.txt (many OLD test fails with PROBLEM_3) ISBA_2L_PHOTO_LAI_TR_ML_PRM_AGRIP_IRRIG_FILE: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: OLD_OFFLINE_SERIAL ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: NEW_OFFLINE_SERIAL (see PROBLEM_4) ISBA_2L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: NEW_OFFLINE_MPI4_ADJ (see PROBLEM_6) ISBA_3L_PHOTO_LAI_TR_ML_PRM_AGRIP_IRRIG_FILE: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: OLD_OFFLINE_SERIAL ISBA_3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: NEW_OFFLINE_MPI4_LIN (see PROBLEM_6) ISBA_3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: NEW_OFFLINE_MPI4_ADJ (see PROBLEM_6) ISBA_3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: NEW_OFFLINE_OMP4 (see PROBLEM_6) ISBA_3L_PHOTO_NIT_SNCRO8_GLACIER_RIL_SNOWDRIFT_SUBLIM: NEW_OFFLINE_MPI2OMP2 (see PROBLEM_6) ISBA_DIF8_SN3L_PHOTO_LAI_TR_ML_PRM_AGRIP_IRRIG_FILE: OLD_OFFLINE_SERIAL PRUN_SEA/PLANTE_SEA.txt: same as with bgfortran case (see PROBLEM_2 and PROBLEM_7) SEAFLUX_SST_DATA_ECUME6_Z01_CANOPY_MERCATOR_CURRENT_CUR_REL_TS_REL_TIME_REL30000000_DIAPYC_PROGSST: NEW_OFFLINE_OMP4 PRUN_TEB/PLANTE_TEB.txt TEB_SIMPLE_9TP_TWO_WALLS_DT_RES_OFF: NEW_OFFLINE_OMP4 (see PROBLEM_8) PROBLEM_3: only happen with ifort OLD version for many tests for "DO JK=1,SIZE(XP_SNOWSWE,2)" Compiler does not consider XP_SNOWSWE as threadprivate pointer in V8_ASSIM version. forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source offline_v8a.x 0000000000DB0EEA unpack_isba_patch 278 spll_unpack_isba_patch_n.f90 offline_v8a.x 0000000000AC23B0 coupling_isba_n_I 1389 spll_coupling_isba_n.f90 offline_v8a.x 0000000000AB4AE3 coupling_isba_n_ 555 spll_coupling_isba_n.f90 offline_v8a.x 0000000000A78113 coupling_isba_can 351 spll_coupling_isba_canopy_n.f90 offline_v8a.x 0000000000A72E5B coupling_isba_oro 269 spll_coupling_isba_orography_n.f90 offline_v8a.x 0000000000911EFD coupling_isba_sva 213 spll_coupling_isba_svat_n.f90 offline_v8a.x 00000000007F5E49 coupling_nature_n 127 spll_coupling_nature_n.f90 offline_v8a.x 0000000000464DE0 coupling_surf_atm 613 spll_coupling_surf_atm_n.f90 offline_v8a.x 0000000000461072 coupling_surf_atm 335 spll_coupling_surf_atm_n.f90 offline_v8a.x 0000000000415F8A MAIN__ 914 spll_modi_offline.f90 libiomp5.so 00002B8F163F2EE3 Unknown Unknown Unknown PROBLEM_4: only happens still on some tests in NEW version at line ZTHICKNESS_INTERMEDIATE = ZSNOW_UPPER - SUM(ZDZOPT(JJ,1:5)) - ZDZOPT(JJ,INB_UPPER_LAYER) contains +INF values, after tracing seems PGSFCSNOW(intent(OUT)) at snow3l_isba.F90 is not correctly initialized? Since this is a part of MEB contribution didn't had time to trace it down. forrtl: error (65): floating invalid Image PC Routine Line Source offline_safe.x 00000000012DD354 snowcro_IP_snownl 3815 spll_snowcro.f90 offline_safe.x 00000000012A4DFA snowcro_ 543 spll_snowcro.f90 offline_safe.x 00000000010CEAAE snow3l_isba_IP_ca 813 spll_snow3l_isba.f90 offline_safe.x 00000000010C5D9F snow3l_isba_ 443 spll_snow3l_isba.f90 offline_safe.x 0000000000EB80E5 isba_ 1036 spll_isba.f90 offline_safe.x 0000000000DB077B coupling_isba_n_I 1030 spll_coupling_isba_n.f90 offline_safe.x 0000000000DA47E1 coupling_isba_n_ 557 spll_coupling_isba_n.f90 offline_safe.x 0000000000BAFAD1 coupling_isba_can 353 spll_coupling_isba_canopy_n.f90 offline_safe.x 00000000009F201F coupling_isba_oro 271 spll_coupling_isba_orography_n.f90 offline_safe.x 00000000009F3CC0 coupling_isba_sva 215 spll_coupling_isba_svat_n.f90 offline_safe.x 00000000005D4418 coupling_nature_n 129 spll_coupling_nature_n.f90 offline_safe.x 0000000000470991 coupling_surf_atm 615 spll_coupling_surf_atm_n.f90 offline_safe.x 000000000046C491 coupling_surf_atm 337 spll_coupling_surf_atm_n.f90 offline_safe.x 00000000004160A7 MAIN__ 920 spll_offline.f90 libiomp5.so 00002B8087A20EE3 Unknown Unknown Unknown PROBLEM_5: only before rev 2921, after fix mode_tartes.F90 with optional cpp define -DRJ_OFIX by limiting array sizes passed to snowcro_call_tartes() it's no longer a FPE trap in mode_tartes.F90 These tests take quite long time, about ~12-15 minutes in SERIAL. forrtl: error (65): floating invalid Image PC Routine Line Source offline_safe.x 0000000000A690A0 mode_tartes_mp_sh 373 spll_mode_tartes.f90 offline_safe.x 0000000000A679F2 mode_tartes_mp_si 472 spll_mode_tartes.f90 offline_safe.x 0000000000A5F37F mode_tartes_mp_ta 137 spll_mode_tartes.f90 offline_safe.x 0000000000A5B881 mode_tartes_mp_sn 1458 spll_mode_tartes.f90 offline_safe.x 0000000000A5A295 mode_tartes_mp_sn 1315 spll_mode_tartes.f90 offline_safe.x 00000000012A8FDC snowcro_ 740 spll_snowcro.f90 offline_safe.x 00000000010CEAAE snow3l_isba_IP_ca 813 spll_snow3l_isba.f90 offline_safe.x 00000000010C5D9F snow3l_isba_ 443 spll_snow3l_isba.f90 offline_safe.x 0000000000EB80E5 isba_ 1036 spll_isba.f90 offline_safe.x 0000000000DB077B coupling_isba_n_I 1030 spll_coupling_isba_n.f90 offline_safe.x 0000000000DA47E1 coupling_isba_n_ 557 spll_coupling_isba_n.f90 offline_safe.x 0000000000BAFAD1 coupling_isba_can 353 spll_coupling_isba_canopy_n.f90 offline_safe.x 00000000009F201F coupling_isba_oro 271 spll_coupling_isba_orography_n.f90 offline_safe.x 00000000009F3CC0 coupling_isba_sva 215 spll_coupling_isba_svat_n.f90 offline_safe.x 00000000005D4418 coupling_nature_n 129 spll_coupling_nature_n.f90 offline_safe.x 0000000000470991 coupling_surf_atm 615 spll_coupling_surf_atm_n.f90 offline_safe.x 000000000046C491 coupling_surf_atm 337 spll_coupling_surf_atm_n.f90 offline_safe.x 00000000004160A7 MAIN__ 920 spll_offline.f90 libiomp5.so 00002B6A5D90DEE3 Unknown Unknown Unknown PROBLEM_6: only happens still on some tests in NEW version Traps at "PGSFCSNOW(:) = PPSN(:) * PGSFCSNOW(:)" in isba_snow_agr.F90 PGSFCSNOW(intent(OUT)) at snow3l_isba.F90 is not correctly initialized? Since this is a part of MEB contribution didn't had time to trace it down. forrtl: error (65): floating invalid Image PC Routine Line Source offline_safe.x 00000000010B6B07 isba_snow_agr_ 374 spll_isba_snow_agr.f90 offline_safe.x 0000000000EBA6D4 isba_ 1141 spll_isba.f90 offline_safe.x 0000000000DB077B coupling_isba_n_I 1030 spll_coupling_isba_n.f90 offline_safe.x 0000000000DA47E1 coupling_isba_n_ 557 spll_coupling_isba_n.f90 offline_safe.x 0000000000BAFAD1 coupling_isba_can 353 spll_coupling_isba_canopy_n.f90 offline_safe.x 00000000009F201F coupling_isba_oro 271 spll_coupling_isba_orography_n.f90 offline_safe.x 00000000009F3CC0 coupling_isba_sva 215 spll_coupling_isba_svat_n.f90 offline_safe.x 00000000005D4418 coupling_nature_n 129 spll_coupling_nature_n.f90 offline_safe.x 0000000000470991 coupling_surf_atm 615 spll_coupling_surf_atm_n.f90 offline_safe.x 000000000046C491 coupling_surf_atm 337 spll_coupling_surf_atm_n.f90 offline_safe.x 00000000004160A7 MAIN__ 920 spll_offline.f90 PROBLEM_7: only happens with trapping ifort in NEW version at line "IF (ASSOCIATED(XDATA_WALL_O_HOR)) DEALLOCATE(XDATA_WALL_O_HOR)" in arrange_cover.F90 Other tests have random crashes when repeated too. OpenMP data race is suspected. forrtl: severe (153): allocatable array or pointer is not allocated Image PC Routine Line Source offline_safe.x 00000000016ADC6A Unknown Unknown Unknown offline_safe.x 00000000016AC766 Unknown Unknown Unknown offline_safe.x 000000000165B2F0 Unknown Unknown Unknown offline_safe.x 00000000015E9B2F Unknown Unknown Unknown offline_safe.x 0000000001627403 Unknown Unknown Unknown offline_safe.x 00000000009A8D09 arrange_cover_ 99 spll_arrange_cover.f90 offline_safe.x 00000000006CB01A ini_data_cover_ 2220 spll_ini_data_cover.f90 offline_safe.x 00000000004911C7 init_surf_atm_n_ 391 spll_init_surf_atm_n.f90 offline_safe.x 0000000000416C60 MAIN__ 688 spll_offline.f90 libiomp5.so 00002B2802257EE3 Unknown Unknown Unknown PROBLEM_8: only happens with trapping ifort in NEW version at line "IF (ASSOCIATED(XDATA_WALL_O_HOR)) DEALLOCATE(XDATA_WALL_O_HOR)" in arrange_cover.F90 Other tests have random crashes when repeated too. OpenMP data race is suspected. forrtl: severe (153): allocatable array or pointer is not allocated Image PC Routine Line Source offline_safe.x 00000000016ADC6A Unknown Unknown Unknown offline_safe.x 00000000016AC766 Unknown Unknown Unknown offline_safe.x 000000000165B2F0 Unknown Unknown Unknown offline_safe.x 00000000015E9B2F Unknown Unknown Unknown offline_safe.x 0000000001627403 Unknown Unknown Unknown offline_safe.x 00000000009A8D09 arrange_cover_ 99 spll_arrange_cover.f90 offline_safe.x 00000000006CB01A ini_data_cover_ 2220 spll_ini_data_cover.f90 offline_safe.x 00000000004911C7 init_surf_atm_n_ 391 spll_init_surf_atm_n.f90 offline_safe.x 0000000000416C60 MAIN__ 688 spll_offline.f90 libiomp5.so 00002B2802257EE3 Unknown Unknown Unknown MD5_status check between OLD and NEW zifort versions: Only RUN_CSTS TESTS haven't bit-for-bit reproducibility in TEB_PROGNOSTIC.OUT.nc diangostic outputs. Actual variable names and data are identical. Just attributes gets wrangled like: RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-0-Diff: RES_OLD/TEB_PROGNOSTIC.OUT.nc->RES_NEW/TEB_PROGNOSTIC.OUT.nc RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-463- VAR("T1_TROOF1"[96,100]:double:DIMS=2:ATTRS=3) ; OK RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-519- ATTR("_FillValue":double:1) = +1.000000000000000E+020 ; OK RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log:584: ATTR("long_name":char:19->24) = "Gr_snow_heat_layer3"->"Roof_Temperature_Layer_1" ; FAIL RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log:679: ATTR("units":char:4->1) = "J/m2"->"K" ; FAIL RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-730- VAR("T1_TROOF2"[96,100]:double:DIMS=2:ATTRS=3) ; OK RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-786- ATTR("_FillValue":double:1) = +1.000000000000000E+020 ; OK -- RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-2305- VAR("T1_TROAD4"[96,100]:double:DIMS=2:ATTRS=3) ; OK RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-2361- ATTR("_FillValue":double:1) = +1.000000000000000E+020 ; OK RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log:2426: ATTR("long_name":char:24->14) = "Road_Temperature_Layer_4"->"Gr_snow_albedo" ; FAIL RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log:2516: ATTR("units":char:1) = "K"->"-" ; FAIL RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-2561- VAR("T1_TROAD5"[96,100]:double:DIMS=2:ATTRS=3) ; OK RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-2617- ATTR("_FillValue":double:1) = +1.000000000000000E+020 ; OK -- RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-9353- VAR("T1_WSN_RF1"[96,1,100]:double:DIMS=3:ATTRS=3) ; OK RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-9412- ATTR("_FillValue":double:1) = +1.000000000000000E+020 ; OK RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log:9477: ATTR("long_name":char:24->19) = "Rf_Snow_Water_Eq_layer_1"->"Gr_snow_heat_layer3" ; FAIL RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log:9572: ATTR("units":char:5->4) = "Kg/m2"->"J/m2" ; FAIL RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-9627- VAR("T1_RSN_RF1"[96,1,100]:double:DIMS=3:ATTRS=3) ; OK RUN_CSTS/results/NAM_SURF_ATM_ALDTHRES_OFFLINE_SERIAL__safe__v8a_TEB_PROGNOSTIC.OUT.nc.log-9686- ATTR("_FillValue":double:1) = +1.000000000000000E+020 ; OK Actual problem is fixed in rj_tech_v8 branch: http://opensource.cnrm-game-meteo.fr/projects/surfex/repository/revisions/2710/diff/ But having integraded 2710 and 2709 revisions would have broken binary reproducibility, thus skipped in V8_RJTECH branch. Having only these changes was achieved mainly by moving READ_SURFX2COV/WRITE_SURFX2COV out of read_surf.F90/write_surf.F90 interfaces. Main goal was to not to have badly writen NETCDF global attributes. Other *.nc *.txt *.lfi(with -DLFI_BFD define) *.fa(with -DLFI_BFB define) outputs are binary identical.