I want to train a Mask R-CNN Model with a Nvidia RTX 4090 GPU but it seems impossible.
There seems to be an issue with the weightloading.
I tried following implementations:
- https://github.com/matterport/Mask_RCNN
- https://github.com/BupyeongHealer/Mask_RCNN_tf_2.x
- https://github.com/tomgross/Mask_RCNN/tree/tensorflow-2.0
- https://github.com/mrk1992/mask-rcnn-tf2-us
- https://github.com/ahmedfgad/Mask-RCNN-TF2
- https://github.com/AarohiSingla/Mask-R-CNN-using-Tensorflow2
With Python 3.7.12 and Tensorflow 1.14/1.15 and Keras 2.3.1 Mask-RCNN works very well.
BUT I can’t use the RTX 4090 with these Versions.
I need at least Tensorflow 2.12.1 to get the right CUDA version of the RTX 4090 (CUDA 11.8).
So when I use Tensorflow 2.12.1 or 2.13.1 there is a special issue, where the loading of the weights is wrong. The weights always gets loaded randomly.
So I train the model for 1 epoch, then I load the weights 3 times and the weights are different.
See the result looking like that:
Results when loading the same weights 3 Times
Note that the results of course will look sketch and weird because I don’t train the mode, but I expected to become the same results for the same weights.
I tried that with Tom Gross’s Implementation and with mrk1992’s Implementation and both got this bug.
I looked many issues but didn’t found something. Maybe you can help.
Does anybody knows how to fix that?
Recreation the bug
For Recreation I give you my conda environment.
Create a environment.yml:
name: maskrcnn
channels:
- conda-forge
- defaults
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=2_gnu
- _tflow_select=2.3.0=mkl
- absl-py=0.15.0=pyhd8ed1ab_0
- aiohttp=3.7.4.post0=py37h5e8e339_1
- alembic=1.13.1=pyhd8ed1ab_0
- alsa-lib=1.2.8=h166bdaf_0
- anyio=3.7.1=pyhd8ed1ab_0
- aom=3.5.0=h27087fc_0
- argon2-cffi=23.1.0=pyhd8ed1ab_0
- argon2-cffi-bindings=21.2.0=py37h540881e_2
- astor=0.8.1=pyh9f0ad1d_0
- async-timeout=3.0.1=py_1000
- attr=2.5.1=h166bdaf_1
- attrs=23.2.0=pyh71513ae_0
- backcall=0.2.0=pyh9f0ad1d_0
- backports=1.0=pyhd8ed1ab_3
- backports.functools_lru_cache=2.0.0=pyhd8ed1ab_0
- bcrypt=3.2.2=py37h540881e_0
- beautifulsoup4=4.12.3=pyha770c72_0
- binutils_impl_linux-64=2.40=ha885e6a_0
- binutils_linux-64=2.40=hdade7a5_3
- bleach=6.1.0=pyhd8ed1ab_0
- blinker=1.6.3=pyhd8ed1ab_0
- bottleneck=1.3.5=py37hda87dfa_0
- brotli=1.1.0=hd590300_1
- brotli-bin=1.1.0=hd590300_1
- brotli-python=1.0.9=py37hd23a5d3_7
- bzip2=1.0.8=hd590300_5
- c-ares=1.28.1=hd590300_0
- ca-certificates=2024.2.2=hbcca054_0
- cached-property=1.5.2=hd8ed1ab_1
- cached_property=1.5.2=pyha770c72_1
- cachetools=5.3.3=pyhd8ed1ab_0
- cairo=1.16.0=ha61ee94_1014
- certifi=2024.2.2=pyhd8ed1ab_0
- cffi=1.15.1=py37h43b0acd_1
- chardet=4.0.0=py37h89c1867_3
- charset-normalizer=3.3.2=pyhd8ed1ab_0
- click=8.1.3=py37h89c1867_0
- cloudpickle=2.2.1=pyhd8ed1ab_0
- colorama=0.4.6=pyhd8ed1ab_0
- colorlog=6.7.0=py37h89c1867_0
- comm=0.2.2=pyhd8ed1ab_0
- cryptography=38.0.2=py37h5994e8b_1
- cycler=0.11.0=pyhd8ed1ab_0
- cython=0.29.32=py37hd23a5d3_0
- cytoolz=0.12.0=py37h540881e_0
- dask-core=2022.2.0=pyhd8ed1ab_0
- dbus=1.13.6=h5008d03_3
- debugpy=1.6.3=py37hd23a5d3_0
- decorator=5.1.1=pyhd8ed1ab_0
- defusedxml=0.7.1=pyhd8ed1ab_0
- dill=0.3.8=pyhd8ed1ab_0
- distro=1.9.0=pyhd8ed1ab_0
- docker-py=6.1.3=pyhd8ed1ab_0
- entrypoints=0.4=pyhd8ed1ab_0
- exceptiongroup=1.2.0=pyhd8ed1ab_2
- expat=2.6.2=h59595ed_0
- ffmpeg=4.4.2=gpl_h8dda1f0_112
- fftw=3.3.10=nompi_hc118613_108
- flask=1.1.2=pyh9f0ad1d_0
- font-ttf-dejavu-sans-mono=2.37=hab24e00_0
- font-ttf-inconsolata=3.000=h77eed37_0
- font-ttf-source-code-pro=2.038=h77eed37_0
- font-ttf-ubuntu=0.83=h77eed37_2
- fontconfig=2.14.2=h14ed4e7_0
- fonts-conda-ecosystem=1=0
- fonts-conda-forge=1=0
- fonttools=4.38.0=py37h540881e_0
- freeglut=3.2.2=h9c3ff4c_1
- freetype=2.12.1=h267a509_2
- fsspec=2023.1.0=pyhd8ed1ab_0
- gast=0.2.2=py_0
- gcc_impl_linux-64=13.2.0=h9eb54c0_7
- gcc_linux-64=13.2.0=h1ed452b_3
- geos=3.11.0=h27087fc_0
- gettext=0.22.5=h59595ed_2
- gettext-tools=0.22.5=h59595ed_2
- gitdb=4.0.11=pyhd8ed1ab_0
- gitpython=3.1.43=pyhd8ed1ab_0
- glib=2.80.2=hf974151_0
- glib-tools=2.80.2=hb6ce0ca_0
- gmp=6.3.0=h59595ed_1
- gnutls=3.7.9=hb077bed_0
- google-auth=2.23.0=pyh1a96a4e_0
- google-auth-oauthlib=0.4.6=pyhd8ed1ab_0
- google-pasta=0.2.0=pyh8c360ce_0
- graphite2=1.3.13=h59595ed_1003
- greenlet=1.1.3=py37hd23a5d3_0
- grpc-cpp=1.48.1=hc2bec63_1
- grpcio=1.48.1=py37h42e856d_1
- gst-plugins-base=1.21.3=h4243ec0_1
- gstreamer=1.21.3=h25f0c4b_1
- gstreamer-orc=0.4.38=hd590300_0
- gunicorn=20.1.0=py37h89c1867_2
- gxx_impl_linux-64=13.2.0=h2a599c4_7
- gxx_linux-64=13.2.0=he8deefe_3
- h5py=3.7.0=nompi_py37hf1ce037_101
- harfbuzz=5.3.0=h418a68e_0
- hdf5=1.12.2=nompi_h4df4325_101
- icu=70.1=h27087fc_0
- idna=3.7=pyhd8ed1ab_0
- imagecodecs-lite=2019.12.3=py37hc105733_5
- imageio=2.34.1=pyh4b66e23_0
- imgaug=0.4.0=pyhd8ed1ab_1
- importlib-metadata=4.11.4=py37h89c1867_0
- importlib_resources=6.0.0=pyhd8ed1ab_0
- imutils=0.5.4=py37h89c1867_2
- ipykernel=6.16.2=pyh210e3f2_0
- ipython=7.33.0=py37h89c1867_0
- ipython_genutils=0.2.0=py_1
- ipywidgets=8.1.2=pyhd8ed1ab_1
- itsdangerous=2.1.2=pyhd8ed1ab_0
- jack=1.9.22=h11f4161_0
- jasper=2.0.33=h0ff4b12_1
- jedi=0.19.1=pyhd8ed1ab_0
- jinja2=3.1.4=pyhd8ed1ab_0
- joblib=1.2.0=pyhd8ed1ab_0
- jpeg=9e=h0b41bf4_3
- jsonschema=4.17.3=pyhd8ed1ab_0
- jupyter=1.0.0=pyhd8ed1ab_10
- jupyter_client=7.4.9=pyhd8ed1ab_0
- jupyter_console=6.5.1=pyhd8ed1ab_0
- jupyter_core=4.11.1=py37h89c1867_0
- jupyter_server=1.23.4=pyhd8ed1ab_0
- jupyterlab_pygments=0.3.0=pyhd8ed1ab_1
- jupyterlab_widgets=3.0.10=pyhd8ed1ab_0
- keras=2.3.1=py37_0
- keras-applications=1.0.8=py_1
- keras-preprocessing=1.1.2=pyhd8ed1ab_0
- kernel-headers_linux-64=2.6.32=he073ed8_17
- keyutils=1.6.1=h166bdaf_0
- kiwisolver=1.4.4=py37h7cecad7_0
- krb5=1.20.1=h81ceb04_0
- lame=3.100=h166bdaf_1003
- lcms2=2.14=h6ed2654_0
- ld_impl_linux-64=2.40=h55db66e_0
- lerc=4.0.0=h27087fc_0
- libabseil=20220623.0=cxx17_h05df665_6
- libaec=1.1.3=h59595ed_0
- libasprintf=0.22.5=h661eb56_2
- libasprintf-devel=0.22.5=h661eb56_2
- libblas=3.9.0=20_linux64_openblas
- libbrotlicommon=1.1.0=hd590300_1
- libbrotlidec=1.1.0=hd590300_1
- libbrotlienc=1.1.0=hd590300_1
- libcap=2.67=he9d0100_0
- libcblas=3.9.0=20_linux64_openblas
- libclang=15.0.7=default_h127d8a8_5
- libclang13=15.0.7=default_h5d6823c_5
- libcups=2.3.3=h36d4200_3
- libcurl=8.1.2=h409715c_0
- libdb=6.2.32=h9c3ff4c_0
- libdeflate=1.14=h166bdaf_0
- libdrm=2.4.120=hd590300_0
- libedit=3.1.20191231=he28a2e2_2
- libev=4.33=hd590300_2
- libevent=2.1.10=h28343ad_4
- libexpat=2.6.2=h59595ed_0
- libffi=3.4.2=h7f98852_5
- libflac=1.4.3=h59595ed_0
- libgcc-devel_linux-64=13.2.0=hceb6213_107
- libgcc-ng=13.2.0=h77fa898_7
- libgcrypt=1.10.3=hd590300_0
- libgettextpo=0.22.5=h59595ed_2
- libgettextpo-devel=0.22.5=h59595ed_2
- libgfortran-ng=13.2.0=h69a702a_7
- libgfortran5=13.2.0=hca663fb_7
- libglib=2.80.2=hf974151_0
- libglu=9.0.0=he1b5a44_1001
- libgomp=13.2.0=h77fa898_7
- libgpg-error=1.49=h4f305b6_0
- libgpuarray=0.7.6=h7f98852_1003
- libiconv=1.17=hd590300_2
- libidn2=2.3.7=hd590300_0
- liblapack=3.9.0=20_linux64_openblas
- liblapacke=3.9.0=20_linux64_openblas
- libllvm11=11.1.0=he0ac6c6_5
- libllvm15=15.0.7=hadd5161_1
- libnghttp2=1.58.0=h47da74e_0
- libnsl=2.0.1=hd590300_0
- libogg=1.3.4=h7f98852_1
- libopenblas=0.3.25=pthreads_h413a1c8_0
- libopencv=4.6.0=py37hfe11ba8_3
- libopus=1.3.1=h7f98852_1
- libpciaccess=0.18=hd590300_0
- libpng=1.6.43=h2797004_0
- libpq=15.3=hbcd7760_1
- libprotobuf=3.20.1=h6239696_4
- libsanitizer=13.2.0=h6ddb7a1_7
- libsndfile=1.2.2=hc60ed4a_1
- libsodium=1.0.18=h36c2ea0_1
- libsqlite=3.45.3=h2797004_0
- libssh2=1.11.0=h0841786_0
- libstdcxx-devel_linux-64=13.2.0=hceb6213_107
- libstdcxx-ng=13.2.0=hc0a3c3a_7
- libsystemd0=253=h8c4010b_1
- libtasn1=4.19.0=h166bdaf_0
- libtiff=4.4.0=h82bc61c_5
- libtool=2.4.7=h27087fc_0
- libudev1=253=h0b41bf4_1
- libunistring=0.9.10=h7f98852_0
- libuuid=2.38.1=h0b41bf4_0
- libva=2.18.0=h0b41bf4_0
- libvorbis=1.3.7=h9c3ff4c_0
- libvpx=1.11.0=h9c3ff4c_3
- libwebp-base=1.4.0=hd590300_0
- libxcb=1.13=h7f98852_1004
- libxkbcommon=1.5.0=h79f4944_1
- libxml2=2.10.3=hca2bb57_4
- libzlib=1.2.13=hd590300_5
- llvmlite=0.39.1=py37h0761922_0
- locket=1.0.0=pyhd8ed1ab_0
- lz4-c=1.9.4=hcb278e6_0
- mako=1.3.5=pyhd8ed1ab_0
- markdown=3.6=pyhd8ed1ab_0
- markupsafe=2.1.1=py37h540881e_1
- matplotlib-base=3.5.3=py37hf395dca_2
- matplotlib-inline=0.1.7=pyhd8ed1ab_0
- mistune=3.0.2=pyhd8ed1ab_0
- mlflow=1.30.0=py37h02d9ccd_0
- mpg123=1.32.6=h59595ed_0
- multidict=6.0.2=py37h540881e_1
- multiprocess=0.70.14=py37h540881e_0
- munkres=1.1.4=pyh9f0ad1d_0
- mysql-common=8.0.33=hf1915f5_6
- mysql-libs=8.0.33=hca2cd23_6
- nbclassic=1.0.0=pyhb4ecaf3_1
- nbclient=0.7.0=pyhd8ed1ab_0
- nbconvert=7.6.0=pyhd8ed1ab_0
- nbconvert-core=7.6.0=pyhd8ed1ab_0
- nbconvert-pandoc=7.6.0=pyhd8ed1ab_0
- nbformat=5.8.0=pyhd8ed1ab_0
- ncurses=6.5=h59595ed_0
- nest-asyncio=1.6.0=pyhd8ed1ab_0
- nettle=3.9.1=h7ab15ed_0
- networkx=2.6.3=pyhd8ed1ab_1
- nomkl=1.0=h5ca1d4c_0
- notebook=6.5.7=pyha770c72_0
- notebook-shim=0.2.4=pyhd8ed1ab_0
- nspr=4.35=h27087fc_0
- nss=3.100=hca3bf56_0
- numba=0.56.3=py37hf081915_0
- numexpr=2.8.3=py37h85a3170_100
- numpy=1.21.6=py37h976b520_0
- oauthlib=3.2.2=pyhd8ed1ab_0
- opencv=4.6.0=py37h89c1867_3
- openh264=2.3.1=hcb278e6_2
- openjpeg=2.5.0=h7d73246_1
- openssl=3.1.5=hd590300_0
- opt_einsum=3.3.0=pyhc1e730c_2
- p11-kit=0.24.1=hc5aa10d_0
- packaging=21.3=pyhd8ed1ab_0
- pandas=1.3.5=py37h8c16a72_0
- pandoc=3.2=ha770c72_0
- pandocfilters=1.5.0=pyhd8ed1ab_0
- paramiko=3.4.0=pyhd8ed1ab_0
- parso=0.8.4=pyhd8ed1ab_0
- partd=1.4.1=pyhd8ed1ab_0
- pcre2=10.43=hcad00b1_0
- pexpect=4.9.0=pyhd8ed1ab_0
- pickleshare=0.7.5=py_1003
- pillow=9.2.0=py37h850a105_2
- pip=24.0=pyhd8ed1ab_0
- pixman=0.43.2=h59595ed_0
- pkgutil-resolve-name=1.3.10=pyhd8ed1ab_1
- prometheus_client=0.17.1=pyhd8ed1ab_0
- prometheus_flask_exporter=0.23.0=pyhd8ed1ab_0
- prompt-toolkit=3.0.42=pyha770c72_0
- prompt_toolkit=3.0.42=hd8ed1ab_0
- protobuf=3.20.1=py37hd23a5d3_0
- psutil=5.9.3=py37h540881e_0
- pthread-stubs=0.4=h36c2ea0_1001
- ptyprocess=0.7.0=pyhd3deb0d_0
- pulseaudio=16.1=hcb278e6_3
- pulseaudio-client=16.1=h5195f5e_3
- pulseaudio-daemon=16.1=ha8d29e2_3
- py-opencv=4.6.0=py37h25bab4e_3
- pyasn1=0.5.1=pyhd8ed1ab_0
- pyasn1-modules=0.3.0=pyhd8ed1ab_0
- pycocotools=2.0.4=py37hda87dfa_2
- pycparser=2.21=pyhd8ed1ab_0
- pygments=2.17.2=pyhd8ed1ab_0
- pygpu=0.7.6=py37hb1e94ed_1003
- pyjwt=2.8.0=pyhd8ed1ab_1
- pynacl=1.5.0=py37h540881e_1
- pyopenssl=23.2.0=pyhd8ed1ab_1
- pyparsing=3.1.2=pyhd8ed1ab_0
- pyrsistent=0.18.1=py37h540881e_1
- pysocks=1.7.1=py37h89c1867_5
- python=3.7.12=hf930737_100_cpython
- python-dateutil=2.9.0=pyhd8ed1ab_0
- python-fastjsonschema=2.19.1=pyhd8ed1ab_0
- python_abi=3.7=4_cp37m
- pytz=2022.7.1=pyhd8ed1ab_0
- pyu2f=0.1.5=pyhd8ed1ab_0
- pywavelets=1.3.0=py37hda87dfa_1
- pywin32-on-windows=0.1.0=pyh1179c8e_3
- pyyaml=6.0=py37h540881e_4
- pyzmq=24.0.1=py37h0c0c2a8_0
- qt-main=5.15.6=hf6cd601_5
- qtconsole-base=5.4.4=pyha770c72_0
- qtpy=2.4.1=pyhd8ed1ab_0
- querystring_parser=1.2.4=py_0
- re2=2022.06.01=h27087fc_1
- readline=8.2=h8228510_1
- requests=2.31.0=pyhd8ed1ab_0
- requests-oauthlib=2.0.0=pyhd8ed1ab_0
- rsa=4.9=pyhd8ed1ab_0
- ruamel.yaml=0.17.10=py37h5e8e339_0
- ruamel.yaml.clib=0.2.6=py37h540881e_1
- scikit-build=0.17.6=pyh4af843d_0
- scikit-image=0.19.3=py37hfb7772e_1
- scikit-learn=1.0.2=py37hf9e9bfc_0
- scipy=1.7.3=py37hf2a6cf1_0
- send2trash=1.8.3=pyh0d859eb_0
- setproctitle=1.3.2=py37h540881e_0
- setuptools=69.0.3=pyhd8ed1ab_0
- shapely=1.8.5=py37ha4e3bd1_0
- six=1.16.0=pyh6c4a22f_0
- smmap=5.0.0=pyhd8ed1ab_0
- sniffio=1.3.1=pyhd8ed1ab_0
- soupsieve=2.3.2.post1=pyhd8ed1ab_0
- sqlalchemy=1.4.42=py37h540881e_0
- sqlite=3.45.3=h2c6b66d_0
- sqlparse=0.4.4=pyhd8ed1ab_0
- svt-av1=1.4.1=hcb278e6_0
- sysroot_linux-64=2.12=he073ed8_17
- tensorboard=2.8.0=pyhd8ed1ab_1
- tensorboard-data-server=0.6.1=py37h52d8a92_0
- tensorboard-plugin-wit=1.8.1=pyhd8ed1ab_0
- tensorflow=2.0.0=mkl_py37h66b46cc_0
- tensorflow-base=2.0.0=mkl_py37h9204916_0
- tensorflow-estimator=2.6.0=py37hcd2ae1e_0
- termcolor=1.1.0=pyhd8ed1ab_3
- terminado=0.17.1=pyh41d4057_0
- theano=1.0.4=py37hf484d3e_1000
- threadpoolctl=3.1.0=pyh8a188c0_0
- tifffile=2020.6.3=py_0
- tinycss2=1.3.0=pyhd8ed1ab_0
- tk=8.6.13=noxft_h4845f30_101
- tomli=2.0.1=pyhd8ed1ab_0
- toolz=0.12.1=pyhd8ed1ab_0
- tornado=6.2=py37h540881e_0
- tqdm=4.66.4=pyhd8ed1ab_0
- traitlets=5.9.0=pyhd8ed1ab_0
- typing-extensions=4.7.1=hd8ed1ab_0
- typing_extensions=4.7.1=pyha770c72_0
- unicodedata2=14.0.0=py37h540881e_1
- urllib3=1.26.18=pyhd8ed1ab_0
- wcwidth=0.2.10=pyhd8ed1ab_0
- webencodings=0.5.1=pyhd8ed1ab_2
- websocket-client=1.6.1=pyhd8ed1ab_0
- werkzeug=0.16.1=py_0
- wheel=0.42.0=pyhd8ed1ab_0
- widgetsnbextension=4.0.10=pyhd8ed1ab_0
- wrapt=1.14.1=py37h540881e_0
- x264=1!164.3095=h166bdaf_2
- x265=3.5=h924138e_3
- xcb-util=0.4.0=h516909a_0
- xcb-util-image=0.4.0=h166bdaf_0
- xcb-util-keysyms=0.4.0=h516909a_0
- xcb-util-renderutil=0.3.9=h166bdaf_0
- xcb-util-wm=0.4.1=h516909a_0
- xkeyboard-config=2.38=h0b41bf4_0
- xorg-fixesproto=5.0=h7f98852_1002
- xorg-inputproto=2.3.2=h7f98852_1002
- xorg-kbproto=1.0.7=h7f98852_1002
- xorg-libice=1.1.1=hd590300_0
- xorg-libsm=1.2.4=h7391055_0
- xorg-libx11=1.8.4=h0b41bf4_0
- xorg-libxau=1.0.11=hd590300_0
- xorg-libxdmcp=1.1.3=h7f98852_0
- xorg-libxext=1.3.4=h0b41bf4_2
- xorg-libxfixes=5.0.3=h7f98852_1004
- xorg-libxi=1.7.10=h7f98852_0
- xorg-libxrender=0.9.10=h7f98852_1003
- xorg-renderproto=0.11.1=h7f98852_1002
- xorg-xextproto=7.3.0=h0b41bf4_1003
- xorg-xproto=7.0.31=h7f98852_1007
- xz=5.2.6=h166bdaf_0
- yaml=0.2.5=h7f98852_2
- yarl=1.7.2=py37h540881e_2
- zeromq=4.3.5=h59595ed_1
- zipp=3.15.0=pyhd8ed1ab_0
- zlib=1.2.13=hd590300_5
- zstd=1.5.6=ha6fb4c9_0
- pip:
- databricks-cli==0.18.0
- tabulate==0.9.0
prefix: /home/local-admin/.conda/envs/maskrcnn
Then just type
conda env create -n maskrcnn python=3.8.19 -f environment.yml
Also download a Maks RCNN Model:
git clone -b tensorflow-2.0 https://github.com/tomgross/Mask_RCNN.git Mask_RCNN
Here is Code to test the model:
import sys
sys.path.append("./Mask_RCNN")
import os
import numpy as np
import cv2
import imutils
import matplotlib.pyplot as plt
import tensorflow as tf
from IPython.display import clear_output
from mrcnn.config import Config
from mrcnn import model as modellib
from mrcnn import visualize
img_path = f"./test.jpg"
image = cv2.imread(img_path)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
image = imutils.resize(image, width=512)
plt.figure(figsize=(10, 10))
plt.axis("off")
plt.imshow(image);
class TestConfig(Config):
NAME = "mask-rcnn test"
GPU_COUNT = 1
IMAGES_PER_GPU = 1
NUM_CLASSES = 2
# BACKBONE = "resnet50"
# IMAGE_MIN_DIM = 800
# IMAGE_MAX_DIM = 1024
CLASS_NAMES = ['BG', 'FG']
TEST_MODEL_PATH = "./test_model.h5"
config = TestConfig()
# config.display()
# create new model with random weights
model = modellib.MaskRCNN(mode="training", config=config, model_dir=os.getcwd())
model.keras_model.save_weights(TEST_MODEL_PATH)
clear_output()
# prepare visualization
fig, ax = plt.subplots(ncols=3, nrows=1, figsize=(20, 5))
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=0.01, hspace=None)
# load the created random weights 3 times
weights = []
differences = []
for i in range(3):
model = modellib.MaskRCNN(mode="inference", config=config, model_dir=os.getcwd())
model.load_weights(TEST_MODEL_PATH, by_name=True)
result = model.detect([image], verbose=0)[0]
ax[i].set_title(f"{i}. Results")
ax[i].axis("off")
visualize.display_instances(image, result['rois'], result['masks'], result['class_ids'], CLASS_NAMES, result['scores'],
ax=ax[i], show_mask=True, show_bbox=True)
weights += [model.keras_model.get_weights()]
clear_output()
# detect differences between the models
for cur_weights_1 in weights:
cur_differences = []
for cur_weights_2 in weights:
all_weigt_differences = [np.abs(w1-w2) for w1, w2 in zip(cur_weights_1, cur_weights_2)]
all_layer_differences = [np.sum(diff) for diff in all_weigt_differences]
cur_differences += [np.sum(all_layer_differences)]
differences += [cur_differences]
complete_sum_difference = np.sum(differences)
if complete_sum_difference == 0.0:
print("Congratulations!nYour implementation works and is ready to segment!")
else:
print("Difference detected. Your Mask RCNN seem to have an problem.")
print("Check if you correctly applied all steps of the installation.")
print("nMore detail of differences:")
print("Absolute Weight Differences:", complete_sum_difference)
print("Comparison: Weight 1, Weight 2, Weight 3")
for idx in range(len(differences)):
print(f"Weight {idx+1}: {differences[idx]}")
# try to remove the test model file
try:
os.remove(TEST_MODEL_PATH)
except Exception:
print(f"Wasn't able to delete the test model at: {TEST_MODEL_PATH}")
plt.show();
And here is the test image:
Test Image
I hope this makes the recreation easy for you.
Does anyone have a solution?
I tried:
- version from mrk1993 + add Anchor Layer Class instea of Lambda + numpy has no attribute bool
- Python: 3.8.19
- Tensorflow-gpu 2.12.1
- Keras –
- backbone 50 + imagesize 1024
- Different results with same weights
- TF2 Version von TomGross + add Anchor Layer Class instea of Lambda + numpy has no attribute bool
- Python: 3.8.19
- Tensorflow-gpu 2.12.1
- Keras –
- backbone 50 + imagesize 1024
- Different results with same weights
- TF2 Version von TomGross + add Anchor Layer Class instea of Lambda + numpy has no attribute bool
- Python: 3.8.19
- Tensorflow-gpu 2.13.1
- Keras –
- Different results with same weights
I also tried tensorflow versions smaller than 2.12.1 but these doesn’t work. You can also check the CUDA versions of the tensorflow versions here: https://www.tensorflow.org/install/source.
I expect same weights when loading the same weights 3 times. To verify, before I trained and used the model on a GTX 1080 Ti and there with Python 3.7.12 and tensorflow 1.15 and Keras 2.3.1 this code above succeed and the weights was equaly, which is also expected.
With the new GPU this looks different.
New contributor
xXAI-botXx is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.