Misakichi’s ログblog

好きなものを紹介したり備忘録のため

Solution to ''Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR'''

My Environment

RTX 2080
Ubuntu 18.04
CUDA 10.0
Nvidia Drivers 450.51
tensorflow-gpu 1.13.1
CuDNN 7.4.2.24-1+cuda10.0
Python 3.6.0


Cuda and cudnn install

qiita.com
I followed this site, and tensorflow-gpu recognized RTX2080.
Cuda initialization is also Ok

Problem: Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

the problem is cudnn.
Whenever I run the code, it stops with error message
'ensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR'
Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR · Issue #24496 · tensorflow/tensorflow


....I repeatedly verify version compatibility, but it is correct.

the solution

github.com

especially,
oscarlinux commented on Feb 19, 2019

ok, made it work in tf-nightly-gpu-2.0-preview and ipython notebook adding this to my code:

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession

config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

appendix

some say 'export TF_FORCE_GPU_ALLOW_GROWTH=true'
in terminal will fix. But didnt work in my environment.
It seems it was a memory problem.
But the reason why is ambiguous.

package versions

Lastly, package versions etc are describes
RTX 2080
Ubuntu 18.04
CUDA 10.0
Nvidia Drivers 450.51
tensorflow-gpu 1.13.1
CuDNN 7.4.2.24-1+cuda10.0
Python 3.6.0
machine@machine-System-Product-Name:~$ pip list
Package Version

                                            • ---------

absl-py 0.10.0
astor 0.8.1
astunparse 1.6.3
bleach 1.5.0
cachetools 4.1.1
certifi 2020.6.20
chardet 3.0.4
cycler 0.10.0
gast 0.3.3
google-auth 1.20.1
google-auth-oauthlib 0.4.1
google-pasta 0.2.0
grpcio 1.31.0
h5py 2.10.0
html5lib 0.9999999
idna 2.10
importlib-metadata 1.7.0
joblib 0.16.0
Keras 2.2.4
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.2
kiwisolver 1.2.0
Markdown 3.2.2
matplotlib 3.3.1
mock 4.0.2
numpy 1.18.5
oauthlib 3.1.0
opt-einsum 3.3.0
pandas 0.25.3
Pillow 7.2.0
pip 20.2.2
protobuf 3.13.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pyparsing 2.4.7
python-dateutil 2.8.1
python-speech-features 0.6
pytz 2020.1
PyYAML 5.3.1
requests 2.24.0
requests-oauthlib 1.3.0
rsa 4.6
scikit-learn 0.23.2
scipy 1.4.1
setuptools 28.8.0
six 1.15.0
tensorboard 1.13.1
tensorboard-plugin-wit 1.7.0
tensorflow-estimator 1.13.0
tensorflow-gpu 1.13.1
tensorflow-tensorboard 1.5.1
termcolor 1.1.0
threadpoolctl 2.1.0
tqdm 4.48.2
urllib3 1.25.10
Werkzeug 1.0.1
wheel 0.35.1
wrapt 1.12.1
zipp 3.1.0