django3 does not come with python3. If it is installed it was either manually installed or it was installed by dependency.
Run
# apt-mark showmanual
to see which packages you have installed manually.
When you write "There is 'python3-django' installed outside of the venv." what do you mean exactly by that? Is the package python3-django installed? Is there a directory of that name?
Is it installed?
# dpkg -l \*django\*
If it is, check
# apt-cache rdepends --installed python3-django
to find out why it has been installed.
If it is not installed by a package but it is in a standard path, e.g. you have the file /usr/bin/django-admin then double check with
# dpkg -S /usr/bin/django-admin
to make sure it doesn't belong to any package.
If the file is there but does not belong to any installed package, it wasn't cleaned out with autoremove nor purge, then you must have installed it in a different way. Then your system would be pretty messed up and I would highly recommend to start over again from scratch instead of trying to fix a broken system. It's always a big mess if some other non-packaged installation installs files into standard paths in conflict with package files...
Cheers.