Hey, check out my Modern Python Projects course. It's an extended version of this workshop!
“How to properly structure a Python project?” - is a very common question. And it’s a difficult one. There is no silver bullet that will work for each project.
Python doesn’t enforce any structure. As long as you know what you are doing, you can organize your project however you like. From the tools and frameworks that I know of, only Django will try to put some scaffolding for you (but you are not forced to use it).
When you first think about the structure of the project, it might not sound complicated. And usually, it won’t be, but if you are not careful, you will probably hit one of the two common problems:
ModuleNotFoundError)ImportError)When you try to import a module, Python will search in 3 places:
python ~/my_module/scripts/start_server.py, Python will look for modules to import inside the ~/my_module/scripts/ directory. If you are starting an interactive session (by running $ python), Python will use the current directory instead.PYTHONPATH environment variableAnd the best way to see all the paths that Python will look inside is to check sys.path:
import sys
print(sys.path)
If you are getting ModuleNotFoundError, the first thing that you should do is to check the sys.path and see if it contains the directories that you think should be there. If it doesn’t, either move the file to a directory from the sys.path (recommended solution) or:
PYTHONPATH to include your directorysys.path.append("/my_module/scripts"). This is a hackish, but usually the fastest solution. Especially if you want to add the parent directory of the current folder (sys.path.append(".."))There are two ways to import modules in Python packages:
from my_module/models/user import get_userfrom ..models/user import get_userPersonally, I stick with absolute imports as this way it’s easier to see from where exactly something comes from. But if you prefer relative imports, you might run into this error:
ValueError: attempted relative import beyond top-level package
It usually happens when you try to import something from the parent/grandparent directory. This and this Stack Overflow questions can give you some explanations on what’s going on and how to fix this. The typical solution is to either:
In general, if you start getting those errors, I suggest to take a step back and rethink the whole architecture, before you start adding sys.path.append() randomly.
Circular import errors happen when two modules try to import something from each other, and Python gets stuck:
# file_a.py
from file_b import hello_world
def hello():
return "hello"
def first_program():
return hello_world()
first_program()
# file_b.py
from file_a import hello
def hello_world():
return hello() + " world"
When you run $ python file_a.py you will get the import error similar to this one:
Traceback (most recent call last):
File "file_a.py", line 1, in <module>
from file_b import hello_world
File "/my_module/file_b.py", line 2, in <module>
from file_a import hello
File "/my_module/file_a.py", line 1, in <module>
from file_b import hello_world
ImportError: cannot import name 'hello_world'
We are running file_a. file_a tries to import something from file_b - and “importing” in Python means “executing all the code from that file”. So we execute file_b, and it tries to import something from file_a. Which will try to import something from file_b, and so on. In the end, it’s a deadlock that Python can’t resolve.
This was an easy case of a circular import that we can solve, for example, by moving hello_world to file_a.py. Usually, those problems will be much more complicated (you will have multiple files involved in a circle).
The only way to deal with this issue is to refactor the application and remove circular imports. Usually, it involves adding another file to gather all your imports in one place. Here are two good examples of how to deal with this issue:
Let’s talk about what can help us avoid those problems. Let’s talk about what can help us avoid those problems.