Project structure

“How to properly structure a Python project?” - is a very common question. And it’s a difficult one. There is no silver bullet that will work for each project.

Python doesn’t enforce any structure. As long as you know what you are doing, you can organize your project however you like. From the tools and frameworks that I know of, only Django will try to put some scaffolding for you (but you are not forced to use it).

When you first think about the structure of the project, it might not sound complicated. And usually, it won’t be, but if you are not careful, you will probably hit one of the two common problems:

  • The file you try to import is not on the import path (usually resulting in ModuleNotFoundError)
  • Circular import errors (usually resulting in ImportError)

Importing modules

When you try to import a module, Python will search in 3 places:

  • The directory containing the script that was used to invoke Python interpreter. In other words, if you run python ~/my_module/scripts/start_server.py, Python will look for modules to import inside the ~/my_module/scripts/ directory. If you are starting an interactive session (by running $ python), Python will use the current directory instead.
  • Additional paths specified in the PYTHONPATH environment variable
  • Installation-dependent default folders (it includes the folder with pip packages)

And the best way to see all the paths that Python will look inside is to check sys.path:

import sys
print(sys.path)

If you are getting ModuleNotFoundError, the first thing that you should do is to check the sys.path and see if it contains the directories that you think should be there. If it doesn’t, either move the file to a directory from the sys.path (recommended solution) or:

  • Modify the PYTHONPATH to include your directory
  • Modify the sys.path directly: sys.path.append("/my_module/scripts"). This is a hackish, but usually the fastest solution. Especially if you want to add the parent directory of the current folder (sys.path.append(".."))

Relative import traps

There are two ways to import modules in Python packages:

  • Absolute import: from my_module/models/user import get_user
  • Relative import: from ..models/user import get_user

Personally, I stick with absolute imports as this way it’s easier to see from where exactly something comes from. But if you prefer relative imports, you might run into this error:

ValueError: attempted relative import beyond top-level package

It usually happens when you try to import something from the parent/grandparent directory. This and this Stack Overflow questions can give you some explanations on what’s going on and how to fix this. The typical solution is to either:

  • move the main file one directory up (the main file is the file that you run to start your application) - if you are working on a Python package
  • or fiddle with sys.path if it’s not a package

In general, if you start getting those errors, I suggest to take a step back and rethink the whole architecture, before you start adding sys.path.append() randomly.

Circular import errors

Circular import errors happen when two modules try to import something from each other, and Python gets stuck:

# file_a.py
from file_b import hello_world

def hello():
    return "hello"

def first_program():
    return hello_world()

first_program()
# file_b.py
from file_a import hello

def hello_world():

    return hello() + " world"

When you run $ python file_a.py you will get the import error similar to this one:

Traceback (most recent call last):
  File "file_a.py", line 1, in <module>
    from file_b import hello_world
  File "/my_module/file_b.py", line 2, in <module>
    from file_a import hello
  File "/my_module/file_a.py", line 1, in <module>
    from file_b import hello_world
ImportError: cannot import name 'hello_world'

We are running file_a. file_a tries to import something from file_b - and “importing” in Python means “executing all the code from that file”. So we execute file_b, and it tries to import something from file_a. Which will try to import something from file_b, and so on. In the end, it’s a deadlock that Python can’t resolve.

This was an easy case of a circular import that we can solve, for example, by moving hello_world to file_a.py. Usually, those problems will be much more complicated (you will have multiple files involved in a circle).

The only way to deal with this issue is to refactor the application and remove circular imports. Usually, it involves adding another file to gather all your imports in one place. Here are two good examples of how to deal with this issue:

Let’s talk about what can help us avoid those problems. Let’s talk about what can help us avoid those problems.