Python is shipped with a lot of modules, we can say that there are python modules for almost everything you need. You need to work with robots, or some hardware in a spaceship, you will find a Python module for that. In this post, we will talk about Python modules and how to create, install, upgrade, reload and use them. The Python module allows you to organize your Python code and makes it easier to maintain later. Inside a module, you can define functions, classes, and variables. You can send your module to your friends and share it with others so they can use it.
Standard Library Modules
Python shipped with a large collection of modules. There are more than 200 modules for common programming tasks.
These modules called the standard library, you can find them in most installations on all platforms.
These modules can be used by importing them using the import statement as we’ve seen in the previous post Python programming basics.
How import statement works
Some newcomers might think that the import statement works like the include directive in C language or require in PHP, but that far from the truth because the other languages all it does is just grabbing the file content.
In Python, the situation is little different. There are three steps performed the first time a program imports a given file.
- Find the module’s file.
- Compile it to bytecode if needed.
- Build the objects from the bytecode.
You type the imported file without the extension and without its full directory path.
Then Python compiles it to bytecode. If the source file is modified, the bytecode file (.pyc file) regenerated automatically.
You deploy your Python programs as just bytecode files and avoid sending the source and Python will load the bytecode files.
The final step of the import operation is to execute the generated bytecode.
Python Search for a Module
As we said earlier, you just type the name of the module without extension or complete path and Python will take care of the rest.
So where Python does the search for the modules?
- The home directory of your program.
- PATH environment variable directories.
- Standard library directories.
- Any .pth files.
- The site-packages home of your third-party extensions.
The combination of these items becomes sys.path list.
Actually, sys.path is the path where Python searches for modules.
You can get your path like this:
You can add new custom paths to search for modules using sys.path.append or
When you type import mymodule, Python will load any of the followings:
- Source code file named mymodule.py.
- Bytecode file named mymodule.pyc.
- Optimized bytecode file named mymodule.pyo.
- A directory named mymodule, for package imports.
- Compiled extension module like mymodule.so on Linux, or mymodule.dll on Windows.
- Compiled module which is written in C.
- ZIP file which is automatically extracted.
- Jython files which are Python for Java.
- .NET component, written in the IronPython version of Python
If you have two different files like mymodule.py and mymodule.dll, Python will load the first file found in the search path during the left-to- right search of sys.path. But what if python found both files in the same directory?
Any one of them can be loaded first, so stay away from falling in this situation.
Creating a Python Module
To create a Python module, nothing special is required, just open your editor and type your code and save your file with .py extension.
All defined functions of the module become its attributes.
For example, if you define a function like the following inside your module:
You can call the function myfunc from that imported module inside your code.
Keep in mind that when creating a module, the module name becomes a variable (without the extension) inside your Python program, so you can’t name a module if.py. Always avoid Python reserved keywords.
Installing Python Modules from Source
The module includes a special file named setup.py that handles the details.
You can install Python modules from the source like this:
$ python setup.py install
This command will install the module within the site-packages folder which could be like this C:\Python27\lib\site-packages
Or like this
Install Python Modules Using pip
You can use pip utility to install modules and packages from the PyPI repository.
Most developers prefer installing modules using this method. The best thing about pip is that it handles dependency checking.
You can install Python modules using pip like this:
$ pip install numpy
Also, you can upgrade your installed modules using pip like this:
$ pip install --upgrade numpy
This command will update all the dependencies of the modules.
After writing your awesome modules or installing them, now you need to reuse it anywhere in your code, or maybe send it to your friends so they can use it, or share it with others on the web. How to import modules in your code?
There are two ways to achieve that, the first way is to use import statement, and the second way is to use from statement. Both of them will find, compile and run.
The major difference is that import statement fetches the whole module, while from statement fetches the specified attributes from the modules.
from mymodule import myfunc
The import Statement
Because import statement loads all attributes from the module so if you want to use any attribute, you have to use the module name like this:
The from Statement
As we said before, the from statement loads only the specified attributes.
Using the from statement makes it less typing because we don’t have to specify the whole module name when calling it.
from mymodule import myfunc
Actually, behind the scenes the from statement is an extension of the import statement, the entire file is also loaded, but you will be provided with all attributes directly from your code.
You can write the from statement like this:
from mymodule import *
The asterisk here means import everything at the top level from the module.
The Danger of Using from Statement
Some Python developers recommend using the import statement over using the from statement.
But is this true? Does the from statement not safe?
Well, if you use the from to import variables, it will overwrite any variable that has the same name if exist without notice.
This problem, of course, doesn’t happen with the import statement. Since you type the module name to get its attributes, so no collision happens.
You should take care when using the from statement. Consider the following situation
print("Hello from module 1")
print("Hello from module 2")
Now, let’s import the above modules and try to use them
from module1 import myfunc
from module2 import myfunc
The output here will be from module 2 since the function will be overwritten.
Actually, this is not a common thing to encounter, but anyway, there is a solution for that.
To avoid the name collision happened in the above example, you can use aliases like this:
from module1 import myfunc as myfunc1
from module2 import myfunc as myfunc2
It’s like a renaming the imports. Sometimes developers use aliases to shorten the module names they are importing when the module name is bigger.
After using aliases, do you think that using the from statements is dangerous? For me, it’s not.
Import Module Scope
You know that we can’t access the module attributes unless we import it.
Consider the following situation
M = 10
M = 20
M = 50
When you run module2, the module1.myfunc() changes the M variable in module1 not the M in module2.
The result will be 50 20.
That means the global scope for module1.myfunc is the file that enclosing it, not the file it runs from.
The import statements don’t promote the visibility of the attributes, and the imported file can’t see the attributes in the importing file.
If you try to re-import the module after importing it on the top of your code, Python will fetch the already loaded module again.
To reload a module, you can use the reload function that can achieve that.
Why do we need to reload the imported module?
The answer is so simple. This allows some parts of your program to be modified without restarting your program. You can imagine any situation that needs dynamic customization.
Python can only reload modules written in Python only. So compiled extensions (like the discussed previously) can’t be reloaded.
You MUST import the module first before you reload it. You can reload modules like this:
from importlib import reload
You can change the code of module1 in your text editor and it will be reloaded.
Reloading modules is a very powerful feature in Python if it is used carefully.
I hope you find the post useful and interesting. Keep coming back.