A very lightweight plug-in infrastructure in Python

For some applications, run-time extensibility is a major requirement. There are lots of examples out there: browsers, media players, photo editors, etc. All these softwares can be easily extended with new functionality using plug-ins. How is this done?

It seems like complex stuff. Indeed, it really is, specially when you are using a bureaucratic language like Java or digging into the low level with C. However, when there aren’t security concerns, the extensions are of limited scope and a language with great introspection power like Python is being used, this can be a piece of cake =P.

Let’s see… suppose the plug-ins provide their services by means of the following contract interface:

class Plugin(object):

    def setup(self):
        raise NotImplementedError

    def teardown(self):
        raise NotImplementedError

    def run(self, *args, **kwards):
        raise NotImplementedError

Given this, a basic plug-in infrastructure should have as features:

• A way to auto-discover subclasses of Plugin on-demand at run-time
• A centralized way to access these subclasses

Thanks to the black magic of Python metaclasses (I’m assuming you are familiar with them; otherwise, see this excellent SO discussion), it’s very simple to implement those features:

class Plugin(object):

    class __metaclass__(type):

        def __init__(cls, name, base, attrs):
            if not hasattr(cls, 'registered'):
                cls.registered = []
            else:
                cls.registered.append(cls)

   ...

Now, every time a subclass of Plugin is defined, it is added to Plugin.registered so that there’s a centralized way to access the plug-ins. But the problem of auto-discovery still remains because a plug-in class must be defined to the metaclass trick work, which requires the import of the modules containing the plug-in classes definitions. However, this is easy to fix:

import imp
import logging
import pkgutil

class Plugin(object):

    class __metaclass__(type):

        ...

    @classmethod
    def load(cls, *paths):
        paths = list(paths)
        cls.registered = []
        for _, name, _ in pkgutil.iter_modules(paths):
            fid, pathname, desc = imp.find_module(name, paths)
            try:
                imp.load_module(name, fid, pathname, desc)
            except Exception as e:
                logging.warning("could not load plugin module '%s': %s",
                                pathname, e.message)
            if fid:
                fid.close()

    ...

The class method load forces the import of any module found in a path list. Consequently, an explicit import is not needed in order to discover the plug-ins, making the application itself fully decoupled of them.

As an usage example, if you had defined subclasses SamplePlugin1 and SamplePlugin2 of Plugin in some module located at "./plugins/", you could access them this way:

>>> Plugin.load("plugins/")
>>> Plugin.registered
[<class 'SamplePlugin1'>, <class 'SamplePlugin2'>]

Of course, this is extremely simple. There’s no sandbox (which implies security issues) and the plug-ins are passive (the application call their methods, instead of them calling methods of a plug-in API). However, for many programs this is enough and anything more complex would be over-engineer.

That’s it. This is a common problem in software engineering, so I hope this is useful. =)