Difference between revisions of "Python setuptools"

From Exterior Memory
Jump to: navigation, search
(setuptools import problems (preliminary))
(Package and Egg Loading Trickery)
 
Line 31: Line 31:
  
 
;Problem:<code>import sphinxcontrib.restbuilder</code> always loaded this package from site-packages directory, despite that the <code>sphinxcontrib.restbuilder</code> module was also available in the current directory, and <code>sys.path</code> contained the current directory as first entry.
 
;Problem:<code>import sphinxcontrib.restbuilder</code> always loaded this package from site-packages directory, despite that the <code>sphinxcontrib.restbuilder</code> module was also available in the current directory, and <code>sys.path</code> contained the current directory as first entry.
;Cause:easy-install 'only' modifies <code>sys.path</code> (overriding <code>$PYTHONPATH</code>). setuptools takes it a step further and creates a <code>.pth<code> file that manipulates <code>sys.modules</code>.
+
;Cause:easy-install 'only' modifies <code>sys.path</code> (overriding <code>$PYTHONPATH</code>). setuptools takes it a step further and creates a <code>.pth</code> file that manipulates <code>sys.modules</code>.
 +
;Solution:<code>del sys.modules['sphinxcontrib']</code> if it exists.
  
 
First of all: Python eggs are a neat way to distribute different ''software packages'' inside the same ''Python package''. Consider the following two directory structures:
 
First of all: Python eggs are a neat way to distribute different ''software packages'' inside the same ''Python package''. Consider the following two directory structures:
Line 54: Line 55:
 
             swf/
 
             swf/
  
The second directory structure allows separate release cycles for all three packages. The downside is that all <code>*.egg</code> directories need to be included in <code>sys.path</code>.
+
The second directory structure allows separate release cycles for all three packages. The downside is that all <code>*.egg</code> directories need to be included in <code>sys.path</code>. However, this leads to problems when a user tries to <code>import sphinxcontrib</code>. Which of these directories should be included? setuptools accommodates for this by adding scripts in <code>.pth</code> files that select the correct packages. This is even the case if the directory structure is merged, as it is in the first example directory structure above.
  
 +
To be exact, setuptools adds an empty module to <code>sys.modules</code> with just the name and the desired path of each module inside an egg folder. As a consequence, as soon as that module is loaded, <code>sys.path</code> is not used, and that directory specified in <code>sys.modules</code> is used.
  
 +
Here is the <code>.pth</code> file installed by setuptools. It is slightly rewritten for clarity.
  
 
  import sys, types, os
 
  import sys, types, os
Line 67: Line 70:
 
  if (p not in mp):
 
  if (p not in mp):
 
     mp.append(p)
 
     mp.append(p)
 +
 +
In case a script must use another version of the module, manipulating <code>sys.path</code> won't help. Instead, take one of these steps:
 +
1. Delete the <code>.pth</code> file beforehand. (This may give problems when normally loading the module if it is in an egg file)
 +
2. Delete the entry in <code>sys.modules</code> for the given module: <code>if 'sphinxcontrib in sys.modules: del sys.modules['sphinxcontrib']</code>, and manipulate <code>sys.path</code> as usual.
 +
3. Replace the entry in <code>sys.modules</code> with one pointing to another path: <code>sys.modules['sphinxcontrib'].__dict__['__path__'] = desired_path</code>
  
  
Line 72: Line 80:
 
* [http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/ Python Packaging: Hate, hate, hate everywhere] by Armin Ronacher gives some background on the different tools (easy-install, setuptools, and pip).
 
* [http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/ Python Packaging: Hate, hate, hate everywhere] by Armin Ronacher gives some background on the different tools (easy-install, setuptools, and pip).
 
* Python [http://docs.python.org/library/site.html site module], which is responsible for defining <code>sys.path</code>, and executing the <code>.pth</code> files during Python boot time.
 
* Python [http://docs.python.org/library/site.html site module], which is responsible for defining <code>sys.path</code>, and executing the <code>.pth</code> files during Python boot time.
 
  
 
[[Category:Python]]
 
[[Category:Python]]

Latest revision as of 15:42, 28 September 2013

The most common way to distribute Python modules is using PyPI, the Python Package Index (formerly known as the Cheese shop).

The tool to download and install packages is pip. Predecessors like EasyInstall are no longer recommended, with only a few exceptions (in particular pip does not support binary distributions).

There are tool to build and upload packages is or setuptools. Make sure to use version 0.7 or later.

There are a few alternatives to setuptools: distutils is a tool in the standard library with limited functionality. The downside of both distutils and setuptools is that they use a script rather than a file format to store metadata, and force the developer an end-user to use the same tool to build and install the software. distutils2 (the module will be named `package`) is an attempt to move to a modern packaging system for Python, but has not been included as of Python 3.4.

Release Steps

Credit: These steps are based on the sphinxcontrib-aafig documentation.

In order to make a PyPi release, do the following steps:

  1. Make sure the repository is up-to date.
  2. Ensure the version is incremented:
    • setup.py must be updated
    • libary/__init__.py must be updated
    • doc/changes.rst must contain a summary of the changes
  3. Make sure all changes are committed, including the version number changes.
  4. Tag the sources with hg tag -m 'Tag mymodule-X.Y' mymodule-X.Y or git tag mymodule-X.Y.
  5. Push the code and tag: hg push or git push --tags origin
  6. Temporarily modify setup.cfg file to comment out the variables tag_build = dev and tag_date = true (do not commit this change).
  7. Register and upload the new release python setup.py register sdist upload.
  8. Generate the documentation with cd doc; make.
  9. Upload the new documentation (to PyPi or to github mypackage.wiki repository).

Package and Egg Loading Trickery

Python has a rather poor system of loading modules. Tools such as setuptools work around this problem, but sometimes these workarounds have unintended consequences.

Problem
import sphinxcontrib.restbuilder always loaded this package from site-packages directory, despite that the sphinxcontrib.restbuilder module was also available in the current directory, and sys.path contained the current directory as first entry.
Cause
easy-install 'only' modifies sys.path (overriding $PYTHONPATH). setuptools takes it a step further and creates a .pth file that manipulates sys.modules.
Solution
del sys.modules['sphinxcontrib'] if it exists.

First of all: Python eggs are a neat way to distribute different software packages inside the same Python package. Consider the following two directory structures:

site-packages/
    sphinxcontrib/
        blockdiag/
        restbuilder/
        swf/

and

site-packages/
    sphinxcontrib-blockdiag-1.2.egg
        sphinxcontrib/
            blockdiag/
    sphinxcontrib-restbuilder-0.1.egg
        sphinxcontrib/
            restbuilder/
    sphinxcontrib-swf-0.3.egg
        sphinxcontrib/
            swf/

The second directory structure allows separate release cycles for all three packages. The downside is that all *.egg directories need to be included in sys.path. However, this leads to problems when a user tries to import sphinxcontrib. Which of these directories should be included? setuptools accommodates for this by adding scripts in .pth files that select the correct packages. This is even the case if the directory structure is merged, as it is in the first example directory structure above.

To be exact, setuptools adds an empty module to sys.modules with just the name and the desired path of each module inside an egg folder. As a consequence, as soon as that module is loaded, sys.path is not used, and that directory specified in sys.modules is used.

Here is the .pth file installed by setuptools. It is slightly rewritten for clarity.

import sys, types, os
p = os.path.join(sys._getframe(1).f_locals['sitedir'], 'sphinxcontrib')
if os.path.exists(os.path.join(p,'__init__.py')):
    mp = []
else:
    m = sys.modules.setdefault('sphinxcontrib', types.ModuleType('sphinxcontrib'))
    mp = m.__dict__.setdefault('__path__',[])
if (p not in mp):
    mp.append(p)

In case a script must use another version of the module, manipulating sys.path won't help. Instead, take one of these steps: 1. Delete the .pth file beforehand. (This may give problems when normally loading the module if it is in an egg file) 2. Delete the entry in sys.modules for the given module: if 'sphinxcontrib in sys.modules: del sys.modules['sphinxcontrib'], and manipulate sys.path as usual. 3. Replace the entry in sys.modules with one pointing to another path: sys.modules['sphinxcontrib'].__dict__['__path__'] = desired_path


Further reading: