Shell environment variables

On all *nix (unix, linux, xenix etc) operating systems, each process has its own set of variables, which, by default, are set by the parent shell environment from which the process is run. A shell is a program used to interface with the operating system of a computer, mostly through a Terminal such as a CLI (Command Line Interface). On machines which run under a such as Macintosh OSX), several shells are available: sh, csh, tsh and bash for example. All *nix operating systems as well as DOS and Microsoft Windows have environment variables; however, they do not all use the same variable names.

So, when a program (such as python) is instantiated ("run"), it can access the values of environment variables of of its parent process, its shell. This is similar to the .profile feature of the Unix shells. MacOSX users, see

http://developer.apple.com/documentation/MacOSX/Conceptual/BPRuntimeConfig/Articles/EnvironmentVars.html

Python is designed to look at a couple of different shell variables to help it find the things it needs. Two of interest here are:

• PYTHONSTARTUP: an environment variable used to set the name of a file which contains the filename of a file containing commands for python to execute immediately on starting up. Example:
% echo "import time" > /Users/drw/myPythonStartup
% echo "print ; print time.asctime(time.localtime()),'...HAVING SOME SPAM TODAY??' " >> /Users/drw/myPythonStartup
% set PYTHONSTARTUP="/Users/drw/myPythonStartup"
% export PYTHONSTARTUP="/Users/drw/myPythonStartup"
% echo $PYTHONSTARTUP
/Users/drw/myPythonStartup
% python
Python 2.4.3 (#1, Mar 30 2006, 11:02:16)
[GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

Tue May 12 19:23:59 2007 ...HAVING SOME SPAM TODAY??
>>>
  


• PYTHONPATH: When a module named spam is imported, the interpreter searches for a file named sspam.py in the current directory, and then in the list of directories specified by the environment variable PYTHONPATH. This has the same syntax as the shell variable PATH, that is, a list of directory names.

  When PYTHONPATH is not set, or when the file is not found there, the search continues in an installation-dependent default path; on Unix, this is usually .:/usr/local/lib/python. Note that because the directory containing the script being run (the "." in .:/usr/local/lib/python) is on the search path, it is important that the script not have the same name as a standard module, or Python will attempt to load the script as a standard module when that module is imported. This will generally be an error.

MacOSX: Where 3rd party modules are located

You may find it helpful to read the Macintosh OSX Installation section of the INSTALL page before reading this section. We observed that when a new version of Python is downloaded, it is installed as a Framework in the /Library/Frameworks/Python.framework/Versions/ directory:

The contents of the Python.framework/Versions directory (on my Intel OSX machine) looks like this:

The 2.4 directory contains the complete Python 2.4 Framework:

A full exploration of these directories is left the to reader; Only one one concerns us here, where third-party modules are stored: /Library/Frameworks/Python.framework/Versions/Current/lib/python2.4/site-packages/

For example, here's the NumPy package Framework:

% ls /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy

If the file numpy.py does not exist but there is a numpy directory, (which is the case here), it looks in that directory for a __init__.py file. If found, it imports and executes the contents. This may include importing other .py files in that directory. This process recurses, so if it finds that that file is a directory, it looks in that directory for a __init__.py file, and so on.

Recursive importing by using __init__.py files is the standard way of managing multi-module projects.

UPDATES TO THIS PAGE

update: 2009110
When the going gets tough - some extra techniques for installing 3rd party python packages in the  right place

drwIntel-3:~ drw$ sudo find /. -name "site-packages" -print

/./Applications/Darwine/Wine.bundle/Contents/lib/python2.3/site-packages
/./Applications/Komodo Edit.app/Contents/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages
/./Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages
/./Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages
/./Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages
/./Library/Python/2.3/site-packages
/./Library/Python/2.5/site-packages
/./System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages
/./System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python2.5/site-packages
/./usr/local/aquarium-2.2.1/demo/seamstress_exchange/site-packages

drwIntel-3:~ drw$

2. Change the symbolic links (symlinks) to the files pointed to by /usr/local/bin/python and /usr/bin/python to the python you want to use. For example, if you want to just use the python at
/Library/Frameworks/Python.framework/Versions/Current/bin/python

% cd /usr/local/bin
% sudo mv python python_old
% sudo ln -s /Library/Frameworks/Python.framework/Versions/Current/bin/python python

3.  If space is tight, you may want to Backup (!!!) and remove any non-apple pythons you don't want to use.

Then install a package that you couldn't previously find and see if it works!

Update 20090120
If you are daunted by the package-building and installing to build SoniPy tools, there is a single build of many of the packages at Chris Fonnesbeck's at Macscience site. Chris uses Enthought's Python Distribution (free for academic and non-profit work). The package is called Scipy superpack available here. There is a series of blog posts there that give hints to what's going on when you download and install a 3rd party python module and it is "not found" when you try to "import xxx".  Please note I have not tested this package. At last look, the download was over 160Mb. If you do, please take a few moments to send an email to report on the experience.