[pytest-overview]
index
api
source
contact
download
py.test - flexible and powerful automated testing
Authors:
Holger Krekel
merlinux GmbH
Date:
13th March 2006
Intro: Benefits of Automated Testing
prove that code changes actually fix a certain issue
minimizing Time to Feedback for developers
reducing overall Time to Market
document usage of plugins
tests as a means of communication
easing entry for newcomers
py.test Purposes & Goals
automated cross-project open source testing tool
flexible per-project customization
reusing test methods/reporting across projects
various iterative test collection methods
support for distributed testing
py lib is a development support library
What is Python?
easy-to-learn flexible OO high level language
glue-language for connecting C++, Java and scripting
used e.g. by Google for deployment/testing/implementation
used by BIND (mainstream DNS internet server) for testing
Jython provides Python for JVM
IronPython provides Python for .NET
CPython is mainstream C-based platform
PyPy - Python in Python implementation
Main drivers of py.test development
PyPy project testing needs (part of EU project)
needs by individual (freely contributing) projects
at least 20 projects using py.test and py lib
higher level innovation plans by merlinux & experts
commercial needs
almost three years of (non-fulltime) development
Authors & copyrights
initial: Holger Krekel, Armin Rigo
major contributors: Jan Balster, Brian Dorsey, Grig Gheorghiu
many others with small patches
MIT license
who is merlinux?
small company founded in 2004 by Holger Krekel and Laura Creighton
purpose: research and development / open source technologies
7 employees (no win32 experts!), 6 freelancers
three larger projects:
PyPy - next generation Python implementation
mailwitness - digital invoicing/signatures
provider of development servers
technologies: virtualization, deployment and testing
Main Features of py.test
simplest possible
assert
approach
clean setup/teardown semantics
stdout/stderr capturing per test
per-project/directory cmdline options (many predefined)
test selection support
customizable auto-collection of tests
more features
...
Main User-Level entry points
py.test.raises(Exc,
func,
*args,
**kwargs)
py.test.fail(msg)
-> fail a test
py.test.skip(msg)
-> skip a test
py.test.ensuretemp(prefix)
-> per-test session temporary directory
conftest.py
can modify almost arbitrary testing aspects (but it's a bit involved)
some py lib components
py.execnet
provides ad-hoc means to distribute programs
py.path
objects abstract local and svn files
py.log
offers (preliminary) logging support
py.xml.html
for programmatic html generation
lazy import ...``import py`` is enough
py.test Implementation
basic picture
Session objects (Terminal and Tcl-GUI)
reporting hooks are on session objects
Collector hierarchy yield iteratively tests
uses py lib extensively (py.path/py.execnet)
"conftest.py" per-directory configuration mechanism
Session objects
responsible for driving the testing process
make use of iterative Collector hierarchies
responsible for reporting (XXX)
can be split to a Frontend and BackendSession for distributed testing (GUI frontend uses it)
Collector objects
Collectors / Test Items form a tree
the tree is build iteratively (driven from Sessions)
collector tree can be viewed with
--collectonly
run()
returns list of (test) names or runs the test
join(name)
returns a sub collector/item
various helper methods to e.g. determine file/location
Extensions: ReST documentation checking
py/documentation/conftest.py
provides test items for checking documentation and link integrity
uses its own collector/testitem hierarchy
invokes
docutils
processing, reports errors
Extensions: Distributed Testing
using py.execnet to dispatch on different python versions
using py.execnet to dispatch tests on other hosts/platforms
currently: Popen, SSH and Socket gateways
missing support pushing tests to "the other side"
missing for deployment on multiple machines
but it's already possible ...
Example using pywinauto from linux
start socketserver.py on windows
connect a SocketGateway e.g. from linux
send tests, execute and report tracebacks through the gateway
remotely use pywinauto to automate testing of GUI work flow
interactive example ...
Status of py lib
mostly developed on linux/OSX
basically all tests pass on win32 as well
but missing some win32 convenience
some support for generation of html/ReST/PDFs reports
py.execnet works rather reliably (pending deeper win32 testing)
flexible configuration but sometimes non-obvious/documented (requires understanding of internals)
Next Steps py lib / py.test
refined py.execnet distribution of programs
more configurable and customizable reporting
implement support for testing distribution
explore refined win32 support
automated collection of unittest.py based tests
make spawning processes/gateways more robust
doctest support
unify logging approaches (py.log.*)
...