py lib
[pytest-overview]

py.test - flexible and powerful automated testing

Authors: Holger Krekel
merlinux GmbH
Date: 13th March 2006

Intro: Benefits of Automated Testing

  • prove that code changes actually fix a certain issue
  • minimizing Time to Feedback for developers
  • reducing overall Time to Market
  • document usage of plugins
  • tests as a means of communication
  • easing entry for newcomers

py.test Purposes & Goals

  • automated cross-project open source testing tool
  • flexible per-project customization
  • reusing test methods/reporting across projects
  • various iterative test collection methods
  • support for distributed testing
  • py lib is a development support library

What is Python?

  • easy-to-learn flexible OO high level language
  • glue-language for connecting C++, Java and scripting
  • used e.g. by Google for deployment/testing/implementation
  • used by BIND (mainstream DNS internet server) for testing
  • Jython provides Python for JVM
  • IronPython provides Python for .NET
  • CPython is mainstream C-based platform
  • PyPy - Python in Python implementation

Main drivers of py.test development

  • PyPy project testing needs (part of EU project)
  • needs by individual (freely contributing) projects
  • at least 20 projects using py.test and py lib
  • higher level innovation plans by merlinux & experts
  • commercial needs
  • almost three years of (non-fulltime) development

Authors & copyrights

  • initial: Holger Krekel, Armin Rigo
  • major contributors: Jan Balster, Brian Dorsey, Grig Gheorghiu
  • many others with small patches
  • MIT license

who is merlinux?

  • small company founded in 2004 by Holger Krekel and Laura Creighton
  • purpose: research and development / open source technologies
  • 7 employees (no win32 experts!), 6 freelancers
  • three larger projects:
    • PyPy - next generation Python implementation
    • mailwitness - digital invoicing/signatures
    • provider of development servers
  • technologies: virtualization, deployment and testing

Main Features of py.test

  • simplest possible assert approach
  • clean setup/teardown semantics
  • stdout/stderr capturing per test
  • per-project/directory cmdline options (many predefined)
  • test selection support
  • customizable auto-collection of tests
  • more features ...

Main User-Level entry points

  • py.test.raises(Exc, func, *args, **kwargs)
  • py.test.fail(msg) -> fail a test
  • py.test.skip(msg) -> skip a test
  • py.test.ensuretemp(prefix) -> per-test session temporary directory
  • conftest.py can modify almost arbitrary testing aspects (but it's a bit involved)

some py lib components

  • py.execnet provides ad-hoc means to distribute programs
  • py.path objects abstract local and svn files
  • py.log offers (preliminary) logging support
  • py.xml.html for programmatic html generation
  • lazy import ...``import py`` is enough

py.test Implementation

  • basic picture
  • Session objects (Terminal and Tcl-GUI)
  • reporting hooks are on session objects
  • Collector hierarchy yield iteratively tests
  • uses py lib extensively (py.path/py.execnet)
  • "conftest.py" per-directory configuration mechanism

Session objects

  • responsible for driving the testing process
  • make use of iterative Collector hierarchies
  • responsible for reporting (XXX)
  • can be split to a Frontend and BackendSession for distributed testing (GUI frontend uses it)

Collector objects

  • Collectors / Test Items form a tree
  • the tree is build iteratively (driven from Sessions)
  • collector tree can be viewed with --collectonly
  • run() returns list of (test) names or runs the test
  • join(name) returns a sub collector/item
  • various helper methods to e.g. determine file/location

Extensions: ReST documentation checking

  • py/documentation/conftest.py provides test items for checking documentation and link integrity
  • uses its own collector/testitem hierarchy
  • invokes docutils processing, reports errors

Extensions: Distributed Testing

  • using py.execnet to dispatch on different python versions
  • using py.execnet to dispatch tests on other hosts/platforms
  • currently: Popen, SSH and Socket gateways
  • missing support pushing tests to "the other side"
  • missing for deployment on multiple machines
  • but it's already possible ...

Example using pywinauto from linux

  • start socketserver.py on windows
  • connect a SocketGateway e.g. from linux
  • send tests, execute and report tracebacks through the gateway
  • remotely use pywinauto to automate testing of GUI work flow
  • interactive example ...

Status of py lib

  • mostly developed on linux/OSX
  • basically all tests pass on win32 as well
  • but missing some win32 convenience
  • some support for generation of html/ReST/PDFs reports
  • py.execnet works rather reliably (pending deeper win32 testing)
  • flexible configuration but sometimes non-obvious/documented (requires understanding of internals)

Next Steps py lib / py.test

  • refined py.execnet distribution of programs
  • more configurable and customizable reporting
  • implement support for testing distribution
  • explore refined win32 support
  • automated collection of unittest.py based tests
  • make spawning processes/gateways more robust
  • doctest support
  • unify logging approaches (py.log.*)
  • ...