Test Runner Python
The unittest module has both a built-in testing framework and a test runner. A testing framework is a set of rules which must be followed while writing test cases, while a test runner is a tool which executes these tests with a bunch of settings, and collects the results. Keep these heavier tests in a separate test suite that is run by some scheduled task, and run all other tests as often as needed. Learn your tools and learn how to run a single test or a test case. Then, when developing a function inside a module, run this function’s tests frequently, ideally automatically when you save the code. This executes the test runner by discovering all classes in this file that inherit from unittest.TestCase. This is one of many ways to execute the unittest test runner. When you have a single test file named test.py, calling python test.py is a great way to get started. Another way is using the unittest command line. Write and run Python code using our online compiler (interpreter). You can use Python Shell like IDLE, and take inputs from the user in our Python compiler.
View project source on Github - autograder.zip - sample solution
Project Description
In this assignment, students will build an infix calculator REPL. Thegoal of this project is to teach the basics of parsing and evaluatinga simple language.
Code Runner Python
Requirements
- Build an infix calculator read-eval-print loop
- The calculator should handle the 4 basic operations, +, -, *, /, with operator precedence
- In addition, it should handle parentheses and negative numbers
- If the user types 'quit', exit the program
- If there are syntax errors in the user input, raise CalculatorException
Dependencies (for tests)
- Python 3.6+
- gradescope-utils provides decorators for setting point values for tests, and running tests with a JSON output. See the Github repository for more on what you can do with it, or you can look at the example tests in this project for some usage examples.
Python 3
Make sure to use pip3
and python3
when writing your code, because our autograder base image does include Python 2 as well, which are currently the defaults for pip
and python
. When installing Python 3, use the apt packages python3
and python3-pip
. If you need a more recent version than what is packaged by Ubuntu 18.04, you can try using a PPA or installing from source.
Example Test
The title of the test case is taken from the first line of thedocstring. This is a unittest
convention. The weight for each test isgiven by the @weight
decorator.
See all testshere
Running Tests
This will load tests from the tests/
directory in your autograder source code,loading only files starting with test
by default.JSONTestRunner
is included in gradescope-utils
, as described below.
setup.sh
This script installs Python and the pip package manager. Then it usespip to install our two external dependencies.
run_autograder
This script copies the student's submission to the target directory,and then executes the test runner Python script.
run_tests.py
This python script loads and runs the tests using the JSONTestRunnerclass from gradescope-utils. This produces the JSON formatted outputto stdout, which is then captured and uploaded by the autograderharness.
framework.py
This is a blank template file for the students to fill in. Note thattheir solution must be called calculator.py for the autograder to workcorrectly.
PyCharm enables usage of the following testing frameworks:
Framework | Code completion | Run/debug configuration | Ability to create a test | Navigation between tests and tests subjects | Ability to run tests | Code inspections |
---|---|---|---|---|---|---|
Python unittests | Yes | Yes | Yes | Yes | Yes | Partially |
Pytest | Yes | Yes | Yes | Yes | Yes | Yes |
Python nosetests | Partially | Yes | Yes | Yes | Yes | Partially |
tox | No | Yes | No | No | Yes | Partially |
TwistedTrial | Yes | Yes | Yes | Yes | Yes | N/A |
Python doctests | N/A | Yes | No | N/A | Yes | Yes |
Available only in PyCharm Professional | ||||||
BDD Testing Framework | Yes | Yes | Yes (for step definitions) | Yes (between steps and features) | Yes | Partially |
Before you start working with the testing framework of your choice, make sure that the desired framework is installed on your machine. Refer to the framework documentation for the installation details.
PyCharm auto-detects a test runner that is installed on your Python interpreter and uses it to run tests. Still, you always have an option to explicitly specify the required test runner in the project settings.
Set a testing framework
To set a test runner, press Ctrl+Alt+S to open the IDE settings and select Tools | Python Integrated Tools, and then select the target test runner from the Default test runner list.
Choose the desired test runner:
If the selected test runner is missing in the specified interpreter, the appropriate notification appears.
Click the Fix button to download and install the corresponding framework.
By default, the suggested default test runner is unittest. However, you can redefine the default framework and change it to nosetest, pytest or TwistedTrial.
Change the default testing framework
From the main menu, select File | New Project Setup | Settings for New Projects for Windows and Linux, or File | New Project Setup | Preferences for New Projects for macOS.
Select Tools | Python Integrated Tools.
In the Testing area, select the test runner that will be default for all newly created projects.
Python Code Runner Download
With the test runner selected, PyCharm suggests the appropriate default run/debug configuration:
Python Test Runner Function
If a user already has the testing run/debug configuration for a specific file and for a specific testing framework, then PyCharm will launch this run/debug configuration, regardless of your chosen default test runner. For more details in how to change or delete such a configuration, see Run/debug configurations.