Library update

README.md

@@ -38,15 +38,16 @@ poetry add --dev pytest-csv-params
 
 Simply decorate your test method with `@csv_params` and the following parameters:
 
-| Parameter    | Type                     | Description                                                                                                                            | Example                              |
-|--------------|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------|
-| `data_file`  | `str`                    | The CSV file to use, relative or absolute path                                                                                         | `"/var/testdata/test1.csv"`          |
-| `base_dir`   | `str` (optional)         | Directory to look up relative CSV files (see `data_file`); overrides the command line argument                                         | `join(dirname(__file__), "assets")`  |
-| `id_col`     | `str` (optional)         | Column name of the CSV that contains test case IDs                                                                                     | `"ID#"`                              |
-| `dialect`    | `csv.Dialect` (optional) | CSV Dialect definition (see [Python CSV Documentation](https://docs.python.org/3/library/csv.html#dialects-and-formatting-parameters)) | `csv.excel_tab`                      |
-| `data_casts` | `dict` (optional)        | Cast Methods for the CSV Data (see "Data Casting" below)                                                                               | `{ "a": int, "b": float }`           |
+| Parameter        | Type                     | Description                                                                                                                            | Example                                                                                        |
+|------------------|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------|
+| `data_file`      | `str`                    | The CSV file to use, relative or absolute path                                                                                         | `"/var/testdata/test1.csv"`                                                                    |
+| `base_dir`       | `str` (optional)         | Directory to look up relative CSV files (see `data_file`); overrides the command line argument                                         | `join(dirname(__file__), "assets")`                                                            |
+| `id_col`         | `str` (optional)         | Column name of the CSV that contains test case IDs                                                                                     | `"ID#"`                                                                                        |
+| `dialect`        | `csv.Dialect` (optional) | CSV Dialect definition (see [Python CSV Documentation](https://docs.python.org/3/library/csv.html#dialects-and-formatting-parameters)) | `csv.excel_tab`                                                                                |
+| `data_casts`     | `dict` (optional)        | Cast Methods for the CSV Data (see "Data Casting" below)                                                                               | `{ "a": int, "b": float }`                                                                     |
+| `header_renames` | `dict` (optional)        | Replace headers from the CSV file, so that they can be used as parameters for the test function (since 0.3.0)                          | `{ "Annual Amount of Bananas": "banana_count", "Cherry export price": "cherry_export_price" }` | 
 
-### CSV File Lookup Order
+## CSV File Lookup Order
 
 CSV files are looked up following this rules:
 
@@ -57,13 +58,13 @@ CSV files are looked up following this rules:
     - If the command line argument is set, the file is looked up there
     - If the command line argument is not set, the file is looked up in the current working directory
 
-### Data Casting
+## Data Casting
 
 When data is read from CSV, they are always parsed as `str`. If you need them in other formats, you can set a method that should be called with the value.
 
 These methods can also be lambdas, and are also good for further transformations.
 
-#### Data Casting Example
+### Data Casting Example
 
 ```python
 from pytest_csv_params.decorator import csv_params
@@ -83,7 +84,7 @@ def test_something(col_x, col_y):
     ...
 ```
 
-### CSV Format
+## CSV Format
 
 The default CSV format is:
 
@@ -94,7 +95,7 @@ The default CSV format is:
 
 **The first line must contain the row names. Row names must match the parameters of the test method (except for an ID column that is configured as such -- see `id_col` decorator parameter).**
 
-#### Example CSV
+### Example CSV
 
 ```text
 "ID#", "part_a", "part_b", "expected_result"
@@ -103,7 +104,21 @@ The default CSV format is:
 "third", 10, 11, 21
 ```
 
-### Usage Example
+### Headers
+
+The header line is very important, as it maps the values to parameters of the test function. The plugin supports you with that. The following rules apply:
+
+- Every character that is not valid in a variable name is replaced by an underscore (`_`)
+- If the character at the start is not a letter or an underscore, it is replaced by an underscore(`_`)
+- If the name is still invalid then, because it's a keyword or a builtin name, an exception is raised (`CsvHeaderNameInvalid`)
+
+If you don't want to change your CSV file, you can use the `header_renames` parameter to the decorator to rename headers as needed.
+
+Headers must be unique, and an Exception is raised if not (`CsvHeaderNameInvalid`).
+
+The header handling was heavily improved in Version 0.3.0.
+
+## Usage Example
 
 This example uses the CSV example from above.
 
@@ -133,8 +148,84 @@ def test_texts(text_a, text_b, text_c):
     assert f"{text_a}:{text_b}" == text_c
 ```
 
+### More complex example
+
+This example features nearly all things the plugin has to offer. You find this example also in the test cases, see `tests/test_complex_example.py`.
+
+The CSV file (`tests/assets/example.csv`):
+
+```text
+"Test ID","Bananas shipped","Single Banana Weight","Apples shipped","Single Apple Weight","Container Size"
+"Order-7","1503","0.5","2545","0.25","1500"
+"Order-15","101","0.55","1474","0.33","550"
+```
+
+The Test (`tests/test_complex_example.py`):
+
+```python
+from math import ceil
+from os.path import join, dirname
+
+from pytest_csv_params.decorator import csv_params
+
+
+@csv_params(
+    data_file="example.csv",
+    base_dir=join(dirname(__file__), "assets"),
+    id_col="Test ID",
+    header_renames={
+        "Bananas shipped": "bananas_shipped",
+        "Single Banana Weight": "banana_weight",
+        "Apples shipped": "apples_shipped",
+        "Single Apple Weight": "apple_weight",
+        "Container Size": "container_size",
+    },
+    data_casts={
+        "bananas_shipped": int,
+        "banana_weight": float,
+        "apples_shipped": int,
+        "apple_weight": float,
+        "container_size": int,
+    },
+)
+def test_container_size_is_big_enough(
+    bananas_shipped: int, banana_weight: float, apples_shipped: int, apple_weight: float, container_size: int
+) -> None:
+    """
+    This is just an example test case for the documentation.
+    """
+
+    gross_weight = (banana_weight * bananas_shipped) + (apple_weight * apples_shipped)
+    assert ceil(gross_weight) <= container_size
+```
+
+If you decide not to rename the columns, the test would look like this:
+
+```python
+@csv_params(
+    data_file="example.csv",
+    base_dir=join(dirname(__file__), "assets"),
+    id_col="Test ID",
+    data_casts={
+        "Bananas_Shipped": int,
+        "Single_Banana_Weight": float,
+        "Apples_Shipped": int,
+        "Single_Apple_Weight": float,
+        "Container_Size": int,
+    },
+)
+def test_container_size_is_big_enough(
+    Bananas_Shipped: int, Single_Banana_Weight: float, Apples_Shipped: int, Single_Apple_Weight: float, Container_Size: int
+) -> None:
+    ...
+```
+
 ## Breaking Changes
 
+### Version 0.3.0
+
+- Column header names that are reserved keywords or builtin names are no longer accepted. You should have been in trouble already if you used them, so nothing should go wrong with this change and existing tests.
+
 ### Version 0.2.0
 
 - The parameter order for `pytest_csv_params.decorator.csv_params` changed to allow the shorthand usage with only a `data_file` as positional argument. If you used keyword arguments only (like the docs recommend), you will not run into trouble.

_ptcsvp/parametrize.py

@@ -9,13 +9,15 @@ import pytest
 from _pytest.mark import MarkDecorator
 
 from _ptcsvp.plugin import BASE_DIR_KEY, Plugin
+from _ptcsvp.varname import make_name_valid
 from pytest_csv_params.dialect import CsvParamsDefaultDialect
 from pytest_csv_params.exception import (
+    CsvHeaderNameInvalid,
     CsvParamsDataFileInaccessible,
     CsvParamsDataFileInvalid,
     CsvParamsDataFileNotFound,
 )
-from pytest_csv_params.types import BaseDir, CsvDialect, DataCasts, DataFile, IdColName
+from pytest_csv_params.types import BaseDir, CsvDialect, DataCasts, DataFile, HeaderRenames, IdColName
 
 
 class TestCaseParameters(TypedDict):
@@ -53,12 +55,28 @@ def read_csv(base_dir: BaseDir, data_file: DataFile, dialect: CsvDialect) -> Lis
     return csv_lines
 
 
-def add_parametrization(
+def clean_headers(current_headers: List[str], replacing: HeaderRenames) -> List[str]:
+    """
+    Clean the CSV file headers
+    """
+    if replacing is not None:
+        for index, header in enumerate(current_headers):
+            replacement = replacing.get(header, None)
+            if replacement is not None:
+                current_headers[index] = replacement
+    current_headers = list(map(make_name_valid, current_headers))
+    if len(current_headers) != len(set(current_headers)):
+        raise CsvHeaderNameInvalid("Header names are not unique")
+    return current_headers
+
+
+def add_parametrization(  # pylint: disable=too-many-arguments
     data_file: DataFile,
     base_dir: BaseDir = None,
     id_col: IdColName = None,
     data_casts: DataCasts = None,
     dialect: CsvDialect = CsvParamsDefaultDialect,
+    header_renames: HeaderRenames = None,
 ) -> MarkDecorator:
     """
     Get data from the files and add things to the tests
@@ -78,6 +96,7 @@ def add_parametrization(
             raise CsvParamsDataFileInvalid(f"Cannot find ID column '{id_col}'") from err
     if len(headers) == 0:
         raise CsvParamsDataFileInvalid("File seems only to have IDs") from None
+    headers = clean_headers(headers, header_renames)
     data: List[TestCaseParameters] = []
     for data_line in csv_lines:
         line = list(map(str, data_line))

_ptcsvp/varname.py

@@ -0,0 +1,44 @@
+"""
+Test if a variable name is valid
+"""
+
+import builtins
+import keyword
+import re
+from string import ascii_letters, digits
+
+from pytest_csv_params.exception import CsvHeaderNameInvalid
+
+VALID_CHARS = ascii_letters + digits
+VARIABLE_NAME = re.compile(r"^[a-zA-Z_][A-Za-z0-9_]{0,1023}$")
+
+
+def is_valid_name(name: str) -> bool:
+    """
+    Checks if the variable name is valid
+    """
+    if (
+        keyword.iskeyword(name)
+        or (hasattr(keyword, "issoftkeyword") and getattr(keyword, "issoftkeyword")(name))
+        or getattr(builtins, name, None) is not None
+    ):
+        return False
+    return VARIABLE_NAME.match(name) is not None
+
+
+def make_name_valid(name: str, replacement_char: str = "_") -> str:
+    """
+    Make a name valid
+    """
+
+    fixed_name = name
+
+    for index, character in enumerate(name):
+        if character in VALID_CHARS:
+            continue
+        fixed_name = f"{fixed_name[:index]}{replacement_char}{fixed_name[index+1:]}"
+    if fixed_name[0] not in ascii_letters:
+        fixed_name = f"{replacement_char}{fixed_name[1:]}"
+    if not is_valid_name(fixed_name):
+        raise CsvHeaderNameInvalid(f"'{fixed_name}' is not a valid variable name")
+    return fixed_name

poetry.lock

@@ -223,9 +223,9 @@ tomli = {version = ">=1.1.0", markers = "python_version < \"3.11\""}
 typing-extensions = ">=3.10"
 
 [package.extras]
-dmypy = ["psutil (>=4.0)"]
-python2 = ["typed-ast (>=1.4.0,<2)"]
 reports = ["lxml"]
+python2 = ["typed-ast (>=1.4.0,<2)"]
+dmypy = ["psutil (>=4.0)"]
 
 [[package]]
 name = "mypy-extensions"
@@ -445,7 +445,7 @@ python-versions = ">=3.7"
 pytest = ">=5.0"
 
 [package.extras]
-dev = ["pre-commit", "tox", "pytest-asyncio"]
+dev = ["pytest-asyncio", "tox", "pre-commit"]
 
 [[package]]
 name = "pytest-mypy"
@@ -673,10 +673,7 @@ iniconfig = [
     {file = "iniconfig-1.1.1-py2.py3-none-any.whl", hash = "sha256:011e24c64b7f47f6ebd835bb12a743f2fbe9a26d4cecaa7f53bc4f35ee9da8b3"},
     {file = "iniconfig-1.1.1.tar.gz", hash = "sha256:bc3af051d7d14b2ee5ef9969666def0cd1a000e121eaea580d4a313df4b37f32"},
 ]
-isort = [
-    {file = "isort-5.10.1-py3-none-any.whl", hash = "sha256:6f62d78e2f89b4500b080fe3a81690850cd254227f27f75c3a0c491a1f351ba7"},
-    {file = "isort-5.10.1.tar.gz", hash = "sha256:e8443a5e7a020e9d7f97f1d7d9cd17c88bcb3bc7e218bf9cf5095fe550be2951"},
-]
+isort = []
 lazy-object-proxy = []
 mccabe = []
 mypy = []
@@ -688,10 +685,7 @@ packaging = [
     {file = "packaging-21.3-py3-none-any.whl", hash = "sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522"},
     {file = "packaging-21.3.tar.gz", hash = "sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb"},
 ]
-pathspec = [
-    {file = "pathspec-0.9.0-py2.py3-none-any.whl", hash = "sha256:7d15c4ddb0b5c802d161efc417ec1a2558ea2653c2e8ad9c19098201dc1c993a"},
-    {file = "pathspec-0.9.0.tar.gz", hash = "sha256:e564499435a2673d586f6b2130bb5b95f04a3ba06f81b8f895b651a3c76aabb1"},
-]
+pathspec = []
 pbr = []
 platformdirs = []
 pluggy = [
@@ -717,9 +711,7 @@ pytest = [
     {file = "pytest-7.1.2.tar.gz", hash = "sha256:a06a0425453864a270bc45e71f783330a7428defb4230fb5e6a731fde06ecd45"},
 ]
 pytest-bandit = []
-pytest-black = [
-    {file = "pytest-black-0.3.12.tar.gz", hash = "sha256:1d339b004f764d6cd0f06e690f6dd748df3d62e6fe1a692d6a5500ac2c5b75a5"},
-]
+pytest-black = []
 pytest-clarity = [
     {file = "pytest-clarity-1.0.1.tar.gz", hash = "sha256:505fe345fad4fe11c6a4187fe683f2c7c52c077caa1e135f3e483fe112db7772"},
 ]
@@ -731,10 +723,7 @@ pytest-isort = []
 pytest-mock = []
 pytest-mypy = []
 pytest-order = []
-pytest-pylint = [
-    {file = "pytest-pylint-0.18.0.tar.gz", hash = "sha256:790c7a8019fab08e59bd3812db1657a01995a975af8b1c6ce95b9aa39d61da27"},
-    {file = "pytest_pylint-0.18.0-py3-none-any.whl", hash = "sha256:b63aaf8b80ff33c8ceaa7f68323ed04102c7790093ccf6bdb261a4c2dc6fd564"},
-]
+pytest-pylint = []
 pyyaml = []
 rich = []
 six = [
@@ -743,10 +732,7 @@ six = [
 ]
 smmap = []
 stevedore = []
-toml = [
-    {file = "toml-0.10.2-py2.py3-none-any.whl", hash = "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b"},
-    {file = "toml-0.10.2.tar.gz", hash = "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f"},
-]
+toml = []
 tomli = [
     {file = "tomli-2.0.1-py3-none-any.whl", hash = "sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc"},
     {file = "tomli-2.0.1.tar.gz", hash = "sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f"},

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "pytest-csv-params"
-version = "0.2.2"
+version = "0.3.0"
 description = "Pytest plugin for Test Case Parametrization with CSV files"
 authors = ["Juergen Edelbluth <csv_params@jued.de>"]
 license = "MIT"

pytest_csv_params/exception.py

@@ -19,3 +19,9 @@ class CsvParamsDataFileInvalid(ValueError):
     """
     CSV Data is somehow invalid
     """
+
+
+class CsvHeaderNameInvalid(ValueError):
+    """
+    Invalid Header Name
+    """

pytest_csv_params/types.py

@@ -14,3 +14,6 @@ IdColName = Optional[str]
 DataFile = str
 
 CsvDialect = Type[csv.Dialect]
+
+HeaderRenamesDict = Dict[str, str]
+HeaderRenames = Optional[HeaderRenamesDict]

tests/assets/example.csv

@@ -0,0 +1,3 @@
+"Test ID","Bananas shipped","Single Banana Weight","Apples shipped","Single Apple Weight","Container Size"
+"Order-7","1503","0.5","2545","0.25","1500"
+"Order-15","101","0.55","1474","0.33","550"

tests/test_clean_headers.py

@@ -0,0 +1,62 @@
+"""
+Test cleaning the headers
+"""
+from typing import List, Optional, Type
+
+import pytest
+
+from _ptcsvp.parametrize import clean_headers
+from pytest_csv_params.exception import CsvHeaderNameInvalid
+from pytest_csv_params.types import HeaderRenames
+
+
+@pytest.mark.parametrize(
+    ["current_headers", "replacing", "expect_exception", "expect_message", "expect_result"],
+    [
+        (
+            ["alpha 3", "_beta_5", "Ga Mm A"],
+            None,
+            None,
+            None,
+            ["alpha_3", "_beta_5", "Ga_Mm_A"],
+        ),
+        (
+            ["abcd" * 300, "_be:ta_ :23", "Ra -/2"],
+            None,
+            CsvHeaderNameInvalid,
+            f"'{'abcd' * 300}' is not a valid variable name",
+            None,
+        ),
+        (
+            ["next_var", "_be:ta_ :23", "Ra -/2"],
+            None,
+            None,
+            None,
+            ["next_var", "_be_ta___23", "Ra___2"],
+        ),
+        (
+            ["next_var", "_be:ta_ :23", "Ra -/2", "Ra___2"],
+            None,
+            CsvHeaderNameInvalid,
+            "Header names are not unique",
+            None,
+        ),
+    ],
+)
+def test_header_cleaning(
+    current_headers: List[str],
+    replacing: HeaderRenames,
+    expect_exception: Optional[Type[ValueError]],
+    expect_message: Optional[str],
+    expect_result: List[str],
+) -> None:
+    """
+    Test Header Cleaning
+    """
+    if expect_exception is not None:
+        with pytest.raises(expect_exception) as raised_error:
+            clean_headers(current_headers, replacing)
+        assert raised_error.value.args[0] == expect_message
+    else:
+        result = clean_headers(current_headers, replacing)
+        assert result == expect_result

tests/test_complex_example.py

@@ -0,0 +1,37 @@
+"""
+Example Test Case from the documentation
+"""
+from math import ceil
+from os.path import dirname, join
+
+from pytest_csv_params.decorator import csv_params
+
+
+@csv_params(
+    data_file="example.csv",
+    base_dir=join(dirname(__file__), "assets"),
+    id_col="Test ID",
+    header_renames={
+        "Bananas shipped": "bananas_shipped",
+        "Single Banana Weight": "banana_weight",
+        "Apples shipped": "apples_shipped",
+        "Single Apple Weight": "apple_weight",
+        "Container Size": "container_size",
+    },
+    data_casts={
+        "bananas_shipped": int,
+        "banana_weight": float,
+        "apples_shipped": int,
+        "apple_weight": float,
+        "container_size": int,
+    },
+)
+def test_container_size_is_big_enough(
+    bananas_shipped: int, banana_weight: float, apples_shipped: int, apple_weight: float, container_size: int
+) -> None:
+    """
+    This is just an example test case for the documentation.
+    """
+
+    gross_weight = (banana_weight * bananas_shipped) + (apple_weight * apples_shipped)
+    assert ceil(gross_weight) <= container_size

tests/test_varname.py

@@ -0,0 +1,97 @@
+"""
+Test the varname handling
+"""
+import sys
+from typing import Optional
+
+import pytest
+
+from _ptcsvp.varname import is_valid_name, make_name_valid
+from pytest_csv_params.exception import CsvHeaderNameInvalid
+
+
+@pytest.mark.parametrize(
+    ["var_name", "is_valid"],
+    [
+        ("var_name", True),
+        ("varname", True),
+        ("_varname", True),
+        ("_varName", True),
+        ("VarName", True),
+        ("VarName_", True),
+        ("Var1Name_", True),
+        ("Var1Name_0", True),
+        ("0_var_name", False),
+        ("var name", False),
+        ("Var Name", False),
+        ("Var-Name", False),
+        ("Var.Name", False),
+        ("Var:Name", False),
+        (":VarName", False),
+        (".VarName", False),
+        (";VarName", False),
+        ("VarName+", False),
+        ("VarName#", False),
+        ("VarNäme", False),
+        ("VarNÖme", False),
+        ("Varßn_ame", False),
+        ("def", False),
+        ("next", False),
+        ("if", False),
+    ],
+)
+def test_is_valid_name(var_name: str, is_valid: bool) -> None:
+    """
+    Test if the varname is considered valid or not
+    """
+
+    assert is_valid_name(var_name) == is_valid
+
+
+@pytest.mark.parametrize(
+    ["var_name", "valid_var_name", "raises_error"],
+    [
+        ("var_name", "var_name", False),
+        ("4var_name", "_var_name", False),
+        ("44_var_name", "_4_var_name", False),
+        ("varName", "varName", False),
+        ("var Name", "var_Name", False),
+        (" varName", "_varName", False),
+        (":varName", "_varName", False),
+        ("var-name", "var_name", False),
+        ("a" * 1025, None, True),
+        ("abcd" * 300, None, True),
+        ("continue", None, True),
+        ("float", None, True),
+        ("str", None, True),
+    ],
+)
+def test_make_name_valid(var_name: str, valid_var_name: Optional[str], raises_error: bool) -> None:
+    """
+    Check if an invalid name goes valid
+    """
+
+    if raises_error:
+        with pytest.raises(CsvHeaderNameInvalid) as raised_error:
+            make_name_valid(var_name)
+        assert raised_error.value.args[0] == f"'{var_name}' is not a valid variable name"
+    else:
+        should_be_valid = make_name_valid(var_name)
+        assert should_be_valid == valid_var_name
+        assert is_valid_name(should_be_valid)
+
+
+@pytest.mark.skipif(sys.version_info < (3, 10), reason="Only relevant for python 3.10 and above")
+@pytest.mark.parametrize(
+    ["name"],
+    [
+        ("match",),
+        ("_",),
+    ],
+)
+def test_310_names(name: str) -> None:
+    """
+    Check if special 3.10 names are considered invalid, when on 3.10
+    """
+
+    assert not is_valid_name(name)