Directory tree

• able-workflow-copier

The able-workflow-copier project tree

├── .github/
│   ├── workflows/
│   │   ├── github-labeler.yml  # (5)
│   │   └── ci.yml  # (1)
│   └── labels.yml  # (6)
├── .vscode  # (2)
├── able-workflow/  # (7)
├── docs/  # (3)
│   └── docs/  #(4)
├── extensions/  # (8)
├── hooks/  # (9)
├── includes/  # (15)
├── schemas/  # (10)
├── scripts/  # (11)
├── tasks/  # (14)
├── template/  # (12)
├── tests/  # (13)

1. GitHub action for pull requests
2. VSCode configuration files
3. mkdocs documentation directory
4. Documentation markdown and assets
5. GitHub action to maintain consistient GitHub issue labels and colors across projects
6. Configuration file for github-labeler GH action
7. Submodules to other able-workflow-*-copier projects
8. jinja2 extensions to be used by copier
9. pre-commit hook scripts
10. Custom json/yaml schemas for validating Copier answers
11. Helper scripts for development
12. The Copier template to be generated
13. pytest tests to be run. SEE: tox.ini
14. Task scripts run by Copier after template generation.
15. Copier yaml files that are included in copier.yml

The able-workflow project template tree

Project directory tree

Copier templates variables

Copier uses a jinja2 syntax for rendering templates with the answers to the templates questions.

• able-workflow-copier
• able-workflow-copier template
  • {{ project_name_slug }}: The name of the project being created.
  • {{ package_name }}: The name of the Python package being created.
• able-workflow-module-copier template
  • {{ module_type }}: The type of module being created (e.g., datasets, features, or models).
  • {{ module_name }}: The name of the module being created.
• able-workflow-etl-copier template
  • {{ etl_name }}: The name of the ETL process being created.
  • {{ conda_env_key }}: The key for the Conda environment from the workflow config (e.g., config["CONDA"]["ENVS"]["{{ conda_env_key }}"]).
• able-workflow-rule-copier template
  • {{ is_package_rule }}: The rule uses the package.
  • {{ rule_name }}: The name of the Snakemake rule being created

Directory tree

Files that are created by a template are marked with a + sign, and files that are created by a Copier template are marked with a * sign. The tree structure is as follows:

┌───── [1] able-workflow-copier
│┌──── [2] able-workflow-module-copier
││┌─── [3] able-workflow-etl-copier
│││┌── [4] able-workflow-rule-copier
││││
Legend ── “+” = created | “*” = modified | blank = untouched
    {{ project_name_slug }}
    │
 +  ├── .copier-answers/ # (3)
 +  │   ├── module-{{ module_type }}-{{ module_name }}.yml
  + │   ├── etl-{{ module_type }}-{{ module_name }}-{{ etl_name }}.yml
   +│   └── rule-{{ module_type }}-{{ module_name }}-{{ rule_name }}.yml
    │
+   ├── .github/
+   │   ├── ISSUE_TEMPLATE/
  + │   │   ├── post-copier-etl.md
 +  │   │   ├── post-copier-module.md
+   │   │   ├── post-copier-package.md
   +│   │   └── post-copier-rule.md
+   │   └── workflows/
+   │       ├── build.yml
+   │       ├── ci.yml
+   │       └── release.yml
    │
+   ├── .vscode/
+   │   ├── extensions.json
+ * │   ├── launch.json
+  *│   └── settings.json
    │
+   ├── config/
+   │   ├── config.yaml
+   │   ├── config.local.example.yaml
+   │   ├── README.md
+   │   ├── datasets/
 +  │   │   └── {{ module_name }}/                # if {{ module_type }} == dataset
 +* │   │       └── config.yaml                   # +: module, *: etl tweaks
+   │   ├── features/
 +  │   │   └── {{ module_name }}/                # if {{ module_type }} == feature
 +* │   │       └── config.yaml
+   │   └── models/
 +  │       └── {{ module_name }}/                # if {{ module_type }} == model
 +* │           └── config.yaml
    │
+   ├── data/
+   │   ├── README.md
 +* │   └── {{ module_name }}/                    # dataset only
 +* │       ├── external/  # (12)
 +* │       ├── interim/  # (13)
 +* │       ├── processed/  # (14)
 +* │       ├── raw/  # (15)
 +* │       └── README.md
    │
+   ├── docs/
+   │   ├── docs/
+   │   │   ├── _css/
+   │   │   ├── _images/
+   │   │   ├── _js/
    │   │   │
    │   │   ├── datasets/
 +  │   │   │   └── {{ module_type }}/
 +  │   │   │       ├── index.md
 +  │   │   │       └── SUMMARY.md
    │   │   │
    │   │   ├── features/
 +  │   │   │   └── {{ module_type }}/
 +  │   │   │       ├── index.md
 +  │   │   │       └── SUMMARY.md
    │   │   │
    │   │   ├── models/
 +  │   │   │   └── {{ module_type }}/
 +  │   │   │       ├── index.md
 +  │   │   │       └── SUMMARY.md
    │   │   │
    │   │   ├── index.md
    │   │   ├── getting-started.md
    │   │   └── SUMMARY.md
    │   │
+   │   ├── gen_ref_pages.py
+   │   ├── mkdocs.yml
+   │   └── README.md
    │
+   ├── hooks/
+   │   └── snakemake_pyproject2conda.py
    │
+   ├── features/
+   │   └── README.md
 +  │   └── {{ module_name }}/README.md           # if feature module
    │
+   ├── logs/rules/
+   │   └── README.md
    │
+   ├── models/
+   │   └── README.md
 +  │   └── {{ module_name }}/README.md           # if model module
+   ├── notebooks/
+   │   └── README.md
+   ├── references/
+   │   └── README.md
+   ├── reports/
+   │   ├── datasets/
+   │   │   └── .gitkeep
 +  │   │   └── {{ module_name }}/README.md       # dataset report stub
+   │   ├── features/
+   │   │   └── .gitkeep
 +  │   │   └── {{ module_name }}/README.md       # feature report stub
+   │   ├── models/
+   │   │   └── .gitkeep
 +  │   │   └── {{ module_name }}/README.md       # model report stub
+   │   └── notebook_templates/
+   │       ├── datasets/
 +  │       │   └── {{ module_name }}/README.md
+   │       ├── features/
 +  │       │   └── {{ module_name }}/README.md
+   │       └── models/
 +  │           └── {{ module_name }}/README.md
    │
+   ├── {{ package_name }}/
    │   │
+   │   ├── datasets/
 +  │   │   ├── {{ module_name }}/
    │   │   │   ├── {{ etl_name }}.py
    │   │   │   │   ├── runner/
    │   │   │   │   │   ├── __init__.py
    │   │   │   │   │   ├── extract_external.py
    │   │   │   │   │   ├── load.py
    │   │   │   │   │   ├── main.py
    │   │   │   │   │   ├── transform.py
    │   │   │   │   │   └── utils.py
    │   │   │   │   ├── __init__.py
    │   │   │   │   ├── extract.py
    │   │   │   │   └── schema.py
    │   │   │   └── __init__.py
+   │   │   └── __init__.py
    │   │
+   │   ├── features/
 +  │   │   ├── {{ module_name }}/
    │   │   │   ├── {{ etl_name }}.py
    │   │   │   │   ├── runner/
    │   │   │   │   │   ├── __init__.py
    │   │   │   │   │   ├── extract_external.py
    │   │   │   │   │   ├── load.py
    │   │   │   │   │   ├── main.py
    │   │   │   │   │   ├── transform.py
    │   │   │   │   │   └── utils.py
    │   │   │   │   ├── __init__.py
    │   │   │   │   ├── extract.py
    │   │   │   │   └── schema.py
    │   │   │   └── __init__.py
+   │   │   └── __init__.py
    │   │
+   │   ├── models/
 +  │   │   ├── {{ module_name }}/
    │   │   │   ├── {{ etl_name }}.py
    │   │   │   │   ├── runner/
    │   │   │   │   │   ├── __init__.py
    │   │   │   │   │   ├── extract_external.py
    │   │   │   │   │   ├── load.py
    │   │   │   │   │   ├── main.py
    │   │   │   │   │   ├── transform.py
    │   │   │   │   │   └── utils.py
    │   │   │   │   ├── __init__.py
    │   │   │   │   ├── extract.py
    │   │   │   │   └── schema.py
    │   │   │   └── __init__.py
+   │   │   └── __init__.py
    │   │
+   │   ├── utils/
+   │   │   ├── __init__.py
+   │   │   └── logging.py
    │   │
+   │   ├── __init__.py
+   │   └── config.py
    │
+   ├── tests/
+   │   ├── docs/
+   │   │   └── test_dag.py
+   │   ├── {{ package_name }}/
+   │   │   ├── datasets/.gitkeep
+   │   │   ├── features/.gitkeep
+   │   │   ├── models/.gitkeep
+   │   │   ├── utils/.gitkeep
+   │   │   └── README.md
+   │   └── worflow/
+   │       ├── rules/
+   │       │   ├── conftext.py
+   │       │   └── README.md
+   │       └── scripts/
+  +│           ├── rules_conda_{{ conda_env_key }}/ # (6)
+   │           ├── rules_conda_CORE/ # (10)
+   │           ├── rules_conda_DOCS/ # (5)
+   │           ├── rules_conda_RUNNER/ # (19)
+   │           ├── rules_global/ # (4)
+   │           │   ├── test_conda_localize_file.py
+   │           │   └── test_pyproject2conda.py
+   │           ├── utils/
+   │           └── README.md
+   ├── workflow/
+   │   ├── envs/
+   │   │   ├── localized/
+   │   │   ├── pyproject2conda/
+   │   │   ├── py312-tox.yaml
+   │   │   └── py312-workflow.yaml
+   │   ├── profiles/
+   │   │   ├── default/config.yaml
+   │   │   └── slurm/
+   │   ├── rules/
+   │   │   ├── datasets/
 +  │   │   │   └── {{ module_name }}/README.md   # dataset rules stub
+   │   │   ├── features/
 +  │   │   │   └── {{ module_name }}/README.md
+   │   │   ├── models/
 +  │   │   │   └── {{ module_name }}/README.md
+   │   │   ├── build.smk
+   │   │   ├── dev.smk
+   │   │   ├── docs.smk
+   │   │   ├── includes.smk  # (18)
+   │   │   ├── reports.smk
+   │   │   └── utils.smk
+   │   ├── schemas/
+   │   │   ├── datasets/
 +  │   │   │   └── {{ module_name }}/README.md
+   │   │   ├── features/
 +  │   │   │   └── {{ module_name }}/README.md
+   │   │   ├── models/
 +  │   │   │   └── {{ module_name }}/README.md
+   │   │   ├── config.schema.json
+   │   │   └── config.local.schema.json
+   │   ├── scripts/
+  +│   │   ├── rules_conda_RUNNER/ # (7)
+   │   │   │   └── weh_interviews_rules.py
+   │   │   ├── rules_conda_CORE/ # (11)
+   │   │   ├── rules_conda_DOCS/ # (8)
+   │   │   │   ├── dag_svg.py
+   │   │   ├── rules_conda_RUNNER/ # (20)
+   │   │   ├── rules_global/ # (9)
+   │   │   │   ├── conda_localize_file.py
+   │   │   │   ├── pyproject2conda.py
+   │   │   └── utils/
+   │   │       ├── __init__.py
+   │   │       └── config_loader.py
+   │   └── Snakefile
+   ├── .env.example
+   ├── .copier-answers.yml
+   ├── .gitattributes
+   ├── .gitignore
+   ├── .pre-commit-config.yaml
+   ├── AGENTS.md
+   ├── CHANGELOG.md
+   ├── codecov.yml
+   ├── LICENSE
+   ├── pyproject.toml
+   ├── README.md
+   ├── snakefmt.toml # (16)
+   └── tox.ini # (17)

1. _
2. _
3. The copier answers files get stored here from each template applied to the project.
4. Tests for rule scripts that can run in the Snakemake global conda environment.
5. Tests for rule scripts that require the config["CONDA"]["ENVS"]["DOCS"] conda environment.
6. Tests for rule scripts that require the config["CONDA"]["ENVS"]["{{ conda_env_key }}"] conda environment.
7. Rule scripts that require the config["CONDA"]["ENVS"]["{{ conda_env_key }}"] conda environment.
8. Rule scripts that require the config["CONDA"]["ENVS"]["DOCS"] conda environment.
9. Rule scripts that can run in the Snakemake global conda environment.
10. Tests for rule scripts that require the {{ package_name }} core conda environment, config["CONDA"]["ENVS"]["CORE"].
11. Rule scripts that require the {{ package_name }} core conda environment, config["CONDA"]["ENVS"]["CORE"].
12. Data copied from an external source in a non-standard format. This folder may be a symlink to another location on disk.
13. Temporary data storage for intermediate workflow steps.
14. Processed data ready for reporting, feature extraction, or modeling.
15. Raw data in a standardized format.
16. Configuration settings for snakemake formatting
17. Main tox settings for resting template creation and code within generated example templates.
18. A singe file to aggregate all the includes.
19. Tests for rule scripts that require the conda environment, config["CONDA"]["ENVS"]["RUNNER"], that includes the {{ package_name }} package and the runner extra dependencies.
20. Rule scripts that require the conda environment, config["CONDA"]["ENVS"]["RUNNER"], that includes the {{ package_name }} package and the runner extra dependencies.