Developing PUDL as an application rather than a library

[zaneselvans]

Background

When we originally packaged up the PUDL repo for installation, we didn't appreciate the distinction between software libraries, frameworks, and applications, and also didn't really know which of those things PUDL was going to become. Initially we imagined that users would run the reproducible ETL pipeline locally and regenerate the outputs, and we followed the Python packaging norms that we encountered most frequently, which were designed around publishing libraries -- abstracted reusable code that could be put to use for a variety of different purposes, i.e. in different applications.

Is PUDL really a library?

It's become increasingly clear that this is not the right way to think about PUDL:

• Most users just want the data in a ready to use form, including useful derived outputs and legible denormalized tables.
• Many users have struggled to get the ETL to run locally at all, and even when they do it's inconvenient to re-run.
• The package now has hundreds of indirect dependencies, which makes using it alongside other libraries challenging.
• As we include larger datasets and more datasets, it only gets harder for users to run the ETL.

Or is PUDL an application?

Instead, PUDL seems more like an application: end-use software that does a particular job, in this case processing a bunch of specific datasets into a coherent whole.

• The data can then be versioned and distributed for re-use with minimal dependencies, as long as it's well documented with metadata and uses standardized formats like SQLite and Apache Parquet.
• Our most recent attempt at doing this kind of encapsulated distribution is the PUDL Intake Catalog, which now contains the PUDL database, our SQLite version of the historical FERC 1 database, the Census DP1 geodatabase, and the EPA CEMS hourly emissions data as Apache Parquet.

Developing PUDL as an application

We've already started making some changes to PUDL to enable continuous deployment and the integration of much more data. Some of the patterns found in the Twelve-Factor App make sense in this context, and so do a lot of the abstractions provided by Dagster.

• Design around ETL being run on cloud resources, except for local testing / development
• Automated nightly builds (see Nightly Data Builds #1177) which continuously deploy their results to publicly available resources
• Persist important intermediate data processing steps in the DB for distribution, to make the data provenance clearer and allow others to make different local data processing decisions (e.g. imputation methods) if they need to.
• Create database views that provide access to denormalized tables
• Output reusable derived values into the database

Proposed changes

Additional complementary changes that could make sense in light of treating PUDL like an application rather than a library:

• Only test and run the ETL on the most recent version of Python
• Use some type of calendar versioning on at least the pudl and pudl-catalog packages.
• Pin all direct and indirect dependencies to an exact version using a tool like poetry or conda-lock

Benefits

• Testing on only the version of Python we expect to use in production means we'd use less CI resources.
• Supporting only the most recent version of Python would mean we could automatically update a lot of the syntax and language constructs to the most recent version with some pre-commit hooks (like pyupgrade and pep585-upgrade), and more fully take advantage of improvements in type hinting (Pydantic has some conflicts with these autoupdate tools and the way they bring new language constructs into older versions of Python, so we can't use them now)
• Calendar versioning would obviously tie released data to the time of its release, which is important given that we're mostly deploying time varying data. People would be able to see how fresh or stale a given release is immediately.
• Calendar versioning would make it clearer which versions of the PUDL software are tied to what versions of the data outputs (PUDL Catalog etc.)
• Pinning all dependencies to specific versions which are known to work together would avoid unexpected breakage when trying to re-run or re-create a previous version of the ETL.

Breaking out reusable components?

There are also a few parts of the PUDL codebase that are potentially more reusable, and could (should?) be broken out into their own packages in separate repositories, and then used by PUDL (and potentially in some of our other projects). This doesn't have to be done now, but would move us gradually in the direction of separating the PUDL application from the more general tools which are used to create it.

• The PUDL Datastore module (which retrieves versioned raw data from the Zenodo archives or cloud object store that caches those archives)
• Pydantic metadata structures (which we use to describe our data sources, and the tables and fields which we use to build the database)

[zaneselvans]

Do folks have feels about the proposed changes? Will those create problems for anyone that we're not thinking about?

Are there any additional portions of the existing PUDL codebase that could/should be broken out into abstracted libraries / packages? Do the two mentioned above make sense?

[bendnorman]

PUDL definitely feels like more of an application than a package like pandas or sqlalchemy. I think treating PUDL as an application will make it much easier to handle dependencies and versioning. Here is a nice blog post about the differences between handling python dependencies for applications and reusable packages.

I'm a big fan of creating separate packages for the Datastore and pydantic metadata tools. These tools would be incredibly helpful for other Catalyst and data engineering projects that deal with Zenodo or need a tool to handle metadata. I've started to jot down some thoughts on how this could be done in #1522.

I think it's important to note that moving towards treating PUDL as an application doesn't mean we don't want people running the ETL and contributing code. We absolutely still want folks to contribute and run the ETL. Treating PUDL as an application will make it even easier to deploy PUDL because all of the dependency versions will be pinned.

I think this transition will affect users who have incorporated a slice of PUDL into their own data processing code. For example, maybe someone calls all of the extract functions but applies their own transformations to the data. We're assuming most users are just interested in the outputs of the ETL, so let us know if this transition would impact your development!

[zaneselvans]

Okay, it sounds like there's a general agreement that we really are making something more like an Application, and it would simplify our lives and set more appropriate user expectations if we made that explicit and set up our packaging around that idea.

[gschivley]

Is it possible to break the analysis module into its own package? Or does the changing database schema make that difficult? PowerGenome uses analysis.mcoe.hr_by_unit() and I agree that the large number of upstream dependencies is a bit painful.

[zaneselvans]

Yeah I could imagine an analysis module -- or a collection of them -- that depends on the pudl-catalog package, so that you know you're getting a particular version of the data that your analysis works with.

Our plan right now is to run stock versions of our analyses and put the results in the databases which we distribute. So there would be the original (relatively) unmolested data, alongside a cleaned up / imputed version, and then also a collection of derived value tables, which would be pretty skinny -- just the primary key columns and the derived values -- which are available on their own, or via database views which provide more human legible denormalized versions of the database tables, with values that are commonly used together, but with the joins stored in the DB itself rather than happening in software on top of it.

[zaneselvans]

But that arrangement wouldn't preclude additional analysis modules from being created that depend on the distributed data catalog.

[zaneselvans]

Thinking about breaking out the metadata structures into their own package, which PUDL would depend on... The raw data archiving process also depends on the metadata to populate some of the metadata fields associated with the Zenodo archives. I wonder if it would make sense to have not only the metadata class infrastructure, but also the metadata definitions encapsulated as a separate package?

Then the archivers could depend on the metdata package rather than all of everything in PUDL. The PUDL Catalog could also depend on the metadata, and be able to generate catalogs with rich metadata without needing to depend on everything in PUDL.

[zaneselvans]

Another nice thing about having the Datastore functionality separated from the main PUDL application would be that we could independently test the Datastore code, including interactions with the flaky-ass Zenodo API, without introducing that flakiness into the main CI / nightly builds (which could entirely rely on the GCS cached version of the raw inputs).

[katie-lamb]

Late to the game here, but I think breaking out the metadata structure into a separate package makes a lot of sense. In addition to the metadata being needed for the PUDL Catalog and archivers, some non-PUDL Catalyst repos have tried to use the PUDL metadata classes and found them to be too intertwined with PUDL to be useful. If it was made into a separate package I think it would also be easier to update for someone with less familiarity with the new structures.

[grgmiller]

PUDL is a dependency of our current project, as we use several of the pudl/analysis functions in our data pipeline. Maybe it would make sense to break out all of the analysis functions into a separate package so that they are easily accessible for others to use in working with the processed data that results from the ETL pipeline?

[grgmiller]

Oops I see this was already asked above: #1669 (reply in thread).

We do also rely on pudl.output.pudltabl in our code, so we would just want to make sure that there continues to be easy access the output tables

[zaneselvans]

Oh yeah, we're definitely not planning to get rid of those outputs, we just want to make them into database tables or database views instead, so that you don't need to install any particular software to access them, and can just slurp them in from the DB and do whatever you want. This will allow you to "install" a given version of the data pretty much in isolation via our intake catalog or to just use the DB directly if you want.