Is PUDL tied to EIA's V1 API and best means of building a PUDL pipeline

[jmg80525]

Is PUDL dependent on EIA's V1 API which deprecates in October? Why did you choose to stop updating the database after 2020? It's a little murky to me how I go about building a pipeline to update and maintain the PUDL database. Is that part of the python tools. Sorry for the newbie question.

[zaneselvans]

Hey there, sorry I didn't see this until this evening.

Neither the EIA (860/923) nor the FERC Form 1 data for 2021 have been released yet. The final annual EIA data typically comes out in September or October. FERC 1 (and all the other FERC forms) are transitioning to XBRL instead of Visual FoxPro this year, and that's been a rocky process -- both for them, and for us. See our FERC XBRL extractor project that @zschira is working on.

We're also just about to remove the EIA API dependence (see #1696, #1343), and are going to be filling in the missing fuel prices ourselves instead (though potentially validating our imputations based on the EIA API numbers, if they contain information from the redacted fuel prices).

[jmg80525]

Thanks for the reply. To summarize, although 923 is released on a monthly basis, PUDL is only updating on an annual basis. Is this because the periodicity of FERC reports (not familiar wit them) is on an annual basis? Any plans to incorporate 930 data? Thanks again.

[zaneselvans]

They release preliminary 923 data on a monthly basis, but the format isn't quite the same, the data gets revised, and only some plants report monthly -- most report annually. The FERC Forms 1 and 714 are also only released annually and we integrate that data with the EIA data. We do periodically update the generator level information using the EIA 860m throughout the year. We don't currently have the capacity to integrate every incremental update, since there are always new issues introduced, and then it all ends up getting revised anyway, and it's only partial data.

We haven't prioritized the EIA 930 because it's already available programmatically via EIA's API, while the generator and boiler level information, and individual fuel deliveries from the EIA-923 are not available through the EIA's API -- it only includes aggregates, which aren't as useful for understanding the finances and efficiency of individual generators.