fix: only evaluate modification time on matched local files #235
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
contributes to astronomer/astronomer-cosmos#1075
Description
There is a very hard to reproduce race-condition that occurs in some environments where dbt is used on a concurrently accessed filesystem, like Airflow with Celery workers. The symptom is one process walking the dbt project tree and hitting an error similar to:
It's unclear what actually creates those temp files (I would love to know!), but that error should not even occur, since the filename doesn't match any extension spec that dbt should be searching for. Note, I've seen these files created for all extensions that dbt would be concerned with (.md, .yml, .sql); it's not a problem localized to just model files or macros.
This PR reorders the operations in
find_matching
such that any file names that don't match the regex pattern are skipped beforegetmtime
is even called.Checklist
changie new
to create a changelog entry