Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]catch_jinja should capture type errors during the rendering phase and display the node information #206

Open
2 tasks done
devmessias opened this issue Oct 16, 2024 · 2 comments · May be fixed by #207
Open
2 tasks done

[Bug]catch_jinja should capture type errors during the rendering phase and display the node information #206

devmessias opened this issue Oct 16, 2024 · 2 comments · May be fixed by #207
Labels
bug Something isn't working

Comments

@devmessias
Copy link

devmessias commented Oct 16, 2024 •
edited
Loading

Is this a new bug in dbt-common?

  • I believe this is a new bug in dbt-common
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

As discussed in the following comment by @jeremyyeo dbt-labs/dbt-core#8276 (comment) on another issue , when there is a type error inside a Jinja template for just one model, the dbt pipeline fails without providing any useful information. I've encountered the same problem several times and spent too much time figuring out what caused it. We can see the same issue appearing in other issues, with users complaining about it:

dbt-labs/dbt-core#4596 (comment).
dbt-labs/dbt-labs-experimental-features#37
dbt-labs/dbt-core#9033

Expected Behavior

  • The user should receive information about the node (model) where the error occurred.

Steps To Reproduce

# dbt_project.yml
name: my_dbt_project
profile: all
config-version: 2
version: 1.0
models:
  my_dbt_project:
    +materialized: table
-- models/foo.sql
{{ config(materialized=-'table') }}

select 1 id

Relevant log output

12:23:14  Running with dbt=1.9.0-b2
12:23:14  Registered adapter: duckdb=1.9.0
12:23:14  Encountered an error:
bad operand type for unary -: 'str'
12:23:14  Traceback (most recent call last):
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/cli/requires.py", line 153, in wrapper
    result, success = func(*args, **kwargs)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/cli/requires.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/cli/requires.py", line 235, in wrapper
    return func(*args, **kwargs)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/cli/requires.py", line 264, in wrapper
    return func(*args, **kwargs)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/cli/requires.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/cli/requires.py", line 327, in wrapper
    setup_manifest(ctx, write=write, write_perf_info=write_perf_info)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/cli/requires.py", line 351, in setup_manifest
    ctx.obj["manifest"] = parse_manifest(
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/parser/manifest.py", line 1998, in parse_manifest
    manifest = ManifestLoader.get_full_manifest(
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/parser/manifest.py", line 308, in get_full_manifest
    manifest = loader.load()
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/parser/manifest.py", line 413, in load
    self.parse_project(
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/parser/manifest.py", line 710, in parse_project
    parser.parse_file(block)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/parser/base.py", line 499, in parse_file
    self.parse_node(file_block)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/parser/base.py", line 468, in parse_node
    self.render_update(node, config)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/parser/models.py", line 349, in render_update
    super().render_update(node, config)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/parser/base.py", line 444, in render_update
    context = self.render_with_context(node, config)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/parser/base.py", line 272, in render_with_context
    get_rendered(parsed_node.raw_code, context, parsed_node, capture_macros=True)
  File "/home/devmessias/phd/pkgs/dbt-my/core/dbt/clients/jinja.py", line 144, in get_rendered
    rendered = render_template(template, ctx, node)
  File "/home/devmessias/phd/pkgs/dbt-common/dbt_common/clients/jinja.py", line 585, in render_template
    return template.render(ctx)
  File "/home/devmessias/phd/pkgs/dbt-common/dbt_common/clients/jinja.py", line 174, in render
    return self.environment.handle_exception()
  File "/home/devmessias/phd/pkgs/dbt-my/.venv/lib/python3.9/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
TypeError: bad operand type for unary -: 'str'

Additional Context

I created an issue on dbt-core, but I think here is the correct place dbt-labs/dbt-core#10856
@devmessias devmessias added bug Something isn't working triage labels Oct 16, 2024
@devmessias devmessias linked a pull request Oct 16, 2024 that will close this issue
Open
5 tasks
@devmessias devmessias mentioned this issue Oct 23, 2024
Open
2 tasks
@dbeatty10 dbeatty10 mentioned this issue Oct 28, 2024
Closed
3 tasks
@dbeatty10
Copy link
Contributor

dbeatty10 commented Oct 28, 2024 •
edited
Loading

@devmessias This is a great idea to avoid that nasty stacktrace in a variety of scenarios 🏆

Reprex

In addition to the one you provided, here's another reprex (originally offered by @jtcohen6 in dbt-labs/dbt-core#4596 (comment) and you also included in dbt-labs/dbt-core#10856 that will give an error that says "'NoneType' object is not iterable":

Create this file:

models/my_model.sql

{% set mylist = None %}

{% for item in mylist %}

  select '{{ item }}' as my_item
  {{ 'union all' if not loop.last }}

{% endfor %}

Run this command:

dbt parse

@dbeatty10 dbeatty10 removed the triage label Oct 28, 2024
@devmessias
Copy link
Author

Hi @dbeatty10, thanks for pointing out this example. With my PR #207, here is the output:
image

I'Ve created a new unit test to test if it's correct handling a type error when iterating over a None value.

c9b8667

I'll wait for the #207 review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: catch jinja typeerrors during render phase devmessias/dbt-common
2 participants
@devmessias @dbeatty10