[Bug] TypeError: 'NoneType' object is not subscriptable

[vaibhav2ghadge]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Our dbt model suddenly stopped working. Our DBT model is running on Athena and storing data on s3
below is the command for how we are running
dbt run --models demo_model --profiles-dir /opt/dbt-dep/mario/dbt/profiles/ --target dummy_target --vars '{"week_start_date": "2024-10-14","week_end_date": "2024-10-20"}'

Expected Behavior

Model should run on athena and should store data on s3

Steps To Reproduce

dbt run --models demo_model --profiles-dir /opt/dbt-dep/mario/dbt/profiles/ --target dummy_target --vars '{"week_start_date": "2024-10-14","week_end_date": "2024-10-20"}'

Relevant log output

12:38:56  Running with dbt=1.4.1
12:38:56  Unable to do partial parsing because config vars, config profile, or config target have changed
12:38:58  Encountered an error:
'NoneType' object is not subscriptable
12:38:58  Traceback (most recent call last):
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/main.py", line 135, in main
    results, succeeded = handle_and_check(args)
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/main.py", line 198, in handle_and_check
    task, res = run_from_args(parsed)
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/main.py", line 245, in run_from_args
    results = task.run()
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/task/runnable.py", line 454, in run
    self._runtime_initialize()
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/task/runnable.py", line 165, in _runtime_initialize
    super()._runtime_initialize()
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/task/runnable.py", line 94, in _runtime_initialize
    self.load_manifest()
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/task/runnable.py", line 81, in load_manifest
    self.manifest = ManifestLoader.get_full_manifest(self.config)
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/parser/manifest.py", line 203, in get_full_manifest
    manifest = loader.load()
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/parser/manifest.py", line 339, in load
    self.parse_project(
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/parser/manifest.py", line 467, in parse_project
    parser.parse_file(block)
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/parser/base.py", line 425, in parse_file
    self.parse_node(file_block)
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/parser/base.py", line 386, in parse_node
    self.render_update(node, config)
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/parser/models.py", line 362, in render_update
    super().render_update(node, config)
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/parser/base.py", line 362, in render_update
    context = self.render_with_context(node, config)
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/parser/base.py", line 240, in render_with_context
    get_rendered(parsed_node.raw_code, context, parsed_node, capture_macros=True)
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/clients/jinja.py", line 590, in get_rendered
    return render_template(template, ctx, node)
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/dbt/clients/jinja.py", line 545, in render_template
    return template.render(ctx)
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/dbt/dbt-env/lib/python3.8/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 488, in top-level template code
TypeError: 'NoneType' object is not subscriptable

Environment

- OS:Ubuntu 18.04.3 LTS
- Python: 3.8
Core:
  - installed: 1.4.1
 

Plugins:
  - redshift:  1.4.0 - Update available!
  - athena:    1.4.0 - Ahead of latest version!
  - postgres:  1.4.1 - Update available!
  - snowflake: 1.4.1 - Update available!

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

No response

[devmessias]

Do you have any kind of unit test in this model and that model uses var / env_var?

[vaibhav2ghadge]

We dont have unit test for this model
No we are not using any env_var or var below is our files
model file

{{ config(
    materialized = 'incremental',
    incremental_strategy = 'insert_overwrite',
    partitioned_by = ['week_end_date'],
    external_location = 's3://path/') }}

SELECT events.domain AS domain,
       events.app_name AS app_name,
       platform_type,
       events.s AS plugin,
       events.v AS sdk_version,
       events.os AS os_version,
       events.pv AS plugin_version,
       events.rv AS runtime_version,
       COALESCE(events.cc,'') AS country_code,
       COALESCE(events.lang,'') AS language,
       COALESCE(events.av,'') AS app_version,
       COALESCE(events.dm,'') AS device_model,
       approx_distinct(events.id) AS active_users,
       cast(approx_set(events.id) AS varbinary) as device_ids_hll,
       date('2024-10-10') AS week_end_date
FROM table events
WHERE (events.event_date >= date('2024-10-01')
       AND events.event_date <= date('2024-10-10')
       AND events.action_type = 'a'
       AND (domain is not null OR domain <> '')
       AND (platform_type is not null OR platform_type <> '')
       AND (app_name is not null OR app_name <> ''))
GROUP BY events.domain,
         events.app_name,
         platform_type,
         events.s,
         events.v,
         events.os,
         events.pv,
         events.rv,
         events.cc,
         events.lang,
         events.av,
         events.dm,
         date('2024-10-10');

This is profile.yml

version: 2
dwh_athena:
  target: target_1
  outputs:
    prod_hs:
      aws_profile_name: dataown
      database: awsdatacatalog
      region_name: region
      s3_staging_dir: s3://path  # update this
      schema: schema
      threads: 1
      type: athena
      work_group: athena_admin
    target_1:
      aws_profile_name: profile
      database: awsdatacatalog
      region_name: region
      s3_staging_dir: s3://path
      schema: schema
      threads: 1
      type: athena
      work_group: athena_admin

dbt debug

14:45:18  Running with dbt=1.4.1
dbt version: 1.4.1
python version: 3.8.0
python path: /opt/dbt/dbt-env/bin/python
os info: Linux-5.4.0-1110-aws-x86_64-with-glibc2.27
Using profiles.yml file at /path/profiles.yml
Using dbt_project.yml file at /opt/path/dbt/project/path/dbt_project.yml

Configuration:
  profiles.yml file [OK found and valid]
  dbt_project.yml file [OK found and valid]

Required dependencies:
 - git [OK found]

Connection:
  s3_staging_dir: s3://path/dbt
  work_group: group
  region_name: region
  database: awsdatacatalog
  schema: schema
  poll_interval: 1.0
  aws_profile_name: profile
  endpoint_url: None
  s3_data_dir: None
  s3_data_naming: schema_table_unique
  Connection test: [OK connection ok]

[vaibhav2ghadge]

After creating local profile as well its not working giving below error

dbt run --models model_v2 --profiles-dir ./profiles.yml
15:29:36  Encountered an error while reading the project:  ERROR: Runtime Error
  Could not find profile named 'dwh_athena'
15:29:36  Encountered an error:
Runtime Error
  Could not run dbt

proflies.yml

dwh_athena:
        target: prod
        outputs:
                prod2:
                        aws_profile_name: profile
                        database: awsdatacatalog
                        region_name: region
                        s3_staging_dir: s3://path/athena/dbt  # update this
                        schema: schema
                        threads: 1
                        type: athena
                        work_group: workgrp

                prod:
                        aws_profile_name: profile
                        database: awsdatacatalog
                        region_name: region
                        s3_staging_dir: s3://path/athena_admin_workgroup/dbt
                        schema: schema
                        threads: 1
                        type: athena
                        work_group: workgrp

[devmessias]

Sorry, I forgot that unit tests are not available in older versions of dbt. I suspected that you were facing the same issue I had because of the exact same error message due vars. But version 1.4.1 is old. Have you tested this in a newer version of dbt, maybe 1.7 or 1.8?

[dbeatty10]

But version 1.4.1 is old. Have you tested this in a newer version of dbt, maybe 1.7 or 1.8?

@vaibhav2ghadge Thanks for opening this issue -- echoing what @devmessias asked, interested if you get this error with dbt-core 1.7 or 1.8 (since 1.4 is EOL)

[vaibhav2ghadge]

Yes, we are facing the same issues on the latest DBT 1.8.7 & athena adapter 1.8.4.
How can we debug exactly where is the issue? our other DBT model working fine but 2 models stopped working suddenly and the error doesn't show what the exact problem is.

[vaibhav2ghadge]

I am able to run DBT model but I need to pass --vars '{"yesterdays_date": "2024-10-22"}' as param without it dbt model is not working and we are not using yesterdays_date any where in SQL if we change yesterdays_date to any other variable its not working

[devmessias]

That is quite strange. If you're not using yesterdays_date in any SQL model, where do you have it? If you're not using it anywhere, how did you come up with using yesterdays_date?"

[vaibhav2ghadge]

yesterdays_date We are using it in another model just to try I have passed this value to those models that are failing then this model started working. Is there any configuration that we are missing?

[devmessias]

If model B depends on A, and A depends on a variable, I think you would get the expected output if this variable was not available when running model B. If that's not the case, and B doesn't depend on A, could you please share the details of models B and A here?

[devmessias]

One issue (dbt-labs/dbt-common#206) here is that a type error in dbt doesn't provide the model information (I have an open PR for that: dbt-labs/dbt-common#207). If you could provide a minimal example of your project, I can run it on my local dbt branch, where I have this behavior fixed in dbt, and we could narrow down where the issue is occurring.