fix(duckdb): return null typed pyarrow arrays and disable creating tables with all null columns in duckdb #9810

cpcloud · 2024-08-09T13:26:08Z

Address all-NULL column handling in the DuckDB backend. null pyarrow Arrays are returned (previously int32), and it is now an error on the Ibis side to create all null columns with create_table in the DuckDB backend. Fixes #9669.

github-actions · 2024-08-09T13:26:52Z

ACTION NEEDED

Ibis follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message.

Please update your PR title and description to match the specification.

ncclementi · 2024-08-12T15:55:48Z

@cpcloud The flink backend is failing, it doesn't like the null type column. I got it to xfail by adding something similar to what you added in _register_in_memory_table() for other backends, in the execute() for flink, although maybe it could go higher up in the to_pyarrow() method. Looks like for flink we don't have the _register_in_memory_table() method.

In backends/flink/__init__.py :

    def execute(self, expr: ir.Expr, **kwargs: Any) -> Any:
        """Execute an expression."""
        self._register_udfs(expr)

        table_expr = expr.as_table()

        if null_columns := table_expr.op().schema.null_fields:
            raise exc.IbisTypeError(
                f"{self.name} cannot yet reliably handle `null` typed columns; "
                f"got null typed columns: {null_columns}")

        sql = self.compile(table_expr, **kwargs)
        df = self._table_env.sql_query(sql).to_pandas()

NickCrews · 2024-09-18T16:06:14Z

@cpcloud im motivated to help this one across the finish line. If I fix the flink tests is this good to merge? Any thoughts regarding my incline comments?

NickCrews

A few thoughts, but thank you very much for this fix! I would love to help push this across the finish line.

If you point me to what you think needs to happen with flink, I can help
if you give a thumbs up to my suggestions, I can implement them

NickCrews · 2024-09-18T15:47:27Z

ibis/expr/schema.py

@@ -67,6 +67,10 @@ def types(self):
    def geospatial(self) -> tuple[str, ...]:
        return tuple(name for name, typ in self.fields.items() if typ.is_geospatial())

+    @attribute
+    def null_fields(self) -> tuple[str, ...]:


I think for the purpose of this PR we need to recurse into nested types, eg if a structure field is Nulltype we need to deal with that too...

This should be done in a follow-up.

In the followup, the semantics of this are going to need to change. Since this is internal-only, I think it is fine for this churn to happen, but just writing this down for when I/we implement it.

We use this for two things:

To check whether we need to do the pyarrow fixup for duckdb, eg if any field or sub-field is null.

To provide nice error messages when trying to create_table() a table-with-null-types, eg

if null_columns := schema.null_fields: raise com.IbisTypeError( f"{self.name} cannot yet reliably handle `null` typed columns; " f"got null typed columns: {null_columns}"

OK, for these two use cases, consider:

ibis.Schema( { "s": "struct<n: null>", "a": "array<null>", "m1": "map<string: null>", "m2": "map<null: string>", "n": "null" } ).null_fields

What should this be? Could return something ibis-internal that isn't really machine-interpretable, and only for this error message, that is like jq's DSL, like ("s.n", "a<items>", "m1<values>", "m2<keys>", "n")? I think that would suffice for our two needs.

NickCrews · 2024-09-18T15:49:37Z

ibis/backends/tests/test_export.py

+    reason="unable to handle null typed columns as input",
+)
+def test_all_null_column(con):
+    t = ibis.memtable({"a": [None]})


Can we add a struct<x: null> and an array column to check for these nested types as well?

In a follow up.

ibis/backends/duckdb/converter.py

cpcloud · 2024-10-22T14:26:44Z

@NickCrews Can you give this another review pass and/or approve?

NickCrews · 2024-10-22T20:23:37Z

ibis/backends/tests/test_export.py

@@ -27,6 +28,7 @@

 pd = pytest.importorskip("pandas")
 pa = pytest.importorskip("pyarrow")
+pat = pytest.importorskip("pyarrow.types")


It would be safe to just do import pyarrow.types as pat, right? If that gets us better IDE/typing support, I would advocate for that, even if it doesn't match the pa = pytest.importorskip("pyarrow") lines above.

but that's a nit, also fine keeping it as is

NickCrews · 2024-10-22T20:48:16Z

@cpcloud I tacked on two commits that I think are improvements, if those pass CI and you are happy, looks great to me to merge. I can then, in a followup PR, tackle adding in the nested null support that I describe in this comment. Thanks!

NickCrews · 2024-10-22T20:57:08Z

Actually, hold on, we aren't converting null-typed scalars correctly. Pushing up a fixup commit soon.

NickCrews · 2024-10-22T21:00:45Z

OK, with that I think I'm happy.

NickCrews · 2024-10-22T21:03:24Z

ibis/backends/duckdb/tests/test_client.py

@@ -420,3 +421,24 @@ def test_memtable_doesnt_leak(con, monkeypatch):
    df = ibis.memtable({"a": [1, 2, 3]}, name=name).execute()
    assert name not in con.list_tables()
    assert len(df) == 3
+
+
+def test_create_table_with_nulls(con):


Can you create null-typed columns in other backends?

cpcloud added bug Incorrect behavior inside of ibis datatypes Issues relating to ibis's datatypes (under `ibis.expr.datatypes`) duckdb The DuckDB backend labels Aug 9, 2024

cpcloud changed the title ~~fix(duckdb): return null typed pyarrow arrays and disable creating tables with all null columns in duckdb.~~ fix(duckdb): return null typed pyarrow arrays and disable creating tables with all null columns in duckdb Aug 9, 2024

cpcloud requested review from gforsyth and jcrist August 9, 2024 13:30

cpcloud force-pushed the all-nulls branch 2 times, most recently from 8f42f56 to 3784885 Compare August 9, 2024 16:56

cpcloud force-pushed the all-nulls branch from 3784885 to d3eee9c Compare September 23, 2024 10:17

cpcloud added this to the 10.0 milestone Sep 23, 2024

cpcloud force-pushed the all-nulls branch from d3eee9c to 1e7904c Compare September 23, 2024 11:25

NickCrews requested changes Sep 24, 2024

View reviewed changes

cpcloud force-pushed the all-nulls branch from 5ee91ef to 71333fc Compare October 8, 2024 10:25

cpcloud added 3 commits October 22, 2024 09:17

chore: add nulls property to quickly check for null columns

94617cb

fix(duckdb): support memtables with all null columns in pyarrow output

b900251

fix(duckdb): disallow creating null-typed columns with duckdb

8a4be61

cpcloud added 5 commits October 22, 2024 09:17

chore(sql): disallow creating null-typed columns with in-memory tables

0cf7b66

chore(flink): clean up memtable support to use hooks and finalization

7cb3797

test(flink): enable xpassing memtable cleanup test

d1470ba

chore(flink): provide flink memtable creator with schema

83044f4

test(flink): fix failing tests

64baaab

cpcloud force-pushed the all-nulls branch from 71333fc to 64baaab Compare October 22, 2024 13:18

NickCrews reviewed Oct 22, 2024

View reviewed changes

NickCrews added 2 commits October 22, 2024 12:40

chore: use .convert_column() inside PyArrowData.convert_table()

9a8b951

chore: refactor null-schema handling in DuckDBBackend.create_table()

01e9e8d

fix(duckdb): handle null-typed scalars in to_pyarrow()

f666ff8

NickCrews reviewed Oct 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(duckdb): return null typed pyarrow arrays and disable creating tables with all null columns in duckdb #9810

fix(duckdb): return null typed pyarrow arrays and disable creating tables with all null columns in duckdb #9810

cpcloud commented Aug 9, 2024

github-actions bot commented Aug 9, 2024

ncclementi commented Aug 12, 2024

NickCrews commented Sep 18, 2024

NickCrews left a comment

NickCrews Sep 18, 2024

cpcloud Oct 8, 2024

NickCrews Oct 22, 2024

NickCrews Sep 18, 2024

cpcloud Oct 22, 2024

cpcloud commented Oct 22, 2024

NickCrews Oct 22, 2024

NickCrews Oct 22, 2024

NickCrews commented Oct 22, 2024 •

edited

Loading

NickCrews commented Oct 22, 2024

NickCrews commented Oct 22, 2024

NickCrews Oct 22, 2024 •

edited

Loading

fix(duckdb): return null typed pyarrow arrays and disable creating tables with all null columns in duckdb #9810

Are you sure you want to change the base?

fix(duckdb): return null typed pyarrow arrays and disable creating tables with all null columns in duckdb #9810

Conversation

cpcloud commented Aug 9, 2024

github-actions bot commented Aug 9, 2024

ncclementi commented Aug 12, 2024

NickCrews commented Sep 18, 2024

NickCrews left a comment

Choose a reason for hiding this comment

NickCrews Sep 18, 2024

Choose a reason for hiding this comment

cpcloud Oct 8, 2024

Choose a reason for hiding this comment

NickCrews Oct 22, 2024

Choose a reason for hiding this comment

NickCrews Sep 18, 2024

Choose a reason for hiding this comment

cpcloud Oct 22, 2024

Choose a reason for hiding this comment

cpcloud commented Oct 22, 2024

NickCrews Oct 22, 2024

Choose a reason for hiding this comment

NickCrews Oct 22, 2024

Choose a reason for hiding this comment

NickCrews commented Oct 22, 2024 • edited Loading

NickCrews commented Oct 22, 2024

NickCrews commented Oct 22, 2024

NickCrews Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

NickCrews commented Oct 22, 2024 •

edited

Loading

NickCrews Oct 22, 2024 •

edited

Loading