You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a large complex structure with *struct, *string, *int64, *bool, *map[string]*struct, map[string][]*string
with 1 TB of records I need to process.
If any of the 10 records error's in my case it's erroring in reader.go line 337 during the marshal.Unmarshal
i.e
if err2 := marshal.Unmarshal(&tmap, b, e, dstList[index], pr.SchemaHandler, prefixPath); err2 != nil {"
After erroring it returns runtime error: index out of range [x] with length x and doesn't send back any successful marshal.Unmarshal records. Causing the application to lose the 10 records.
WORK·A·ROUND
If I set pr, err := reader.NewParquetReader(fr, nil, 1) // NP-> 1 int64 parallel number
and
github.com/xitongsys/parquet-go v1.6.2
###PROBLEM
I have a large complex structure with *struct, *string, *int64, *bool, *map[string]*struct, map[string][]*string
with 1 TB of records I need to process.
If I run
If any of the 10 records error's in my case it's erroring in reader.go line 337 during the marshal.Unmarshal
i.e
if err2 := marshal.Unmarshal(&tmap, b, e, dstList[index], pr.SchemaHandler, prefixPath); err2 != nil {"
After erroring it returns
runtime error: index out of range [x] with length x
and doesn't send back any successful marshal.Unmarshal records. Causing the application to lose the 10 records.WORK·A·ROUND
If I set
pr, err := reader.NewParquetReader(fr, nil, 1) // NP-> 1 int64 parallel number
and
then I only skip the 1 unprocessed
marshal.Unmarshal
but this make the process slow down x10.Does anyone have any suggestions to help me with this error or speed up this process?
-Stan
The text was updated successfully, but these errors were encountered: