Error creating index (error on compacting)

[johny65]

ArcadeDB Version:

24.11.1 (ArcadeDB Server v24.11.1 (build b466f48/1730841732933/main))

OS and JDK Version:

Ubuntu 24.10, JDK 21.0.5 (Linux 6.11.0-8-generic - OpenJDK 64-Bit Server VM 21.0.5 (Temurin-21.0.5+11))

Expected behavior

To create an index successfully (with SQL create index on Type(prop) notunique).

Actual behavior

I have a document type AuditLog with 135+ millions of documents, when I execute create index sentence on a property, in the server logs several times it logs something like this:

[LSMTreeIndex] - Creating a new entry in index 'AuditLog_0_95902170898792(9)' root page [#8652:23621]->94 (entry in page=93 threadId=75)...............Error on executing compaction of index 'AuditLog_0_95902170898792'
java.lang.IllegalArgumentException: Invalid position -4472603 (size=2097144)
	at com.arcadedb.database.Binary.position(Binary.java:170)
	at com.arcadedb.database.Binary.putByteArray(Binary.java:291)
	at com.arcadedb.index.lsm.LSMTreeIndexCompacted.appendDuringCompaction(LSMTreeIndexCompacted.java:125)
	at com.arcadedb.index.lsm.LSMTreeIndexCompactor.compact(LSMTreeIndexCompactor.java:234)
	at com.arcadedb.index.lsm.LSMTreeIndex.compact(LSMTreeIndex.java:231)
	at com.arcadedb.database.async.DatabaseAsyncIndexCompaction.execute(DatabaseAsyncIndexCompaction.java:43)
	at com.arcadedb.database.async.DatabaseAsyncExecutorImpl$AsyncThread.run(DatabaseAsyncExecutorImpl.java:136)

The process continues despite the errors. When I run a select that would use the index, the server logs:

[FetchFromTypeExecutionStep] <ArcadeDB_0> Attempt to scan type 'AuditLog' of total size 47,61GB 1 times. This operation is very expensive, consider using an index

Steps to reproduce

I dont't know if it's occurred because of something in the data or why.

Note:

Running select count(*) from Type on a type with millions of documents, is very slow... on my AuditLog type previously mentioned, counting documents takes several minutes... is this slow counting so by design?

[lvca]

Counting should be instantaneous, unless you have a WHERE condition.

About the compaction error, could you please try to rebuild the index with the rebuild index command?

[johny65]

Hi. Rebuilding the index keeps throwing the same errors...

Counting should be instantaneous, unless you have a WHERE condition.

Counting from Console is indeed instantaneous:

{auditdb}> select count(*) from AuditLog

+--------+---------+
|NAME    |VALUE    |
+--------+---------+
|count(*)|135787733|
+--------+---------+
Command executed in 32ms

But same count on Studio, it takes several minutes. On this case, server logs:

[FetchFromTypeExecutionStep] <ArcadeDB_0> Attempt to scan type 'AuditLog' of total size 47,61GB 1 times. This operation is very expensive, consider using an index

[lvca]

Commands executed from Studio are adding a default limit to avoid the browser crashes with thousands or more records to render. For this reason, I think the optimizer is not smart enough and it simply scans the whole type. It's a bug, checking into the code.

[lvca]

About the compaction error, I'd love to have those files. Is it possible to have zipped?

[johny65]

I can share you a reduced version of the database where the error can be replicated. Where or how do I send it to you?

[lvca]

Could you please send it to support at arcadedata dot com? Thanks!