Errors on large Neo4j import files

[adk8673]

Hello - I'm creating a proof of concept for a new feature on our application. I'm working on moving data from existing data store (sql server) into an ArcadeDB instance. I used a file in the neo4j export format and this worked great when the number of entities (both vertices and edges) was around 55,000-60,000. I created another extract of our data with more rows (around 250,000 entities, which is still much smaller than our final amount of data.

With this file I'm getting the following error:
2023-02-03 20:36:33.766 INFO [SourceDiscovery] Analyzing url: file:///data/arcadedb-data/import/testOutput.jsonl...
2023-02-03 20:36:33.770 INFO [SourceDiscovery] Recognized format Neo4j (parsingLimitBytes=9.54MB parsingLimitEntries=0)
2023-02-03 20:36:33.770 INFO [Importer] Checking schema...Checking Neo4j database from file 'null'...
Creation of the schema: types, properties and indexes

• Creation of vertices started
  Status update: parsed 0 (0/sec - 0%) - 0 records (0/sec) - 0 vertices (0/sec) - 0 edges (0/sec) - 0 skipped edges - 0 linked edges (0/sec - 0%)

ERROR: Error on importing database

I checked the logs and I don't see any other output beyond what's listed above. It's possible there some setting I need to enable but I haven't had any luck finding any settings that would help. I apologize if this is addressed in the documentation - I checked and didn't see anything that seemed to be related.

[lvca]

Is there a way you can share the import file with the support team?

[adk8673]

Okay, I've reproduced the issue with this file that has none of our internal schema. I changed the file extension from .jsonl to .txt to upload it but otherwise is the file I just used.
testOutput.txt

[lvca]

I finally had the time to test importing your file and I see an error. The same record is present twice:

{"type":"node","id":"M74516","labels":["Entity2"],"properties":{}}
{"type":"node","id":"M74516","labels":["Entity2"],"properties":{}}

I used the console to import the database:

bin/console.sh -b "create database issue829;set verbose=3;import database file://./testOutput.jsonl"

[adk8673]

Thank you - not sure how I missed that. Appreciate your help.

[lvca]

I think even if duplicates, the import shouldn't stop, but just give some warnings. Pushing a fix on it soon.

[lvca]

Ok, just pushed in main branch. We still have the issue with compaction to resolve and it's more complicated than expected, but we're working on it.