ArangoDB batch mode doesn't work with Java Driver

Question

I am evaluating ArangoDB using spring batch.

I tried to insert some data and, without batch mode, it works as expected.

However, if batch mode is on, the execution of the program hangs.

I am using arango 2.3.3 and com.arangodb:arangodb-java-driver:[2.2-SNAPSHOT,2.2]

    arangoDriver.startBatchMode();

    for(Account acc : items){
        acc.getRecordHash();
        acc.getIdHash();
        arangoDriver.createDocument("AccountCollection", acc);
    }

    arangoDriver.executeBatch();

Any ideas what I am doing wrong?

How did you create the arango driver object. I tried importing the ArangoDriver class the package doesnot exist. Is there any place I can refer to create an object like this. — pacman
– pacman, Commented Dec 17, 2020 at 11:54

florian · Accepted Answer · 2014-12-29 15:49:57Z

I tried to reproduce what you are trying, first of all does the collection "AccountCollection" exist ? If not you would get an error in the batch result but still the program should not hang. i created a unittest :

@Test
  public void test_StartCancelExecuteBatchMode() throws ArangoException {

    driver.startBatchMode();

    ArrayList<Account> items = new ArrayList<Account>();
    items.add(new Account());
    items.add(new Account());
    items.add(new Account());
    items.add(new Account());

    for(Account acc : items){
      acc.getRecordHash();
      acc.getIdHash();
      driver.createDocument("AccountCollection", acc, true, false);
    }

    driver.executeBatch();

  }

This works perfectly and returns:

EOB 16:47:01.862 [main] DEBUG com.arangodb.http.HttpManager - [RES]http-POST: statusCode=200 16:47:01.862 [main] DEBUG com.arangodb.http.HttpManager - [RES]http-POST: text=--dlmtrMLTPRT Content-Type: application/x-arango-batchpart Content-Id: request1

HTTP/1.1 202 Accepted Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033214501 Content-Type: application/json; charset=utf-8 Etag: "48033214501" Content-Length: 95

{"error":false,"_id":"AccountCollection/48033214501","_rev":"48033214501","_key":"48033214501"} --dlmtrMLTPRT Content-Type: application/x-arango-batchpart Content-Id: request2

HTTP/1.1 202 Accepted Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033411109 Content-Type: application/json; charset=utf-8 Etag: "48033411109" Content-Length: 95

{"error":false,"_id":"AccountCollection/48033411109","_rev":"48033411109","_key":"48033411109"} --dlmtrMLTPRT Content-Type: application/x-arango-batchpart Content-Id: request3

HTTP/1.1 202 Accepted Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033607717 Content-Type: application/json; charset=utf-8 Etag: "48033607717" Content-Length: 95

{"error":false,"_id":"AccountCollection/48033607717","_rev":"48033607717","_key":"48033607717"} --dlmtrMLTPRT Content-Type: application/x-arango-batchpart Content-Id: request4

HTTP/1.1 202 Accepted Location: /_db/unitTestDatabase/_api/document/AccountCollection/48033804325 Content-Type: application/json; charset=utf-8 Etag: "48033804325" Content-Length: 95

{"error":false,"_id":"AccountCollection/48033804325","_rev":"48033804325","_key":"48033804325"} --dlmtrMLTPRT--

But even when i create intentional errors the application never "hangs". Frank just sent me your source code, i take a look into it. Can you try to find out where the programm is hanging ? is "executeBatch" reached at all ?

florian · Accepted Answer · 2014-12-30 15:06:37Z

0

I already imported 1.6 Mio documents with your code and still everything works. i guess it might be necessary to monitor your system resources during the import, if anything unusual occurs let us now. Generally speaking it does not seem to be the best practice to perform a one-time bulk import like this using the java api. i would recommend to use arangoimp to import the data directly into the database, this will be much faster. it is documented here

answered Dec 30, 2014 at 15:06

florian

2631 silver badge4 bronze badges

Comments

fceller · Accepted Answer · 2015-01-05 11:17:10Z

0

You need to increase the number of open file descriptors. The Mac has a very low limit (256). ArangoDB stores the data in datafiles of a certain chunk size. With large datasets more files are need (and some fd are already used for communication and other stuff).

When ArangoDB runs out of file descriptors, it can neither extend the dataset nor answer new questions. Therefore the import process will hang.

answered Jan 5, 2015 at 11:17

fceller

2,77417 silver badges19 bronze badges

Collectives™ on Stack Overflow

ArangoDB batch mode doesn't work with Java Driver

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related