Setting collection in MarkLogic

Question

I have a requirement where I have to set collection to the existing documents. The thing is I have around 20 million records. I am running below query from query console. It is throwing time out error. I also tried out limit=N option in below query. At max I was able to achieve N=40000, after that it's again throwing time out error. Please help me with any faster query or approach.

for $each in cts:uri-match("/data/employee/*") return xdmp:document-set-collections($each, "employee")

Creating a CORB batch job would likely be a good solution. developer.marklogic.com/code/corb — Mike Gardner
– Mike Gardner, Commented Nov 8, 2018 at 18:55
Batching up scales better, and is overall faster. There are various ways, including CORB. See also stackoverflow.com/a/52953123/918496 some more.. — grtjn
– grtjn, Commented Nov 8, 2018 at 19:13

rjrudin · Accepted Answer · 2018-11-11 22:33:56Z

1

ml-gradle has OOTB support for this, no code needed. See https://github.com/marklogic-community/ml-gradle/wiki/DMSDK-Tasks#trying-it-out and look at "mlAddCollections".

answered Nov 11, 2018 at 22:33

rjrudin

2,24211 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Nikunj Vekariya Over a year ago

Yes, I do remember there is a task with that name. Is there a similar task we can use through roxy framework?

rjrudin Over a year ago

You could use corb with roxy - easiest approach would be with an ad hoc module. But you can always add a build.gradle file to your Roxy project for the sole purpose of using ml-gradle tasks that aren't related to a deployment.

Navin Rawat · Accepted Answer · 2018-11-12 11:54:44Z

Batching is the right option to perform this type of task. Use xdmp:spawn-function() function to put multiple tasks at a time on the task server. You just need to indentify the number of records which your query can finish within 10 minute or 1 hr as you required.

For example, if query can execute 5000 records within 10 minute:

let $total-records := xdmp:estimate(collection())
let $batch-size := 5000
let $pagination := 0
for $records in 1 to fn:ceiling($total-records  div $batch-size )
let $start := fn:sum($pagination + 1)
let $end := fn:sum($batch-size + $pagination)
let $_ := xdmp:set($pagination, $end)
return
  xdmp:spawn-function
  (
  function(),
  for $each in cts:uri-match("/data/employee/*")[$start to $end]
  return xdmp:document-set-collections($each, "employee")
  )

Collectives™ on Stack Overflow

Setting collection in MarkLogic

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related