New to elasticsearch and mongodb, i ma facing problem ingesting data from mongodb to es in bulk indexing. To sync up the data i m using mongo-connector and elastic2-doc-manager. I am able to successfuly ingest data from one collection. But when it comes to metadata collections created by django2_tables in the mongodb. Elasticsearch errors out. By the way it worked fine earlier not sure why and how. Perhaps i m missing or forgot some es bulk configuration. Any help would be highly appreciated.
Thanks, Attaching logs from elasticsearch and mongo-connector. log from elasticsearch:
[2020-01-24T12:55:04,898][DEBUG][o.e.a.b.TransportShardBulkAction] [Qasims-MBP.hub] [djangodb01][0] failed to execute bulk item (index) index {[djangodb01][auth_permission][5e2a32540c31b2edc41dd9e6], source[{"id": 32, "name": "Can view project", "content_type_id": 8, "codename": "view_project"}]}
java.lang.IllegalArgumentException: Rejecting mapping update to [djangodb01] as the final mapping would have more than 1 type: [spiderCollection1, auth_permission]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:272) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:238) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.5.1.jar:7.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
log from mongo-connector:
2020-01-24 12:55:04,909 [ERROR] mongo_connector.doc_managers.elastic2_doc_manager:505 - Bulk request failed with exception
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 497, in send_buffered_operations
successes, errors = bulk(self.elastic, action_buffer)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/elasticsearch/helpers/__init__.py", line 257, in bulk
for ok, item in streaming_bulk(client, actions, **kwargs):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/elasticsearch/helpers/__init__.py", line 188, in streaming_bulk
for data, (ok, info) in zip(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/elasticsearch/helpers/__init__.py", line 137, in _process_bulk_chunk
raise BulkIndexError('%i document(s) failed to index.' % len(errors), errors)
elasticsearch.helpers.BulkIndexError: ('40 document(s) failed to index.', [{'index': {'_index': 'djangodb01', '_type': 'django_content_type', '_id': '5e2a32540c31b2edc41dd9bf', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'Rejecting mapping update to [djangodb01] as the final mapping would have more than 1 type: [spiderCollection1, django_content_type]'}, 'data': {'id': 1, 'app_label': 'admin', 'model': 'logentry'}}}, {'index': {'_index': 'djangodb01', '_type': 'auth_permission', '_id': '5e2a32540c31b2edc41dd9c0', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'Rejecting mapping update to [djangodb01] as the final mapping would have more than 1 type: [spiderCollection1, auth_permission]'}, 'data': {'id': 1, 'name': 'Can add log entry', 'content_type_id': 1, 'codename': 'add_logentry'}}}, {'index': {'_index': 'djangodb01', '_type': 'auth_permission', '_id': '5e2a32540c31b2edc41dd9c1', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'Rejecting mapping update to [djangodb01] as the final mapping would have more than 1 type: [spiderCollection1, auth_permission]'}, 'data': {'id': 2, 'name': 'Can change log entry', 'content_type_id': 1, 'codename': 'change_logentry'}}},
