I need to do a full extract (SELECT * FROM) on i series DB2 database. I am using Copy Activity which fails on the source extract due to special character encodings:
Error code 2200
Failure type User configuration issue
Details Failure happened on 'Source' side. ErrorCode=DB2DriverRunFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error thrown from driver. Sql code: '-343',Source=Microsoft.DataTransfer.ClientLibrary.Db2Connector,''Type=Microsoft.HostIntegration.DrdaClient.DrdaException,Message=HISMPCB0001 In BasePrimitiveConverter an exception has occurred. Exception description: Output buffer is smaller than required size 12 SQLSTATE=HY000 SQLCODE=-343,Source=Microsoft.HostIntegration.Drda.Requester,'
I've researched similar issues here: https://learn.microsoft.com/en-us/answers/questions/467456/failure-happened-source-side-in-copy-activity-for.html
I believe that the source of the issue is native ADF connector's data-type mapping for DB2: DB2 data types are converted to interim data types used internally within the service when loading into the sink. These interim data types seem to be incompatible, causing failure at the source:
https://learn.microsoft.com/en-us/azure/data-factory/connector-db2?tabs=data-factory#data-type-mapping-for-db2
I've tried the following unsuccessfully:
- Adding fault tolerance to skip incompatible rows
- Changing datatypes for incompatible columns
Playing around with datatyping on ingest query works for some columns, but not others -- it is an inconsistent solution, looking for advice on how fix this and keep full source query as is.
