Skip to main content
412 questions
Newest Active Bountied Unanswered
Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

I have a databricks external table which is pointed at an S3 bucket which contains an ever-growing number of parquet files (currently around 2000 of them). Each row in the file is timestamped to ...
Chris Hunt's user avatar
  • 119
0 votes
0 answers
73 views

If I have a external table set up in Databricks that points to a CSV file (or a folder full of CSV files), each record corresponds to a row in (one of) the file(s). Is there any way to determine the ...
Chris Hunt's user avatar
  • 119
-1 votes
1 answer
88 views

Configuration Database Engine = mybusiness.database.windows.net Records Database = Records --> dbo.accesslogs Legal Database = Legal --> dbo.personalinfo Access Database = Access --> dbo....
CyberVector Workspace's user avatar
  • 1
0 votes
1 answer
70 views

I'm building a database on Starburst using unmanaged (or external) tables that read CSV files stored on S3. The file path structure will be as follows: bucket_name/file_name/year/month/day The CSV ...
ennezetaqu's user avatar
  • 17
0 votes
1 answer
44 views

I am new to hive just completed theory now doing some assignment, I am trying to load the csv data which looks like this : csv_file I have my HQL query I have checked the path for csv file is correct ...
Akhilesh's user avatar
  • 1
0 votes
1 answer
189 views

I originally set the Storage location in my DLT as abfss://{container}@{storageaccount}.dfs.core.windows.net/... But when running the DLT I got the following error: So I decided to leave the above ...
tommyhmt's user avatar
  • 316
0 votes
2 answers
117 views

I have an external table defined within my Databricks database that's pointing to a particular folder within an S3 bucket. There are multiple csv files in the folder, the contents of which all appear ...
Chris Hunt's user avatar
  • 119
0 votes
0 answers
44 views

I am trying to overwrite multiple partitions in a large table. Basically I have my main external S3 table sandbox, partitioned by part: scala> q("select * from sandbox") +---+-------------...
kot's user avatar
  • 85
1 vote
2 answers
252 views

What is the difference between a Managed Table with a location other than DBFS root (I created a schema with Location and created a managed table in that schema)and an External Table in Databricks?
qudsif's user avatar
  • 103
2 votes
1 answer
154 views

I am trying to create external table in snowflake, which should be refreshed whenever we have a new file in the source directory. External table is not producing any result even though my stage shows ...
DEARINE's user avatar
  • 21
0 votes
1 answer
274 views

I'm trying to create an external table on Synapse serverless pool to access a delta table on ADLS gen 2. I first created an external source as shown here. --Create a database scoped credential using ...
Anupam Chand's user avatar
  • 2,779
0 votes
2 answers
102 views

I have a BigQuery Table with about 200,000,000 rows. I have an external table that holds rows that need to have a value updated by having the row to be updated's unique id, and thestring that needs to ...
Ben Delany's user avatar
  • 107
0 votes
2 answers
371 views

According to the GCP official documentation You can query an external data source in BigQuery by using a permanent table or a temporary table https://cloud.google.com/bigquery/docs/external-tables#...
Juanjo's user avatar
  • 103
1 vote
1 answer
225 views

I have a number of Azure SQL databases that I would like to have access an external table in a local SQL Server database (not named instance). The local SQL Server table has a trigger that can send ...
Velocedge's user avatar
  • 1,504
1 vote
1 answer
108 views

I have a .csv file data.csv stored at location: dbfs:/raw/data/externalTables/emp_data_folder/emp_data.csv Here is a sample of the data in the file: Alice,25,50000,North Bob,30,60000,South Charlie,35,...
DumbCoder's user avatar
  • 515
0 votes
1 answer
55 views

I am trying to load CSV and Parquet format files into one external table in Vertica DB but not finding any options, in worst case planning to load it in two different tables and make a union but union ...
NARENDRA's user avatar
  • 395
0 votes
2 answers
165 views

I am trying to create a new external table (kind=delta) in Azure Data Explorer. My code is standard: .create-or-alter external table external_someexport (col1:int, col2:int, mydate:datetime ) kind=...
Werner's user avatar
  • 160
0 votes
1 answer
461 views

I have an external Delta table in Databricks with data files stored in an S3 backend. I want to completely remove(drop) this external Delta table, including all data files and Delta log files from the ...
mayur wadhwani's user avatar
  • 13
1 vote
0 answers
44 views

I have CSV data that looks like id, xml 1, "{\"field1\": 1234, \"field2\": \"some string with a comma,\"}" 2, "{\"field1\": 5678, \"field3\&...
swygerts's user avatar
  • 227
0 votes
1 answer
789 views

I have an existing data base in Azure that I need to replicate in another Azure database as External tables. To this end I need to make a script that creates all these 200+ tables. I’m looking for a ...
Garry_G's user avatar
  • 309
0 votes
0 answers
735 views

I have a BigQuery table that's configured to be an External Table looking at Cloud Storage. The Source URI is: gs://[bucketname]/test_file* I did not specify maxBadRecords in my table creation request,...
Jeffrey Van Laethem's user avatar
  • 2,631
1 vote
0 answers
228 views

Im trying to refresh an external table but it is not working. I need to REFRESH the table manually since I don't have an explicit periodicity to make it automatically. Also read the documentation: ...
Valentina Flores's user avatar
  • 11
1 vote
1 answer
427 views

I am copying files from ftp to ADLS in bronze folder under the partitioned folder heirarchy like bronze | | +----- __run_date__=2023-10-23 | | ...
yash394's user avatar
  • 23
0 votes
1 answer
238 views

I am currently facing an issue(below is the error code) when trying to import data from a .csv file to OracleDB. Now, I have a stored procedure so that I could easily call it out. Now, every time I ...
alhambra's user avatar
  • 13
0 votes
1 answer
698 views

I have e-commerce product data in JSON format. This data is frequently updated. I am planning to load this data into BigQuery. Given that the JSON files are huge (a few hundred attributes) and there ...
Amlan's user avatar
  • 253
0 votes
1 answer
204 views

While creating external table with SQL loader to load data, I used "FILLER" keyword specified as in oracle docs as below. A filler field, specified by BOUNDFILLER or FILLER is a data file ...
Erik's user avatar
  • 396
0 votes
0 answers
340 views

I am trying to take a clone of external dataset's tables using a script. Before that I need to take out DDL statements of all tables and dataset. In INFORMATION_SCHEMA.TABLES, DDL column is NULL for ...
Pratiksha Ratnaparkhi's user avatar
  • 31
0 votes
1 answer
137 views

Is it possible to create a Kusto external table that is connected to an Azure storage account ? The idea is not to connect a csv or parquet file in a container, but an ADLSgen2 Table like this: ....
Werner's user avatar
  • 160
0 votes
1 answer
167 views

I need to populate my EXTERNAL TABLE from a .csv file received as input. Within my .csv file I have 3 fields which are respectively: ORDER_DATE -START -END Which are with the following format: ...
MarioS's user avatar
  • 1
0 votes
1 answer
341 views

I am creating an external table in Synapse serverless pool using data stored in a CSV file. I am using file format as DELIMITEDTEXT with Field Terminator as ",". Now one of my columns has ...
Ahan's user avatar
  • 21
0 votes
0 answers
407 views

I created a folder named "datosext2" on C:\ I created a .txt file named "estados" on C:\datosext2 On sqldeveloper, I created a directory said folder GRANT CREATE ANY DIRECTORY TO ...
Luis Emilio González Ruiz's user avatar
  • 1
0 votes
0 answers
319 views

I have one external table that has only 2 columns and 3 rows in it and when i run query to fetch data using external table that take almost 15 mins to get the data. I am not sure why that much slow as ...
Bharat's user avatar
  • 2,464
0 votes
1 answer
429 views

I would like to create a native table in BigQuery from a CSV file with DDL statements in a way that replicates the "create table from..." feature in BigQuery's UI as shown in the screenshot ...
Downforu's user avatar
  • 359
2 votes
1 answer
862 views

I have many external tables that sits over csv files in S3. I have string values in numeric columns that should be parsed as null, e.g. 'na'. when I query the stage with the file format, this works as ...
em456's user avatar
  • 441
0 votes
1 answer
108 views

I would like to create a database table in an external file outside of the standard data directory. The following works: create table eh_extern (c1 integer) engine=innodb data directory = '/tmp'; I ...
AlexM's user avatar
  • 3
0 votes
1 answer
738 views

I'm trying to create an external table in BigQuery off partitioned parquet data found in GCS. I'm doing this by running the following SQL statement: CREATE OR REPLACE EXTERNAL TABLE `project-id.poc....
Frank Pinto's user avatar
  • 163
0 votes
1 answer
993 views

I have a bigquery table that is connected to a google sheet link. I am going to add an additional column to this google sheet and wanted to know if the bigquery table schema will change to match? If ...
unnest_me's user avatar
  • 233
1 vote
0 answers
144 views

I'm querying redshift spectrum and certain fields are showing up null without any explanation. I've checked SVL_S3LOG SVL_SPECTRUM_SCAN_ERROR SYS_EXTERNAL_QUERY_ERROR And they are all empty. In the ...
RSHAP's user avatar
  • 2,446
0 votes
1 answer
480 views

I'm having issues creating an external table in SQL Server which points to another SQL Server. These are the steps I've taken: CREATE EXTERNAL DATA SOURCE Polybase_ServerTest WITH ( LOCATION = '...
Liem Nguyen's user avatar
  • 99
0 votes
1 answer
1k views

I'm writing a parquet file to an azure data lake storage system, via databricks. I'm using the following to write the parquet file: cultural_split.write.partitionBy("filename").mode('...
Samuel Appleton's user avatar
  • 31
0 votes
1 answer
1k views

I have a situation I would like to know if possible to combat. I currently have a BigQueryUpsertTableOperator that goes to a google cloud storage and creates one external table from all the csv files ...
unnest_me's user avatar
  • 233
0 votes
1 answer
608 views

I am trying to create an External Table in Azure Data Explorer using a user-assigned managed identity. The managed identity has been assigned to the cluster and I am using the statement as defined in ...
Werner's user avatar
  • 160
-1 votes
1 answer
112 views

I have a DAG airflow task where it goes into a specific bucket and creates an external table from all the files inside together. I am getting the error (CSV table references column position 19, but ...
unnest_me's user avatar
  • 233
1 vote
1 answer
741 views

I have a task that reads all files from a specific bucket path. All these files follow the same schema/format, and only difference is data is from each day so the file name is blahblah-(date).csv. I ...
unnest_me's user avatar
  • 233
0 votes
2 answers
684 views

In a Synapse pipeline, I'm trying to use a CETAS script activity from a parquet file that I generate before (from an Azure SQL database). The source database has some tables containing LOB columns (...
Jean-Christophe Rat-Patron's user avatar
  • 13
1 vote
1 answer
541 views

I created a CSV file using a comma to separate fields. There are two field values on each line separated by a comma. I created an external table using two fields and referencing the filespec for the ...
C Matt's user avatar
  • 35
0 votes
1 answer
2k views

I'm using an Azure Synapse Serverless SQL database. I have a first copy activity that uses Azure SQL tables as sources and an Azure storage gen2, where I store .parquet files as sink. From these ....
Jean-Christophe Rat-Patron's user avatar
  • 13
1 vote
1 answer
161 views

Typically when partitioning an external table, I will use 1-day boundaries with the syntax .create external table extTbl ( Timestamp: datetime, Data: dynamic ) kind = blob partition by (...
Peter Vandivier's user avatar
  • 718
1 vote
2 answers
1k views

There are many files which going to be land on gcs bucket daily. We have to merge them all and create a new file copied to new location. I can do this with reading all the files which starts from ...
Vikrant Singh Rana's user avatar
  • 4,749
1 vote
1 answer
2k views

We have our primary data in snowflake and we want to share it with some users over in databricks. The ask is not to "copy the data into databricks" as it will add to the cost. Rather just ...
user2221179's user avatar
  • 83

1
2 3 4 5
…
9