Questions tagged [csv]
The csv tag has no summary.
34 questions
0
votes
4
answers
501
views
What is a suitable format for writing large amounts of data within few milliseconds
I have a c++ code that needs to store some data whenever an event is triggered. The data contains about 3000 floating point values. So each of these values needs to be written in a file when the event ...
- 133
4
votes
1
answer
340
views
When do regression models outperform naive methods?
Case1:
I have the following task to do: Training by the consecutive 3 days to predict the each 4th day. Each day data represents one CSV file which has dimension 24x25. Every datapoints of each CSV ...
- 37
4
votes
5
answers
3k
views
The best way to handle exceptions?
I have the following method, which needs to return a List, but exceptions might occur along the way, and I don't want to handle them in my application, mainly because I don't even know how to handle ...
- 75
2
votes
8
answers
1k
views
Fail fast is brittle
I am creating a CSV consumer (with Java). There is one field / column that should contain one of the values "Rename" or "Move".
I implemented this by allowing mixed case of letters,...
- 4,670
6
votes
1
answer
365
views
What to do when test data takes up most of repo size?
TL;DR: Given a repo with a lot of big fixture tests, when should I look for a solution where the golden files are not part of the repo anymore? Where should they be stored?
The setup
Testing a ...
- 169
-1
votes
1
answer
193
views
Loading in and Autocompleting a CSV File React
I'm currently working on a React Application with Firebase that involves a ton of data manipulation and reading. One specific feature is the ability to autofill a form based on a very large CSV file (...
- 23
1
vote
4
answers
905
views
System Design: Efficient way to import hundreds of different CSV formats
OVERVIEW
We have multiple data providers who provide us information from a large set of data sources. These end data sources provide data in CSV format.
Since, the CSV formats are not consistent ...
- 365
2
votes
3
answers
2k
views
Back-end solution for pulling from CSV files
I'm building a data visualization that displays COVID information for the United States, at the city, state, and county level.
The ultimate source of truth are three CSVs published by the New York ...
- 137
-5
votes
1
answer
68
views
How is data stored prior to downloading?
I go to several sites that offer their data in XML, JSON, CSV formats. As shown here,
https://www.space.org/basicspacedata/query/class/boxscore/format/csv, I choose the format. Question is how is ...
- 101
0
votes
1
answer
106
views
A program design question: Good idea using HDFS in c for reading large data?
I have mainly three groups of CSV files (each file is divided into several small files): First group of CSV files have 600+ GB in total (MAYBE 200+ GB if in int, cause CSV calculates by char right?), ...
- 101
-2
votes
1
answer
334
views
Store line numbers of CSV in the database
In our environment, we often have to import CSV files into a database, where each line represents a record. Though there is no actual business use case for storing the original line numbers inside the ...
- 141
0
votes
1
answer
161
views
Generating CSV export
I created a CSV export that works like the code below. There is a LinkedHashMap where the keys are the column title and values are functions where certain properties are read.
By reording the lines ...
- 141
0
votes
2
answers
1k
views
Passing csv files between regions
I have a desktop app that generates numerous csv files. The application always runs on PCs that have been configured for the English (United Kingdom) region/format, but it's feasible that files might ...
- 1,145
50
votes
4
answers
46k
views
Should UTF-8 CSV files contain a BOM (byte order mark)?
Our line-of-business software allows the user to save certain data as CSV. Since there are a lot of different formats (all called "CSV") in use in the wild, we are tying to decide what the &...
- 9,868
1
vote
1
answer
159
views
Storing survey data from csv to xml
I've been tasked with taking a sample of survey results that are stored in CSV and to convert them to XML.
I'm still fairly new to XML, but have come up with a couple of layouts I could use and was ...
- 43
3
votes
1
answer
128
views
Meaningful response to the user after his uploaded CSV was processed?
A user uploads a CSV to the server to be inserted into the database and I would like to return a meaningful response to this user.
My thoughts :
{
inserted : x records
discarded : y records
}
...
- 181
5
votes
2
answers
6k
views
Multi-level validation in C#
I have a Console project reads inputs from CSV file and tries to save them to database.
For that, I created a class Person that maps a CSV row.
The CSV file has two columns Name and Age. Person ...
- 195
4
votes
5
answers
10k
views
Import large csv files
I've been tasked to query 2 large csv files which are roughly 1 GB in size each. The files contain related data, so file one may contain a list of order numbers, Order Dates, etc and the file may ...
- 43
-1
votes
1
answer
143
views
Need help creating a database of map polygons
As a side project with my coworkers -- we are oil and gas geologists -- we have been creating an interactive map-based learning tool for other geologists.
We are having trouble building the database, ...
- 17
2
votes
2
answers
2k
views
SQL query or C# .net code for csv files import?
My aim here is to find out the best possible and feasible solution for my dilemma. I want to import some csv file (may contain around 50~60K records) into database after some manipulation into the ...
- 121
0
votes
1
answer
959
views
Optional text qualifier in CSV file as input to SSIS package
I receive several CSV files each quarter to process through an SSIS package, and the formatting is inconsistent. This quarter, I encountered what I term optional text qualifiers--text qualifer of ...
- 278
2
votes
3
answers
474
views
Mapping different XML and CSV feeds
Not sure if this is the right venue to be asking this but here goes.
A little background.
I'm trying to build an ecommerce app that would allow sellers from other venues--like, amazon and newegg--to ...
- 129
0
votes
1
answer
40
views
standard format to describe opening schedules
I am working on a project in which I have to model opening schedules (eg a swimming pool is opened from 10:30am to 5pm on Tuesday from january to May. I try to fit this data in a csv and I started ...
- 8,030
0
votes
1
answer
289
views
Improve processing of data (and structure) from flat file to CMS Database
I have a flat file (CSV) that brings me the data I want, but it needs to be cleaned and processed before it can be used, so I built an intermediate Database (MySQL) in order to get the structure from ...
1
vote
0
answers
1k
views
how to use Sql query to search text in CSV file and avoid iteration
I have one A File and many input files ,All files have CSV format . I want to match text of one column of all input files with one of the column of A file .
I have stored all column value of File A in ...
- 119
0
votes
2
answers
701
views
Large csv's to html report
I am working on a web front end + front end services.
I receive good sized csv files (10k lines). My service processes them and condenses them into one larger csv file (up to 300k lines).
This ...
- 1,874
0
votes
1
answer
2k
views
How to POST CSV with XMLHttpRequest
I would like to send a CSV file via POST in a XMLHttpRequest, but I am unsure of two things. First is there anything to distinguish a CSV file from a string split up by comma's? And what sort of ...
- 399
22
votes
6
answers
38k
views
Is CSV a good alternative to XML and JSON? [closed]
Is CSV considered a good option against XML and JSON for programming languages?
I generally use XML and JSON (or sometimes a plain text file) as flat file storage. However, recently I came across an ...
- 1,891
5
votes
4
answers
14k
views
Best way of validating Class properties
Background:
I have a CSV file, which I need to ready and validate each element in each row and create a collection of a class, having valid data.
i.e
CSV File looks like:
EmpID,FirstName,...
- 153
1
vote
3
answers
2k
views
Why do CSV file formats normally use quoting instead of escaping?
There are a lot of variants of the CSV "standard" (or lack thereof). I've never personally see any that use an escape character (like \) instead of surrounding each field with double quotes. Instead ...
- 560
0
votes
1
answer
1k
views
Sync csv file using nodejs
There is a remote csv file that gets updated every second or so.
I need to download it(on a Windows machine) ONCE and always sync local file with the remote one.
Obviously, downloading the whole ...
- 103
28
votes
5
answers
52k
views
Can the csv format be defined by a regex?
A colleague and I have recently argued over whether a pure regex is capable of fully encapsulating the csv format, such that it is capable of parsing all files with any given escape char, quote char, ...
- 3,576
8
votes
4
answers
9k
views
CSV file generation using Java
We have a requirement in our project to generate a big CSV file every 2 hours using a Java program.
This file will have around 60,000 lines (around 120 characters per line). I am not sure about the ...
- 2,657
14
votes
5
answers
13k
views
unit tests for a csv parser
What tests should I use to unit test a CSV parser?
I have a simple CSV parser in C#, and I want to be sure that I have good unit test coverage of all the common (and uncommon) edge cases. What tests ...
- 1,945