90 questions
2
votes
1
answer
153
views
How to fix broken German special characters when reading CSV file?
I am trying to load a German CSV file into R on Fedora Linux 42, but the region names containing special characters (like u-umlaut, o-umlaut, eszett) are not rendered correctly.
Here's a simplified ...
- 811
0
votes
1
answer
73
views
Able to read FileInfo, but cannot read contents within the file: urlopen error [Errno 5] Input/output error
I have 2 environments, Staging & Production.
Both using the exact code, but one environment can read the contents within the file, while other cannot.
I can see both FileInfo using 2 codes:
Python:...
- 21
-3
votes
1
answer
86
views
How to read log files in Pandas dataframe having multiple delimiters? [closed]
My log file format is as below :
2016-09-28 04:30:30, Info CBS Loaded Servicing Stack v6.1.7601.23505 with Core: C:\Windows\winsxs\amd64_microsoft-windows-...
- 19
1
vote
4
answers
374
views
Handling bad lines when reading csv files with pd.read_csv()
I am trying to parse some csv files with a few thousand rows. The csv is structured with comma as a delimiter and quotation marks enclosing each field. I want to use pd.read_csv() to parse the data ...
0
votes
2
answers
122
views
UnicodeDecodeError codec can't decode error using pandas read_csv
I'm opening a csv file using pandas.
import pandas as pd
df = pd.read_csv('/file/planned.csv')
I'm opening a file that contains about 2,000 records collected from all over the places in the world. ...
- 85
0
votes
1
answer
46
views
Pandas text file import: automate selection of 1 table when many exist within each file
Using Pandas 2.2.1 within Python 3.12, I'm importing data from text files that have a varying number of rows of metadata and multiple tables that precede the actual data to be imported. The number of ...
- 83
1
vote
2
answers
58
views
Reading large multi-part table from file and combing its parts into one tibble
I have this huge file (> 25 Mb) of the following structure:
ITEM: TIMESTEP
0
ITEM: NUMBER OF ATOMS
14748
ITEM: BOX BOUNDS ss ss ss
-1.3314357502021994e+02 1.1517122459132779e+02
-1....
- 25
0
votes
0
answers
31
views
How to save JMeter Response Body to the same CSV File used for the request
I am using a csv file for a POST request in JMeter.
sample POST body data
sample list of names on CSV file
The API will return a response body and I want to save the data from the response body into ...
- 1
1
vote
1
answer
38
views
customize header pandas to html
I created a web application to search content in CSV files.for now i only able to put all content into 1 column with comma as delimiter ,
I want to display the data as shown in the image.
My python ...
- 79
0
votes
2
answers
103
views
How to best use read_csv parameters when headers are on different rows, and then make 1st column datetime index
Ive been having trouble reading and updating a csv from yfinance, due to the data in the first few rows of the downloaded csv:
1st row contains the column headers I want (also header - 'Price' - I ...
- 167
-2
votes
1
answer
80
views
pd.read_csv: error when "," comma is present in string column
I am reading in a csv file locally.The sep for the data is "|", so I specify pd.read_csv(repo, sep="|"). The data loads.
However, when I examine the data, any time there is a comma ...
anon
0
votes
0
answers
58
views
Read CSV - There should be a unique row for each daily entry combining health metrics and supplement usage
Can you help me with the interpretation of this sentence ?
Does this mean you should group by date rows with different supplement usage?
Or should I have 2 rows, one for magnesium and one for ...
- 3
1
vote
3
answers
129
views
Randomly shard large CSV into smaller CSVs
I am aiming to shuffle a csv file too large to load into the RAM of my device at once. My plan was to use csv.DictReader to go through the entire file and write each line randomly into one of a set of ...
- 13
1
vote
1
answer
94
views
Panda read_csv, ignore line that contain specific string
I've a dataframe that list datalogger name and there password. The password is generated inside my script if the datalogger have a blank in the password field. And if there is not a generic password, ...
- 305
1
vote
1
answer
46
views
Read csv without filling up empty values
I have the following csv I want to read into Python (Spyder) and count the amount of blank values in column 2:
column 1
column 2
A
N/A
B
N/A
C
N/A
D
E
N/A
F
N/A
G
H
N/A
In this case, there are two ...
- 863
0
votes
1
answer
93
views
Change Datatype of CSV Import with Python/Pandas
I have a CSV file where I cannot change the object data types to INT64, STR64, BOOL, FLOAT64, etc.
How can one solve this?
The data import from the CSV file looks like:
filepath = 'filename.csv'
df = ...
- 1
0
votes
1
answer
132
views
pandas' skiprows speed/efficiency [closed]
I've got huge csv files and a few thousands of files (each file running into Gbs and some running into Mbs). However, my interest is only the last n rows (say 50 records) of each of these files. My ...
- 39
2
votes
1
answer
568
views
Contradictory error when using Polars read_csv() with multiple files for csv.gz
I'm trying to read multiple csv.gz files into a dataframe but it's not working as I expect.
When I use this globbing pattern:
pl.read_csv('folder_1\*.csv.gz')
It returns this error:
ComputeError: ...
- 75
0
votes
1
answer
24
views
URL not loading in pd.read_csv however the file is read openly from browser
I am trying to get a csv using a link "https://nsearchives.nseindia.com/content/equities/eq_band_changes_16072024.csv" but it is not downloading.
`import pandas as pd
...
- 15
1
vote
1
answer
475
views
<Missing> Values in Cell Arrays Created from Matlab readcell()
I am uploading data from an Excel file which was provided in a specific format, for which a minimum reproducible example is shown below:
I am trying to save each column into an array, using the ...
- 141
0
votes
1
answer
861
views
Jupyter Notebooks read CSV - using JupyterLite from under 'Try Jupyter' page
I am using Jupyter Notebooks under "https://jupyter.org/try" to read my csv file. I tried the following methods, and none of them works. I installed and imported pandas as pd.
top3 = pd....
- 1
0
votes
3
answers
125
views
Reading CSV Files And Stripping Values
I am currently new with coding in Python with CSV files, and need help with the following code:
import csv
import random
# Initialize an empty list to store the CSV variables
CSVars = []
# Read the ...
- 13
0
votes
3
answers
194
views
Pandas read csv simultaneously passing usecols and names args
When reading a CSV file as a pandas dataframe, an error is raised when trying to select a subset of columns based on original column names (usecols=) and renaming the selected columns (names=). ...
- 1,314
0
votes
0
answers
22
views
FileNotFoundError: [Errno 2] No such file or directory: '/content/gdrive/MyDrive/RMDS/CDA MEI 2022/ke12 Statistics Inference 2/datahouse_pricing.csv'
I have a problem to read my csv file from google colab, i stored my csv file at gdrive and copy that path as i use to read it and the csv file is found by list dir but it can't be open
enter image ...
0
votes
1
answer
272
views
Pandas data frame read_CSV: why automatically make first column data as index
I'm looking at sensor data with Pandas dataframes. The text file I have does not have an index field.
It starts with time data, but when I use this code to load data from a file into a Pandas data ...
0
votes
2
answers
250
views
Ignore trailing commas in CSV file when using CsvHelper?
I need to read a CSV file with a trailing comma at the end of each line:
column1,column2, // header
value1,value2, // values
Note that the CSV file has only two columns. However the trailing comma ...
- 18.5k
0
votes
1
answer
226
views
pandas read_csv silently not parsing separate date & time fields despite provision of date_format
I'm am reading a CSV file AirQualityUCI.csv containing date data [1]. Here are a sample of 10 lines, including the 1st line for field names:
Date;Time;CO(GT);PT08.S1(CO);NMHC(GT);C6H6(GT);PT08.S2(...
- 1,265
0
votes
0
answers
40
views
using Pandas reading a textcolumn longer than 4096 characters from csv
In my csv there is a JSON object, which could be longer than 4096 characters. The pd.read_csv seems to have a problem because i cannot convert this to a valid JSON object to read from.
I used pd....
- 1
1
vote
1
answer
678
views
Pandas read_csv with 'keep_default_na=False' causing change in data type of values. How to prevent this?
I'm trying to read a csv file which has 'NA' value as data. When I use the 'keep_default_na = false' option to keep NA values it's impacting other columns with empty rows.
The Data:
csv pic
colA ...
0
votes
3
answers
154
views
Keep double quotes in R read_csv to read JSON data
I have a csv file imported from a sql database, one of the column is made of JSON data. I used to be able to read_csv the file and then parse the JSON data using fromJSON.
However, now when I read the ...
- 35
0
votes
2
answers
92
views
importing weird string csv as float
U_d = pd.read_csv("Uzud.csv", decimal = ',')
print(U_d)
what comes out:
f in kHz U in mV Amplitude in Vpp entfernung in cm
43.7 52 1 5.0 9.0
...
- 5
0
votes
1
answer
1k
views
Read_csv() in Pandas doesn't recognize one of its arguments: date_format
I have a Jupyter notebook that has worked in the past. Recently however (without upgrading anything) I have run into this problem: read_csv() got an unexpected keyword argument 'date_format'
The cell ...
- 73
1
vote
1
answer
56
views
Pandas read csv where with id in column 1, headers in col 2 and values in col 3? [duplicate]
I have a data in CSV in below format:
"/some/page-1.md","title","My title 1"
"/some/page-1.md","description","My description 1"
"/some/...
- 13
0
votes
2
answers
964
views
Problem when reading a CSV file using Java
I am facing a problem reading my CSV file with Java.
The java code I am using to read the CSV file is:
package collectData;
import java.io.FileReader;
import java.io.IOException;
import java.io....
- 11
0
votes
3
answers
279
views
When reading a csv file through pandas, all columns become a single column
I am unable to print a DataFrame from a CSV file. It was originally in text format, but I converted to csv through the online site. So the file looks like this:
CSV code:
"clientId,timestamp,log&...
- 13
1
vote
5
answers
152
views
I want to read a csv file with pandas, I want to skip rows but I want to keep the original line numbers
I want to keep track of the original line numbers.
I tried using the skiprows parameter of pd.read_csv(). The original line numbers are not preserved so.
If I start reading at row 100, then the first ...
- 69
-1
votes
1
answer
689
views
Using a for loop with two variables in Python [closed]
I want to use two variables in a loop. Currently, I am running into an error using the following code; The filenames generally contain four column (time, power, time, power) with a lot of rows (up to ...
- 143
0
votes
1
answer
458
views
Best way to remove dataframe columns where every value is the same
Let's say I have a DataFrame with multiple columns where the values are all the same e.g. df = pd.DataFrame({'State':['Texas','Texas','Texas','Texas'], 'County':['Harris','Harris','Harris','Harris'], '...
1
vote
2
answers
99
views
Pandas read_csv() with multiple delimiters not working
I have a csv file I'm trying to read as a pandas dataframe. I need to skip the first 19 rows of comments. I have headers in the 20th row and data in subsequent rows. Only issue is the header row ...
- 11
-1
votes
1
answer
215
views
MIME type for reading CSV file in an Android app does not work
maybe I can get help here. I'm working on an Android app in Java and try to pick a .csv file to import some data:
public void onButtonImportClick(View view) {
Log.d(TAG, "...
- 41
4
votes
1
answer
2k
views
FutureWarning: Support for nested sequences for 'parse_dates' in pd.read_csv is deprecated. How to combine date and time columns with pd.to_datetime?
Here is an example of my .csv file:
date, time, value
20240112,085917,11
20240112,085917,22
I used to import it to DataFrame with the following way:
df = pd.read_csv(csv_file, parse_dates=[['date', '...
- 621
-1
votes
1
answer
413
views
Type error in read_csv() into the keyword
Pandas in the read_csv() in type error shown as:
TypeError Traceback (most recent call last)
Cell In[30], line 1
----> 1 pd.read_csv('Datasets/BX-Books.csv', sep=';'...
0
votes
0
answers
12
views
filepath doesn't have clarity whether it's only the path or file with path
https://pandas.pydata.org/docs/user_guide/io.html#io-read-csv-table
I see you frequently say FILEPATH_OR_BUFFER with no explanation that the path includes the file name not in the variable name ...
1
vote
1
answer
38
views
Pandas read_csv with column behind value
I want to read a csv file with following structure:
1 file, 13 insertions, 23 deletions
1 file, 43 deletions
2 file, 7 insertions
4 file, 13 insertions, 45 deletions
I would like to get a Dataframe, ...
0
votes
1
answer
78
views
decimal parameter using pandas.read_csv
I have a CSV with 19 columns. Many of the values are decimals formatted like '56,2' [german/eureopean format]. No big issue, I thought using the 'decimal=' parameter:
self.imported_csv = pd.read_csv(...
3
votes
3
answers
771
views
Python pandas different separators for header and data
I wrote a nice script that parses into a pandas dataframe some monolitic output for easy plotting and manipulation.
I can read different sections of the lines and turn them into a bigger dataframe ...
- 33
-2
votes
1
answer
55
views
read_csv2 is multiplying some integers by 1000, affecting only the sum() function, how do I correct it?
I'm reading data from an excel spreadsheet with read_csv2 (the data is using ; as a separator).
There's only two columns, let's call them product and count. count is always an integer or NA.
My ...
1
vote
1
answer
142
views
How to pin point the problem read_delim is having?
Just to preface, I'm very fresh with R, and sorry for the "special characters".
I'm currently tryign to read this CSV file I'm working with. Here is my code
X17_01_24_Rawdata_SSB_fish_2021 &...
1
vote
3
answers
83
views
can not separate csv file with thousands and comma [duplicate]
I need to read csv file with comma and also string and numbers but number contains comma in it like 1,260. Also csv file is seperated by comma so i can not read file in right way. How could i ...
- 47
-1
votes
1
answer
63
views
applying read_csv to a Series of filenames only loads the first dataframe
When I run this block of code:
import pandas as pd
import os
working_dir = os.getcwd()+'/'
files = pd.Series(os.listdir(working_dir))
input_files = files[files.str.contains('.csv')]
input_files = ...
- 470