138 questions
Advice
1
vote
1
replies
38
views
Why do B-tree disk optimizations work when the OS controls physical disk layout?
I understand the standard explanation for why B-trees are used in databases: they minimize disk seeks by packing many keys into each node, keeping the tree shallow (3-4 levels), and enabling efficient ...
- 11
0
votes
1
answer
75
views
What does the Linux 0.11 kernel do while the disk prepares data after hd_out?
I’m exploring the Linux 0.11 source code, specifically how it reads data from the disk into memory. I’ve been analyzing the hd_out function and the associated do_hd_request in hd.c.
Here’s the ...
- 3
0
votes
1
answer
143
views
Delphi IOresult misbehaving
I have an application that saves data to an xml file but when data cannot be interpreted it writes a message to a textfile. I am using Delphi 11, Windows 10, 64bit compilation
The procedure to write ...
0
votes
0
answers
82
views
How to get write access to device?
When I write to a disk device or a drive, sometimes I get an access denied error:
HANDLE hd=CreateFile("\\\\.\\PhysicalDrive1",GENERIC_WRITE,FILE_SHARE_READ|FILE_SHARE_WRITE, NULL,...
- 978
1
vote
1
answer
630
views
Excessive disk I/O when using VSCode remote SSH for Next.js development
I'm encountering an issue where Node.js processes are consuming excessive disk I/O when I'm using Visual Studio Code (VSCode) with Remote SSH to develop Next.js applications on a cloud server. This ...
- 11
1
vote
0
answers
80
views
Reading diskIO returns strange devicenames
Greetings Stack Overflow community!
I'm currently working on a Go script to fetch disk I/O operations using my API. However, I'm encountering a peculiar issue where the script returns empty values. In ...
- 410
0
votes
0
answers
3k
views
How to get a disk IO time rate from prometheus
I have a server running standard Prometheus node exporter. Metrics are properly collected.
Now I need to display in Grafana rate of time spend for IO operations on all devices.
In the /metrics for my ...
- 596
1
vote
0
answers
519
views
Per-process I/O stats in Linux - TASKSTATS_CMD_ATTR_TGID not working
Linux provides /proc/[pid]/io interface to get I/O stats of a process. But in addition to the accumulated I/O of live threads, read_bytes and write_bytes fields also include the I/O of dead threads ...
- 707
0
votes
1
answer
288
views
What is measured by changing the `file-block-size` parameter in sysbench fileio test?
I was trying to measure the system performance with sysbench fileio test. However, I'm not sure what am I playing with when I change that file-block-size parameter.
Previously I thought it was the ...
- 32
-1
votes
1
answer
95
views
Why CPU reset when no `cli`
I am trying to write a bootloader by myself, but qemu-system-i386 CPU will reset probabilistically. But bochs does not have this problem. Why?
my code is mbr.s and loader.s:
# cat mbr.s
%include "...
- 19
1
vote
1
answer
460
views
How can I do an unbuffered disk read in Python?
I need to read a sector from the physical disk, but without using the system cache.
I tried this:
import os
disk_path = "/dev/sdc"
try:
disk_fd = os.open(disk_path, os.O_RDONLY | os....
- 3,182
1
vote
0
answers
113
views
How DBMS accesses disk pages when OS only allows files access?
The SQL databases generally use B-trees to store index (and sometimes data as well) in the disk. The B-tree is a tree of disk pages. To create B-tree, the DBMS would require access to pages on the ...
- 311
0
votes
1
answer
255
views
Kafka disk access while reading and writing
According to Kafka doc https://kafka.apache.org/documentation/#persistence, Kafka utilities squential read/write performace of spnning disks and hence have high throughput. But how can this high ...
- 3,812
1
vote
0
answers
2k
views
Postgresql - random_page_cost =1.1 and NVMe disk: slower query
My SysAdmin colleague told me that the Postgres'hosts use NVME disk.
How can I check that with Linux command?
Why does the planner/optimizer seem to get it wrong when I set random_page_cost =1.1
To ...
- 25
2
votes
0
answers
681
views
why direct io much slower than non-direct io on SSD(performance measured after system cache cleared)?
I'm new to optimizing disk IO performance. I compared the performance of reading from file with or without direct IO enabled. The chunk size is 512KiB. As Direct IO reads data from disk directly to ...
- 323
1
vote
2
answers
1k
views
What is the most efficient way to copy many files programmatically?
Once upon a time long ago, we had a bash script that works out a list of files that need to be copied based on some criteria (basically like a filtered version of cp -rf).
This was too slow and was ...
- 5,819
-1
votes
2
answers
302
views
unexpected I/O improvement with openmp
I have 5 vectors with different sizes in a vector (levelFreqs) and I have written a sequential code and a parallel code to store these vectors on a single disk (no parallel file system). What I see is ...
- 63
1
vote
0
answers
878
views
testing sequential disk write performance with fio and iostat
I am trying to make sense of sequential disk write performance on a spinning hard disk. I am using direct and sync io to bypass the page cache. For small block size (4KB) fio reports an iops of ~11. ...
- 642
0
votes
0
answers
176
views
Read data with h5py slow due to `make_fid`
As a part of my data processing pipeline I'm reading many hdf files on a network drive, potentially away from the physical machine. After profiling (using cProfile) my code which does basically the ...
- 13.2k
-1
votes
1
answer
145
views
Trobleshooting slow writes on hpux
Could anyone offer any troubleshooting ideas or pointers on where/how to get more information on the difference between sys and real time from the output below?
It is my understanding that the ...
- 124
1
vote
1
answer
452
views
High disk I/O (read) on Cassandra nodes
We have 3 nodes Cassandra cluster.
We have an application that uses a keyspace that creates a hightload on disks, on read. The problem has a cumulative effect. The more days we interact with the ...
- 11
0
votes
0
answers
748
views
What is the difference between Disk IO Utilisation and Disk IO Saturation in Grafana?
Our product uses the grafana tool to monitor a multi node, cpu, executor application. I don't know much as I am new in this field. At a particular moment I noticed that the "Disk IO Saturation&...
- 141
0
votes
1
answer
95
views
Why can't a disk block store multiple index tree nodes
Recently, I was studying the implementation of index tree in database, and learned that using B+tree can store as many keys as possible in a disk block, so that the search process can read as few ...
- 55
-3
votes
1
answer
64
views
Fizzbuzz in C accessing disk
I have made a simple fizzbuzz program in C, just for practicing some aspects of the C language. I am still learning it, I am coming from higher level languages.
While running the program itself, with ...
2
votes
0
answers
57
views
Does a SQL Server Update statement for nvarchar overwrite the same address on disk if the new value is the same size? (Microsoft SQL Server) [closed]
Is it possible to overwrite the same disk address on a TSQL update statement? A use case would be preventing recovery of the original text on disk for security or anonymity reasons.
CREATE TABLE Test(
...
- 2,643
1
vote
0
answers
147
views
storing a std::bitset to disk takes up too much memory
I have difficulties finding out why my bitset takes up 29MB of memory when writing it to disk. I.e. the file that is written out is 29MB large.
I write out the bitset in the following way:
#include &...
- 315
2
votes
1
answer
841
views
Best way to save a large binary file by chunks asynchronously in C++
I'm developing a C++ application whose output is a single big binary file (a couple of GBs, basically a large sequence of floats). The content of this file is generated asynchronoulsy by parallel ...
- 101
0
votes
1
answer
1k
views
How to get the correct device to limit block IO for docker container?
I need to put a limit on block IO operations speed for a number of docker containers.
To achieve this, I need to do something like:
docker run -it --device-read-bps /dev/sda:1mb ubuntu,according to ...
- 883
1
vote
2
answers
375
views
Can I write a file to a specific cluster location?
You know, when an application opens a file and write to it, the system chooses in which cluster will be stored. I want to choose myself ! Let me tell you what I really want to do... In fact, I don't ...
- 3,182
0
votes
0
answers
876
views
How to simulate low disk write speed in Ubuntu?
I use dd command as follows, to check the disk speed of my ubuntu box:
dd if=/dev/zero of=/tmp/test1.img bs=640M count=1 oflag=dsync
That gives me the actual disk speed. But how do I simulate a case ...
- 622
0
votes
0
answers
206
views
Using multiple files vs multiple buffers. What to use?
I have a program which creates two lists of numbers, sorted and stored on disk. The task is is to merge both the lists using linear merge. They both are sorted and fast to merge. One list contains 1,...
user12320641
0
votes
1
answer
265
views
What could cause MySQL to intermittently fail to return a row?
I have a MySQL database that I am running very simple queries against as part of a webapp. I have received reports from users starting today that they got an error saying that their account doesn't ...
- 156
0
votes
2
answers
2k
views
Google Cloud Platform zonal SSDs higher performance persistence than local SSDs?
I've been running a database application that writes data synchronously to disk, and so, looking for the best disk throughput. I've found that GCP's local SSDs are supposed to provide the best ...
- 138
0
votes
0
answers
412
views
Read data from DVD with maximum speed
I'm trying to make a disk dump program (like dd) with C language on Windows.
The program is almost finished, but there is one problem: reading data from DVD is very slow.
According to the ...
- 113
0
votes
0
answers
136
views
What kinds of i/o error returns when remove disk abnormally?
What kinds of i/o error returns when remove disk abnormally(physically)?
I use linux aio functions to i/o files.
If I remove disk abnormally after "io_submit(control blocks with i/o infos)",
...
- 225
0
votes
1
answer
1k
views
Get usage disk of a Windows process by name in Python
I would like to create a script that wait until a specific process has 0 MB/s of disk usage.
import time
import psutil
def get_process(name):
process = None
for p in psutil.process_iter():
...
- 259
2
votes
0
answers
508
views
How to clear the hard disk cache programmatically?
Is there a method in Windows to clear the whole disk cache? I am doing benchmarks on a disk and try to measure which of my methods has the best performance in C#. The problem is though that the disk ...
- 1,815
0
votes
2
answers
11k
views
Zabbix slow disk request responses
I have installed Zabbix-server 5.0 on a Docker Container, following the official guide (for postgresql): https://www.zabbix.com/documentation/current/manual/installation/containers
After that, I ...
0
votes
0
answers
2k
views
mariadb high Disk IO and IO Wait
We have two mariadb servers Master and Slave each on PR and DR, We are observing high disk io above 90% and iowait some time above 20, when mariadb backup start at night iowait goes above 30 and ...
5
votes
5
answers
2k
views
Fastest way to merge millions of files
There are 81 million files (!) stored in one directory on a remote machine. All files end in ".paintedHaploDiversity". I would like to merge those files into one called allOutputs_3.5 in the ...
- 18.4k
7
votes
0
answers
587
views
Should getExternalCacheDir() be called outside Main Thread?
I am having a call to getExternalCacheDir() function on my Android App, and this is causing an ANR for some of Android 10 users having the following devices (HUAWEI P30 lite, Galaxy A40, Xperia XZ3).
...
- 1,936
1
vote
0
answers
166
views
ATA PIO disk reading reads at wierd offsets
I'm writing a kernel, and have hit an error that I don't understand, and can't fix. I'm developing a filesystem for it, because I wanted to quickly implement pesistent storage. The reads from the disk ...
- 23
1
vote
1
answer
2k
views
Check/Output input data generated by FIO's
I am using FIO tool on linux to run some IO's. I am interested to look at data contents that are generated as part of the FIO command.
My command:
sudo fio --name=randwrite --ioengine=libaio --iodepth=...
- 45
0
votes
2
answers
426
views
Get filename with High I/O read request
My Jenkins server is giving high read I/O of 5-10MB and there are no jenkins job is running on it.
I was able to find the process(jenkins) with command iotop -oP
and with lsof -p 20874 i'm getting ...
- 735
0
votes
1
answer
694
views
What happens when disk space runs out?
Suppose I do:
with open("temp.txt", "w" as f):
while True:
f.write(1)
What shall happen when I come close to completely using up my disk space? It seems like a problem ...
- 1,311
0
votes
1
answer
131
views
c# Disk I\O efficiency improvement read multiple lines at a time
It's common knowledge that disk I\O is expensive.
I typically read one line at a time using C# from a .txt file.
It just crossed my mind that if C# were to give you a way of reading 100 lines a time, ...
0
votes
0
answers
170
views
Is it possible to fix my USB that can't even be read by Disk Management
I'm not sure if this is a duplicate, but every time I find a thread about corrupted USB, disk management is still at least able to recognize a drive. My USB is so bricked that when I try to use disk ...
0
votes
2
answers
2k
views
MySQL heavy disk activity even with no queries running
Trying to troubleshoot an issue with a mysterious disk io bottleneck caused by MySQL.
I'm using the following commands to test disk read/write speed:
#write
dd if=/dev/zero of=/tmp/writetest bs=1M ...
- 69
0
votes
1
answer
749
views
What is the mechanism of snapshotting in Redis?
We can define in Redis configuration file about snapshotting to happen after certain interval of time. I want to clear out that whether that snapshotting process in differential or it creates complete ...
11
votes
1
answer
26k
views
Can anyone explain me the difference between NET I/O vs BLOCK I/O in docker stats?
I am getting the following stats for my redis while running docker stats command.
I want to know about NET I/O and BLOCK I/O. I read few documentation they are saying about BLOCK I/O is something ...
- 173