2

I'm working on an application for a lab project and I'm making it in C#. It's supposed to import results from a text file that is exported from the application we use to run the tests and so far, I've hit a road block.

I've gotten the program to save around 250 decimal values as a single-dimension array but then I'm trying to get the array itself to be able to saved in an SQL database so that I can later retrieve the array and use the decimal values to construct a plot of the points.

I need the entire array to be imported into the database as one single value though because the lab project has several specimens each with their own set of 250 or so Decimal points (which will be stored as arrays, too)

Thanks for your help.

EDIT: Thanks for the quick replies, guys but the problem is that its not just results from a specimen with only 1 test ran. Each specimen itself has the same test performed on them with different decibel levels over 15 times. Each test has its own sets of 250 results and we have many specimens.

Also, the specimens already have a unique ID assigned to them and it'd be stored as a String not an Int. What I'm planning on doing is having a separate table in the DB for each specimen and have each row include info on the decibel level of the test and store the array serialized...

I think this would work because we will NOT need to access individual points in the data straight from the database; I'm just using the database to store the data out of memory since there's so much of it. I'm going to query the database for the array and other info and then use zedgraph to plot the points in the array and compare multiple specimens simultaneously.

4
  • 1
    Regarding the edit, it sounds like you had already made your mind up before you asked the question. Given that the highest voted replies are telling you "don't do this" why did you even ask for advice if you're not going to listen anyway? Commented Aug 7, 2009 at 15:43
  • 1
    Easy. There was a specific question: how do you do X? The "highest voted" answers are all "Don't do X". That's not about ignoring advice - the answers don't answer the question. Commented Aug 8, 2009 at 12:03
  • 2
    this is a good example of why scientists and engineers should not be allowed near databases ;-) Commented Aug 8, 2009 at 17:16
  • @Greg Beech: The problem is that I am a high school student volunteering at the lab and making the program for a project. The method of storing it was basically decided by the scientists prior to me really starting to work on the program and nothing I say can really change their minds... @nader: thanks man =D Commented Aug 9, 2009 at 4:53

7 Answers 7

18

Short answer is absolutely not. These are two completely different data structures. There are work arounds like putting it in a blob or comma separating a text column. But, I really hate those. It doesn't allow you to do math at the SQL Server level.

IMO, the best option includes having more than one column in your table. Add an identifier so you know which array the data point belongs to.

For example:

AutoId    Specimen    Measurement
   1         A            42
   2         A            45.001
   3         B            47.92

Then, to get your results:

select 
    measurement 
from 
    mytable 
where 
    specimen = 'A'
order by
    autoid asc

Edit: You're planning on doing a separate 250 row table for each specimen? That's absolutely overkill. Just use one table, have the specimen identifier as a column (as shown), and index that column. SQL Server can handle millions upon millions of rows markedly well. Databases are really good at that. Why not play to their strengths instead of trying to recreate C# data structures?

Sign up to request clarification or add additional context in comments.

6 Comments

+1 good answer. I took the liberty of adding an "order by" clause to the query because I assume the order of the data points is important in this case, and although it's very likely the autoid would be set as the clustered PK and also very likely that the database will use this to order the results, the ordering isn't guaranteed unless you specify it.
Well, I don't need to do math at the server level or anything; I just need to be able to query the database so the stuff doesn't need to be stored in memory on the computer since there are so many specimens and tests ran on them.
What if the array is not the same in each record? And if it is the same the the problem is in the structure of the table from the beginning.
@backslash: What now? I have absolutely no idea what you're asking. @Unk: This still provides you the cleanest way to store your records. It's human readable, and you can quickly verify results if need be.
I'm not planning on doing a separate 250 row table per specimen, I'm planning on doing a separate table per specimen and the unique ID in each table will then be the ID of the test performed on it since identical tests are performed on all the specimens. The table will end up having 15 rows with (probably) a serialized array for the actual data as a field.
|
3

I need the entire array to be imported into the database as one single value though because the lab project has several specimens each with their own set of 250 or so Decimal points (which will be stored as arrays, too)

So you're trying to pound a nail, should you use an old shoe or a glass bottle?

The answer here isn't "serialize the array into XML and store it in a record". You really want to strive for correct database design, and in your case the simplest design is:

Specimens
---------
specimenID (pk int not null)

SpecimenData
------------
dataID (pk int not null
specimenID (fk int not null, points to Specimens table)
awesomeValue (decimal not null)

Querying for data is very straightforward:

SELECT * FROM SpecimenData where specimenID = @specimenID

1 Comment

The thing is that there's more to the experimentation than just a specimen and a value; there are multiple tests ran on the same specimen with their own results and there are different groups of specimens (but this is included in the Specimen ID)... there are Group Numbers, Record numbers, Aquisition Durations, decibel Levels.. it would just be too complicated for me to have each value stored separately since I need to graph it with zedGraph and everything.
1

As long as you don't to access the the individual values in your queries, you can serialize the array and store it as a blob in the database.

3 Comments

eh: They might not need it now, but give it enough time and they'll want to query on individual measurements. That's how it always goes. Definitely better to put these in their own table.
I completely agree with Joel. If it's a relational database then use it like a relational database.
@Joel: I agree doing it correctly in the DB is the way to go. In this case, the OP may just not know better. But I disagree that its always required, that they will want to query individual values. And if individual queries are required, the data can always be adjusted later. Finally, there may well be constraints on what modifications are allowed in the DB (tho that's unlikely in this case)
1

Presumably you could serialize the decimal array in C# to a byte array, and save that in a binary field on a table. Your table would have two fields: SpecimenID, DecimalArrayBytes

Alternately you could have a many to many type table and not store the array in one piece, having fields: SpecimenID, DecimalValue, and use SQL like

SELECT DecimalValue FROM Table WHERE SpecimenID = X

1 Comment

This is the solution I was going to suggest.
0

You can serialize the array and store it as a single chunk of xml/binary/json. Here is an example of serializing it as xml.

public static string Serialize<T>(T obj)
{
    StringBuilder sb = new StringBuilder();
    DataContractSerializer ser = new DataContractSerializer(typeof(T));
    ser.WriteObject(XmlWriter.Create(sb), obj);
    return sb.ToString();
}

2 Comments

Serialization is what he is looking for, but I don't think it's the best answer to this problem.
Yep, i was just trying to answer his specific question, not trying to rearchitect his solution.
0

You want two tables. One to store an index, the other to store the decimal values. Something like this:

create table arrayKey (
   arrayId int identity(1,1) not null
)

create table arrayValue (
   arrayID int not null,
   sequence int identity(1,1) not null,
   storedDecimal decimal(12,2) not null
)

Insert into arrayKey to get an ID to use. All of the decimal values would get stored into arrayValue using the ID and the decimal value to store. Insert them one at a time.

When you retrieve them, you can group them by arrayID so that they all come out together. If you need to retrieve them in the same order you stored them, sort by sequence.

Comments

0

Although any given example might be impractical, via programming you can engineer any shape of peg into any shape of hole.

You could serialize your data for storage in a varbinary, XML-ize it for storage into a SQL Server XML type, etc.

Pick a route to analyze and carefully consider. You can create custom CLR libraries for SQL as well so the virtual sky is the limit.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.