0

I have an IoT device that logs data every 3 minutes and uploads that data to Azure Table Storage every 10 minutes. This means the table entities have the built in Timestamp property and a date property. Sadly, date is stored as a string and not as an actual datetime. Timestamp is automatically generated as the time Azure got the data, but date is the timestamp for when the device recorded the data. This means that several entities can have the same timestamp, but different dates (actual data):

Timestamp                       date                    tagName tagValue
2025-04-03T18:20:19.2764867Z    2025-04-03T18:19:55Z    LEVEL   52.2        
2025-04-03T18:20:19.2754824Z    2025-04-03T18:11:14Z    LEVEL   52.2        
2025-04-03T18:20:19.2754824Z    2025-04-03T18:15:35Z    LEVEL   52.2
2025-04-03T18:10:20.8985209Z    2025-04-03T18:02:54Z    LEVEL   52.2        
2025-04-03T18:00:19.3550008Z    2025-04-03T17:58:33Z    LEVEL   52.2        
2025-04-03T18:00:19.3550008Z    2025-04-03T17:54:13Z    LEVEL   52.2        
2025-04-03T17:50:20.126223Z     2025-04-03T17:45:52Z    LEVEL   52.2        
2025-04-03T17:50:20.126223Z     2025-04-03T17:49:53Z    LEVEL   52.2

My goal is to query the table for the most recent level reading using the REST API. My plan to accomplish this is to get all entities uploaded in the last hour then sort through them to find the most recent date property. I've tried two approaches:

This URL returns data 90% of the time:

https://mystorage.table.core.windows.net/HistoryValues?sv=2019-02-02&st=2025-04-02T15%3A33%3A02Z&se=2030-04-03T15%3A33%3A00Z&sp=r&sig=<mysig>&tn=HistoryValues&$filter=Timestamp%20ge%20datetime%272025-04-02T20:33:02Z%27%20and%20tagName%20eq%20%27LEVEL%27&$select=tagValue,date

This one never returns data:

https://mystorage.table.core.windows.net/HistoryValues?sv=2019-02-02&st=2025-04-02T15%3A33%3A02Z&se=2030-04-03T15%3A33%3A00Z&sp=r&sig=<mysig>&tn=HistoryValues&$filter=date%20ge%20%272025-04-02T20:33:02Z%27%20and%20tagName%20eq%20%27TOWER_LEVEL%27&$select=tagValue,date 

Both return HTTP status 200, but only the first query returns entities. The second does not return any entities. If put the filter date ge '2025-04-02T20:33:02Z' and tagName eq 'TOWER_LEVEL' in Azure Storage Explorer, it currently returns 312 entities.

The simple solution seems to be to filter one the Timestamp. I ran the function that queries by timestamp every 3 minutes overnight, and every 30 minutes no entities are returned. At 5:03 -> data, 5:06 -> data, 5:09 -> data, etc. until 5:30 when 0 entities were returned. At 5:33 I got data. This continued: 5:36 -> data, 5:39 -> data, ..., 5:57 -> data, but at 6:00 zero entities were returned again. But 6:03, 6:06, 6:09, etc all work as expected until 6:30 when I got no data. This pattern held all night long. No data was ever returned at the top and bottom of the hour. This is why I thought I would try to filter by date, but this is not working at all.

Why would the Timestamp query fail every 30 minutes even though there is clearly data that matches the query? Why does my date filter work in Storage Explorer but not in the REST query? Any advice on how to achieve the goal of getting the most recent level reading when date is stored as a string?

7
  • You can only filter by the Timestamp field in the query, not the date field. So, you should filter by Timestamp to get the data and then use the date field in your application to find the most recent reading. Commented Apr 4 at 5:50
  • How you are sending logs from azure IoT/IoT device to azure storage Commented Apr 4 at 6:07
  • @Sampath The device send data via MQTT to an Azure IoT Hub, then the IoT Hub routes to various queues based on Event Grid Subscription. Queue triggered functions handle the messages to store the MQTT data in table storage. Changing all that is infeasible. Commented Apr 4 at 13:09
  • @Sampath tests show that querying by date does work. $filter=date%20ge%20%272025-04-03T11:57:24Z%27 returned the same data in the browser as it did in Storage Explorer. Two things I find interesting: : does not have to be url encoded (both ways work) and the table does not need () (both mystorage.table.core.windows.net/HistoryValues and mystorage.table.core.windows.net/HistoryValues() work) despite the documentation saying otherwise. Commented Apr 4 at 13:19
  • 1
    Use the Timestamp field to fetch entities from the past 65 minutes to avoid issues during write times like :00 and :30. Filter by tagName = 'LEVEL' in the REST query. Filter the results by the 'date' field is a string type on the client side by converting it to Date objects. Commented Apr 7 at 8:36

2 Answers 2

1

Azure Table Storage REST API

I understood that Azure Table Storage not always returning data, particularly when filtering by the date field or querying at :00 and :30. I was able to get the latest LEVEL reading based on the device’s actual reading time (date), even it is stored as a string.

Used the Timestamp field to get the data from the last 65 minutes, so we will not miss any data during times when writes are still being processed .Filtered only the rows where tagName is LEVEL.

Queried Azure Table Storage using REST API

require("dotenv").config();
const fetch = require("node-fetch");
const account = process.env.STORAGE_ACCOUNT;
const table = process.env.TABLE_NAME;
const tagName = process.env.TAG_NAME || "LEVEL";
const sasToken = process.env.SAS_TOKEN;
const storageUrl = `https://${account}.table.core.windows.net/${table}`;
function getPastTime(minutesAgo) {
  const date = new Date(Date.now() - minutesAgo * 60 * 1000);
  return date.toISOString();
}
async function getLatestReading() {
  try {
    const timestampFrom = getPastTime(65); 
    const encodedFilter = encodeURIComponent(
      `Timestamp ge datetime'${timestampFrom}' and tagName eq '${tagName}'`
    );
    const encodedSelect = encodeURIComponent("tagValue,date,tagName");
    const url = `${storageUrl}()?$filter=${encodedFilter}&$select=${encodedSelect}&${sasToken}`;
    console.log("Requesting URL:\n", url, "\n");
    const response = await fetch(url, {
      method: "GET",
      headers: {
        Accept: "application/json;odata=nometadata",
      },
    });
    if (!response.ok) {
      throw new Error(`HTTP ${response.status} - ${response.statusText}`);
    }
    const data = await response.json();
    const sorted = data.value
      .filter((item) => item.date)
      .sort(
        (a, b) =>
          new Date(b.date.replace("Z", "+00:00")) -
          new Date(a.date.replace("Z", "+00:00"))
      );
    const latest = sorted[0];
    if (latest) {
      console.log("Latest LEVEL reading:");
      console.log(latest);
    } else {
      console.log("No LEVEL readings found in the past 65 minutes.");
    }
  } catch (err) {
    console.error("Error:", err.message);
  }
}
getLatestReading();

Output:

enter image description here

enter image description here

enter image description here

Refer: Querying tables and entities

Sign up to request clarification or add additional context in comments.

3 Comments

I'll have to try looking back 65 minutes. Since I'm looking back 60 minutes and there should be data uploaded every 10 minutes, there should be at least 5 bunches of data for a total of at least 16 readings. IDK why 65 would work any better than 60, but I've been at this game long enough to see things that shouldn't make a difference be the thing that makes or breaks it..
@Tim Using 65 minutes instead of 60 gives a buffer for any slight delays in data processing or storage. Since date is just a string, we use Timestamp to fetch data and sort by date afterward to get the latest reading.
@Tim Thank you for letting us know you found an alternate solution. We request you to post it as an answer and mark it as accepted. Also, thank you for the positive note below the answer.
0

I eventually achieved my goal by taking a different approach. Rather than generating a query string to pull the last 60 or 65 minutes of data and processing it to find the most current value, I created an Azure function app and table storage to store the most current value of the sensor reading.

The table, SensorDataExport, is set up so that the device's ID is the PartitionKey and the desired sensor reading is the RowKey. It has one additional property named sensorValue. The function app loads all the entities from SensorDataExport and loads the last 65 minutes of data from the devices historical logs table. It then finds the most current value and writes it back to the SensorDataExport table. I didn't try loading only 60 minutes of data, so maybe 65 minutes is a magic number. My function app runs every minute, but I think I could adjust that back to every 3 minutes. As a bonus, since PartitionKey and RowKey are indexed properties, the query string returns data in 1-2 seconds vs the 4-6 seconds it took before. All of this worked flawlessly for the 48 hour test I did.

Sample entity from the table:

PartitionKey  RowKey  Timestamp                     tagValue
1234567       LEVEL   2025-04-11T14:31:30.1186383Z  53.2

And the Azure Function App:

using Microsoft.Azure.Functions.Worker;
using Azure.Data.Tables;
using Azure;

namespace UpdateSensorDataExport
{
    public class UpdateSensorDataExport
    {
        [Function("UpdateSensorDataExport")]
        public async Task Run([TimerTrigger("0 * * * * *", RunOnStartup = true)] TimerInfo myTimer)
        {
            var serviceClient = new TableServiceClient(Environment.GetEnvironmentVariable("AzureWebJobsStorage"));
            var globalTable = serviceClient.GetTableClient("SensorDataExport");

            await foreach (var entity in globalTable.QueryAsync<TableEntity>())
            {
                string device = entity.PartitionKey;
                string sensorName = entity.RowKey;

                string historyTableName = $"{device}HistoryValues";

                var historyTable = serviceClient.GetTableClient(historyTableName);

                // Get latest history value using string 'date', limited to entities from last 65 minutes
                var cutoff = DateTimeOffset.UtcNow.AddMinutes(-65);
                var historyEntities = historyTable
                    .Query<TableEntity>(e => e.GetString("tagName") == sensorName && e.Timestamp >= cutoff)
                    .Where(e => e.ContainsKey("date"))
                    .ToList();
                var latestHistory = historyEntities
                    .OrderByDescending(e => DateTime.Parse(e.GetString("date")))
                    .FirstOrDefault();

                if (latestHistory != null)
                {
                    string newTagValue = GetSensorValueAsString(latestHistory);

                    // Update GlobalTagExport
                    entity["tagValue"] = newTagValue;
                    Console.WriteLine(newTagValue);
                    await globalTable.UpdateEntityAsync(entity, ETag.All, TableUpdateMode.Replace);
                }
            }
        }
        private static string GetSensorValueAsString(TableEntity entity)
        {
            var value = entity["tagValue"];
            return value?.ToString();
        }
    }
}

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.