8

I have around 20000 rows in my azure table . I wanted to query all the rows in the azure table . But due to certain azure limitation i am getting only 1000 rows.

My code

from azure.storage import TableService
table_service = TableService(account_name='xxx', account_key='YYY')
i=0
tasks=table_service.query_entities('ValidOutputTable',"PartitionKey eq 'tasksSeattle'")
for task in tasks:
    i+=1
    print task.RowKey,task.DomainUrl,task.Status    
print i

I want to get all the rows from the validoutputtable .Is there a way to do so

Improve this question

3 Answers 3

Reset to default
8

But due to certain azure limitation i am getting only 1000 rows.

This is a documented limitation. Each query request to Azure Table will return no more than 1000 rows. If there are more than 1000 entities, table service will return a continuation token that must be used to fetch next set of entities (See Remarks section here: http://msdn.microsoft.com/en-us/library/azure/dd179421.aspx)

Please see the sample code to fetch all entities from a table:

from azure import *
from azure.storage import TableService

table_service = TableService(account_name='xxx', account_key='yyy')
i=0
next_pk = None
next_rk = None
while True:
    entities=table_service.query_entities('Address',"PartitionKey eq 'Address'", next_partition_key = next_pk, next_row_key = next_rk, top=1000)
    i+=1
    for entity in entities:
        print(entity.AddressLine1)
    if hasattr(entities, 'x_ms_continuation'):
        x_ms_continuation = getattr(entities, 'x_ms_continuation')
        next_pk = x_ms_continuation['nextpartitionkey']
        next_rk = x_ms_continuation['nextrowkey']
    else:
        break;
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

7

Azure Table Storage has a new python library in preview release that is available for installation via pip. To install use the following pip command

pip install azure-data-tables

To query all rows for a given table with the newest library, you can use the following code snippet:

from azure.data.tables import TableClient

key = os.environ['TABLES_PRIMARY_STORAGE_ACCOUNT_KEY']
account_name = os.environ['tables_storage_account_name']
endpoint = os.environ['TABLES_STORAGE_ENDPOINT_SUFFIX']
account_url = "{}.table.{}".format(account_name, endpoint)
table_name = "myBigTable"

with TableClient(account_url=account_url, credential=key, table_name=table_name) as table_client:

    try:
        table_client.create_table()
    except:
        pass

    i = 0
    for entity in table_client.list_entities():
        print(entity['value'])
        i += 1
        if i % 100 == 0:
            print(i)

Your outlook would look like this: (modified for brevity, assuming there are 2000 entities)

...
1100
1200
1300
...
Improve this answer

1 Comment

There should be a special award for updating answers to the latest code. Thank you!
6

Update 2019

Just running for loop on the query result (as author of the topic does) - will get all the data from the query.

from azure.cosmosdb.table.tableservice import TableService

table_service = TableService(account_name='accont_name', account_key='key')

#counter to keep track of records
counter=0

# get the rows. Debugger shows the object has only 100 records
rows = table_service.query_entities(table,"PartitionKey eq 'mykey'")

for row in rows:
    if (counter%100 == 0):
        # just to keep output smaller, print every 100 records
        print("Processing {} record".format(counter))
    counter+=1

The output proves that loop goes over a 1000 records

...
Processing 363500 record
Processing 363600 record
...
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.