নেস্টেড অভিধান এবং তালিকাগুলিতে একটি কী এর সমস্ত উপস্থিতি সন্ধান করুন

Question

আমার কাছে এর মতো একটি অভিধান রয়েছে:

{ "id" : "abcde",
  "key1" : "blah",
  "key2" : "blah blah",
  "nestedlist" : [ 
    { "id" : "qwerty",
      "nestednestedlist" : [ 
        { "id" : "xyz",
          "keyA" : "blah blah blah" },
        { "id" : "fghi",
          "keyZ" : "blah blah blah" }],
      "anothernestednestedlist" : [ 
        { "id" : "asdf",
          "keyQ" : "blah blah" },
        { "id" : "yuiop",
          "keyW" : "blah" }] } ] } 

Basically a dictionary with nested lists, dictionaries, and strings, of arbitrary depth.

What is the best way of traversing this to extract the values of every "id" key? I want to achieve the equivalent of an XPath query like "//id". The value of "id" is always a string.

So from my example, the output I need is basically:

["abcde", "qwerty", "xyz", "fghi", "asdf", "yuiop"]

Order is not important.

Comments

See also: stackoverflow.com/questions/7681301/… stackoverflow.com/a/16508328/42223

— dreftymac

Most of your solutions blow up if we pass None as input. Do you care about robustness? (since this is now being used as canonical question)

— smci

Answer 1

I found this Q/A very interesting, since it provides several different solutions for the same problem. I took all these functions and tested them with a complex dictionary object. I had to take two functions out of the test, because they had to many fail results and they did not support returning lists or dicts as values, which i find essential, since a function should be prepared for almost any data to come.

So i pumped the other functions in 100.000 iterations through the timeit module and output came to following result:

0.11 usec/pass on gen_dict_extract(k,o)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
6.03 usec/pass on find_all_items(k,o)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
0.15 usec/pass on findkeys(k,o)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1.79 usec/pass on get_recursively(k,o)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
0.14 usec/pass on find(k,o)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
0.36 usec/pass on dict_extract(k,o)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All functions had the same needle to search for ('logging') and the same dictionary object, which is constructed like this:

o = { 'temparature': '50', 
      'logging': {
        'handlers': {
          'console': {
            'formatter': 'simple', 
            'class': 'logging.StreamHandler', 
            'stream': 'ext://sys.stdout', 
            'level': 'DEBUG'
          }
        },
        'loggers': {
          'simpleExample': {
            'handlers': ['console'], 
            'propagate': 'no', 
            'level': 'INFO'
          },
         'root': {
           'handlers': ['console'], 
           'level': 'DEBUG'
         }
       }, 
       'version': '1', 
       'formatters': {
         'simple': {
           'datefmt': "'%Y-%m-%d %H:%M:%S'", 
           'format': '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
         }
       }
     }, 
     'treatment': {'second': 5, 'last': 4, 'first': 4},   
     'treatment_plan': [[4, 5, 4], [4, 5, 4], [5, 5, 5]]
}

All functions delivered the same result, but the time differences are dramatic! The function gen_dict_extract(k,o) is my function adapted from the functions here, actually it is pretty much like the find function from Alfe, with the main difference, that i am checking if the given object has iteritems function, in case strings are passed during recursion:

def gen_dict_extract(key, var):
    if hasattr(var,'iteritems'):
        for k, v in var.iteritems():
            if k == key:
                yield v
            if isinstance(v, dict):
                for result in gen_dict_extract(key, v):
                    yield result
            elif isinstance(v, list):
                for d in v:
                    for result in gen_dict_extract(key, d):
                        yield result

So this variant is the fastest and safest of the functions here. And find_all_items is incredibly slow and far off the second slowest get_recursivley while the rest, except dict_extract, is close to each other. The functions fun and keyHole only work if you are looking for strings.

Interesting learning aspect here :)

Comments

If you want to search for multiple keys as I did, just: (1) change to gen_dict_extract(keys, var) (2) put for key in keys: as line 2 & indent the rest (3) change the first yield to yield {key: v}

— Bruno Bronosky

You're comparing apples to oranges. Running a function that returns a generator takes less time than running a function that returns a finished result. Try timeit on next(functionname(k, o) for all the generator solutions.

— kaleissin

hasattr(var, 'items') for python3

— gobrewers14

Did you consider to strip the if hasattr part for a version using try to catch the exception in case the call fails (see pastebin.com/ZXvVtV0g for a possible implementation)? That would reduce the doubled lookup of the attribute iteritems (once for hasattr() and once for the call) and thus probably reduce the runtime (which seems important to you). Didn't make any benchmarks, though.

— Alfe

For anyone visiting this page now that Python 3 has taken over, remember that iteritems has become items.

— Mike Williamson

Answer 2

d = { "id" : "abcde",
    "key1" : "blah",
    "key2" : "blah blah",
    "nestedlist" : [ 
    { "id" : "qwerty",
        "nestednestedlist" : [ 
        { "id" : "xyz", "keyA" : "blah blah blah" },
        { "id" : "fghi", "keyZ" : "blah blah blah" }],
        "anothernestednestedlist" : [ 
        { "id" : "asdf", "keyQ" : "blah blah" },
        { "id" : "yuiop", "keyW" : "blah" }] } ] } 


def fun(d):
    if 'id' in d:
        yield d['id']
    for k in d:
        if isinstance(d[k], list):
            for i in d[k]:
                for j in fun(i):
                    yield j

---

>>> list(fun(d))
['abcde', 'qwerty', 'xyz', 'fghi', 'asdf', 'yuiop']

Comments

The only thing I would change is for k in d to for k,value in d.items() with the subsequent use of value instead of d[k].

— ovgolovin

Thanks, this works great. Required very slight modification because my lists can contain strings as well as dicts (which I didn't mention), but otherwise perfect.

— Matt Swain

This fits a very narrow case, you owe it to yourself to consider the answer from "hexerei software" called gen_dict_extract

— Bruno Bronosky

I got the error "TypeError: argument of type 'NoneType' is not iterable"

— xiaoshir

This solution doesn't seem to support lists

— Alex R

Answer 3

d = { "id" : "abcde",
    "key1" : "blah",
    "key2" : "blah blah",
    "nestedlist" : [
    { "id" : "qwerty",
        "nestednestedlist" : [
        { "id" : "xyz", "keyA" : "blah blah blah" },
        { "id" : "fghi", "keyZ" : "blah blah blah" }],
        "anothernestednestedlist" : [
        { "id" : "asdf", "keyQ" : "blah blah" },
        { "id" : "yuiop", "keyW" : "blah" }] } ] }


def findkeys(node, kv):
    if isinstance(node, list):
        for i in node:
            for x in findkeys(i, kv):
               yield x
    elif isinstance(node, dict):
        if kv in node:
            yield node[kv]
        for j in node.values():
            for x in findkeys(j, kv):
                yield x

print(list(findkeys(d, 'id')))

Comments

This example worked with every complex dictionary I tested. Well done.

This should be the accepted answer, it can find keys that are within dictionaries that are nested within list of lists etc.

— Anthon

This works in Python3 as well, as long as the print statement at the end is modified. None of the solutions above this worked for an API response with lists nested inside dicts listed inside lists, etc, but this one worked beautifully.

— Andy Forceno

Answer 4

def find(key, value):
  for k, v in value.iteritems():
    if k == key:
      yield v
    elif isinstance(v, dict):
      for result in find(key, v):
        yield result
    elif isinstance(v, list):
      for d in v:
        for result in find(key, d):
          yield result

EDIT: @Anthon noticed that this will not work for directly nested lists. If you have this in your input, you can use this:

def find(key, value):
  for k, v in (value.iteritems() if isinstance(value, dict) else
               enumerate(value) if isinstance(value, list) else []):
    if k == key:
      yield v
    elif isinstance(v, (dict, list)):
      for result in find(key, v):
        yield result

But I think the original version is easier to understand, so I will leave it.

Comments

This works great as well, but likewise runs into issues if it encounters a list that directly contains a string (which I forgot to include in my example). I think adding in an isinstance check for a dict before the last two lines solves this.

— Matt Swain

Thanks for the accolades, but I'd be prouder to get them for the cleanliness of my code than for its speed.

— Alfe

95% of the time, yes. The remaining (rare) occasions are the ones in which some time limitation might force me to choose a faster version over a cleaner one. But I don't like this. It always means to put a load of work onto my successor who will have to maintain that code. It is a risk because my successor might get confused. I will have to write a lot of comments then, maybe a whole document explaining my motivations, timing experiments, their results etc. That's way more work for me and all colleagues to get it done properly. Cleaner is way simpler.

— Alfe

@Alfe - thanks for this answer. I had a need to extract all occurences of a string in a nested dict for a specific use case of Elasticsearch and this code was useful with a minor modification - stackoverflow.com/questions/40586020/…

— Saurabh Hirani

This completely breaks on lists directly contained in lists.

— Anthon

Answer 5

Another variation, which includes the nested path to the found results (note: this version doesn't consider lists):

def find_all_items(obj, key, keys=None):
    """
    Example of use:
    d = {'a': 1, 'b': 2, 'c': {'a': 3, 'd': 4, 'e': {'a': 9, 'b': 3}, 'j': {'c': 4}}}
    for k, v in find_all_items(d, 'a'):
        print "* {} = {} *".format('->'.join(k), v)    
    """
    ret = []
    if not keys:
        keys = []
    if key in obj:
        out_keys = keys + [key]
        ret.append((out_keys, obj[key]))
    for k, v in obj.items():
        if isinstance(v, dict):
            found_items = find_all_items(v, key, keys=(keys+[k]))
            ret += found_items
    return ret

Answer 6

I just wanted to iterate on @hexerei-software's excellent answer using yield from and accepting top-level lists.

def gen_dict_extract(var, key):
    if isinstance(var, dict):
        for k, v in var.items():
            if k == key:
                yield v
            if isinstance(v, (dict, list)):
                yield from gen_dict_extract(v, key)
    elif isinstance(var, list):
        for d in var:
            yield from gen_dict_extract(d, key)

Comments

Excellent mod to @hexerei-software's answer: succinct and allows list-of-dicts! I'm using this along with @bruno-bronosky's suggestions in his comments to use for key in keys. Also I added to the 2nd isinstance to (list, tuple) for even more variety. ;)

— Cometsong

Answer 7

This function recursively searches a dictionary containing nested dictionaries and lists. It builds a list called fields_found, which contains the value for every time the field is found. The 'field' is the key I'm looking for in the dictionary and its nested lists and dictionaries.

def get_recursively(search_dict, field):
    """Takes a dict with nested lists and dicts,
    and searches all dicts for a key of the field
    provided.
    """
    fields_found = []

    for key, value in search_dict.iteritems():

        if key == field:
            fields_found.append(value)

        elif isinstance(value, dict):
            results = get_recursively(value, field)
            for result in results:
                fields_found.append(result)

        elif isinstance(value, list):
            for item in value:
                if isinstance(item, dict):
                    more_results = get_recursively(item, field)
                    for another_result in more_results:
                        fields_found.append(another_result)

    return fields_found

Comments

You could use fields_found.extend(more_results) instead of running another loop. Would look a bit cleaner in my opinion.

— sapit

Answer 8

Here is my stab at it:

def keyHole(k2b,o):
  # print "Checking for %s in "%k2b,o
  if isinstance(o, dict):
    for k, v in o.iteritems():
      if k == k2b and not hasattr(v, '__iter__'): yield v
      else:
        for r in  keyHole(k2b,v): yield r
  elif hasattr(o, '__iter__'):
    for r in [ keyHole(k2b,i) for i in o ]:
      for r2 in r: yield r2
  return

Ex.:

>>> findMe = {'Me':{'a':2,'Me':'bop'},'z':{'Me':4}}
>>> keyHole('Me',findMe)
<generator object keyHole at 0x105eccb90>
>>> [ x for x in keyHole('Me',findMe) ]
['bop', 4]

Answer 9

Following up on @hexerei software's answer and @bruno-bronosky's comment, if you want to iterate over a list/set of keys:

def gen_dict_extract(var, keys):
   for key in keys:
      if hasattr(var, 'items'):
         for k, v in var.items():
            if k == key:
               yield v
            if isinstance(v, dict):
               for result in gen_dict_extract([key], v):
                  yield result
            elif isinstance(v, list):
               for d in v:
                  for result in gen_dict_extract([key], d):
                     yield result    

Note that I'm passing a list with a single element ([key]}, instead of the string key.

Answer 10

pip install nested-lookup does exactly what you are looking for:

document = [ { 'taco' : 42 } , { 'salsa' : [ { 'burrito' : { 'taco' : 69 } } ] } ]

>>> print(nested_lookup('taco', document))
[42, 69]