[SOLVED] Parsing json blob with Python

flukeyLinux · 07-01-2022, 08:34 AM

Hello,
I'm having trouble finding much on this on the web. Any help would be massively appreciated.

I have the following json objects:

Code:

{
    "uuid": "gslbservice",
    "name": "toplevelurl.com",
    "groups": [
        {
            "name": "albpool1",
            "members": [
                {
                    "fqdn": "alb1.com",
                    "enabled": true
                }
            ],
        },
        {
            "name": "albpool2",
            "members": [
                {
                    "fqdn": "alb2.com",
                    "enabled": true
                }
            ],
        }
    ]
}

What I am trying to do is loop through all "members" objects (there could be 2,3 or 4) and updated the "enabled" key to "True" if for example the fqdn = "alb2.com."

I can do this with the first object with:

Code:

          route = "enable"
                fqdn = "alb1"
                for member in element['groups']:
                    if args.fqdn in member['members'][0]['fqdn']:
                        if args.route == "enable":
                            parsed['enabled'] = True

I think my problem is this line:

Code:

if args.fqdn in member['members'][0]['fqdn']

-- explicitly the [0].

is there a way to loop through all objects?

thanks in advance

pan64 · 07-01-2022, 09:03 AM

yes, you replace that 0 with a loop, like:

Code:

for member in element['groups']:
    for data in member['members']:
        if args.fqdn in data['fqdn']:
            ....

flukeyLinux · 07-01-2022, 09:37 AM

Quote:

Originally Posted by pan64

yes, you replace that 0 with a loop, like:

Code:

for member in element['groups']:
    for data in member['members']:
        if args.fqdn in data['fqdn']:
            ....

Lovely, so simple

thank you.

do you happen to know how to update a json key that is nested?

Code:

parsed['enabled'] = True

works.
But it is added at the wrong level.

Code:

parsed(member['enabled']) = True

Does not work.

this is my last hurdle

pan64 · 07-01-2022, 09:41 AM

Quote:

Originally Posted by flukeyLinux

Lovely, so simple

thank you.

do you happen to know how to update a json key that is nested?

Code:

parsed['enabled'] = True

works.
But it is added at the wrong level.

Code:

parsed(member['enabled']) = True

Does not work.

this is my last hurdle

this is the same as here:

Code:

member['members'][0]['fqdn'] = new_value

(I have no idea what is that parsed

flukeyLinux · 07-01-2022, 09:54 AM

Quote:

Originally Posted by pan64

this is the same as here:

Code:

member['members'][0]['fqdn'] = new_value

(I have no idea what is that parsed

parsed is basically the json data:

Code:

parsed = json.loads(json_data)

This does not work:

Code:

member['members'][0]['fqdn'] = new_value

parsed['members'][0]['enabled'] = False
KeyError: 'members'

messed about with a few variations of that with no joy.

boughtonp · 07-01-2022, 09:59 AM

Quote:

Originally Posted by flukeyLinux

parsed['members'][0]['enabled'] = False
KeyError: 'members'

The error message is clear - there is a key error with 'members' - i.e. there is no key with that name.

This is equally evident from merely reading the JSON snippet in post #1.

sundialsvcs · 07-01-2022, 10:03 AM

Just grab the JSON, decode it, and print the resulting structure.

The correct way to handle JSON is always to first decode it, then manipulate the resulting data structure, then encode it. Treat the JSON conversion process as a "black box." Python, like every other language these days, uses "known good" software to do this. Do not attempt to manipulate the JSON data textually.

flukeyLinux · 07-01-2022, 10:29 AM

Quote:

Originally Posted by boughtonp

The error message is clear - there is a key error with 'members' - i.e. there is no key with that name.

This is equally evident from merely reading the JSON snippet in post #1.

As you can "clearly" see (just to be as condescending as you

) from my first comment. the json blob contains object = "members".

Hence why I am asking the question...

flukeyLinux · 07-01-2022, 10:30 AM

Quote:

Originally Posted by sundialsvcs

Just grab the JSON, decode it, and print the resulting structure.

The correct way to handle JSON is always to first decode it, then manipulate the resulting data structure, then encode it. Treat the JSON conversion process as a "black box." Python, like every other language these days, uses "known good" software to do this. Do not attempt to manipulate the JSON data textually.

This information is good. I am new enough to python and have not come across decoding the json first. I must look that up. I am more of a bash head and this would be very easy there. But I need to use python as its a lambda.

boughtonp · 07-01-2022, 10:45 AM

Quote:

Originally Posted by flukeyLinux

...the json blob contains "members".

Hence why I am asking the question...

You need to back-up and learn basic JSON before going any further.

flukeyLinux · 07-01-2022, 10:54 AM

Quote:

Originally Posted by boughtonp

You need to back-up and learn basic JSON before going any further.

Good stuff. thanks lad. I had thought this forum was all part of it. good to know though....

dugan · 07-01-2022, 11:10 AM

Well, first, JSON in your code sample isn't valid. You can see that by pasting it into https://jsonlint.com/ ...

Also, I'm not sure if you know how to iterate through a dictionary. Looping through it in a "for in" loop will get you its keys. Usually you want to call its .keys() (remember, explicit is better than implicit), .values() or .items() method and loop through that, depending on whether you need its keys, values, or both.

ondoho · 07-02-2022, 02:35 AM

As a side remark I want to add that in Linux-speak "blob" means an opaque piece of data (a file), something you cannot (easily) read, e.g. a binary closed source blob, some firmware etc.

JSON is not that.
It is a simple text format, you can look at it in a text editor etc.

There's no need to "decode" it, just format it nicely.
What python does, it loads it into a structure that makes it easier to work with.
There are other (command line) tools that can do similar things, jq comes to mind.

ntubski · 07-02-2022, 09:07 AM

Quote:

Originally Posted by flukeyLinux

What I am trying to do is loop through all "members" objects (there could be 2,3 or 4) and updated the "enabled" key to "True" if for example the fqdn = "alb2.com."

This is slightly confusing because it's already "true" in your example input?

Quote:

Originally Posted by flukeyLinux

parsed is basically the json data:

Code:

parsed = json.loads(json_data)

Assuming element is somehow a subpart of parsed, I'm guessing you want

Code:

for member in element['groups']:
    for data in member['members']:
        if args.fqdn in data['fqdn']: # should this be "=" instead of "in"?
            data['enabled'] = True

Quote:

Originally Posted by flukeyLinux

I am new enough to python and have not come across decoding the json first.

Your parsed = json.loads(json_data) step is decoding.

Quote:

Originally Posted by ondoho

As a side remark I want to add that in Linux-speak "blob" means an opaque piece of data (a file)[...]
There's no need to "decode" it, just format it nicely.

I think sundialscvs is just using non-standard terminology, as usual (in this case, "decode" = "parse", "blob" = "unparsed data").

Quote:

Originally Posted by dugan

Also, I'm not sure if you know how to iterate through a dictionary.

They're iterating over the arrays, not the dictionaries, I think? (I'm assuming that the JSON array gets parsed into a Python array.)

sundialsvcs · 07-02-2022, 10:17 AM

@ondoho: What I am specifically saying is that you should treat the JSON data "as a blob." Don't try to "format it" – it doesn't need to be pretty – and don't try to write it yourself. Every programming language has encode/decode support for JSON (and other formats), and uses a standard set of binaries to do it. In this way you know that you can do what needs be done – to interchange structured data reliably with another client. You're handed a "blob" and the means to turn it into a data structure; then, to turn a data structure back into another "blob." And, if you do it this way, you can be confident that it will work correctly.