python: passing parameters through a functions tree

masavini · 03-09-2022, 05:01 AM

Hi,
I have a functions tree in which a requests session should be opened at the top of the tree, but the session itself should be used by the functions in the bottom of the tree.

please consider this simple example:

Code:

def scrape(sites_list):
    for site in sites_list:
        with requests.Session() as req_session:
            funcA(site, req_session)

def funcA(site, req_session):
    funcB(req_session)

def funcB(req_session):
    req_session.get('site.com')

To let funcB() use req_session should I pass it as a parameter through the whole functions tree as show above? Or is there a smarter way to do it?

pan64 · 03-09-2022, 06:07 AM

depends on your code (if it is OOP or not). You can use global variable, a class member (for example).

masavini · 03-09-2022, 09:08 AM

i've not used OOP so far.

i thought i could use something like this:

Code:

$ cat main.py
import sys

def main():
    for site in ['site1', 'site2']:
        site_module = importlib.import_module(site)
        site_scraper = getattr(site_module, f'{site}_scraper')
        site_scraper()
        site_session = getattr(site_module, f'{site}_session')
        site_session.close()

$ cat site1.py
import requests

site1_session = requests.Session()


def site1_scraper():
    post_scraper()
    comment_scraper()


def post_scraper():
    response = site1_session.get(post_url)


def comment_scraper():
    response = site1_session.get(comment_url)

for what i understand, the with construct would automatically close each session in case of error.
could i get something similar with a try/except construct inside main()?

dugan · 03-09-2022, 02:42 PM

I'd just pass it up.

pan64 · 03-10-2022, 12:06 AM

I don't really understand the question. And also I don't understand why don't you try your idea. Would be better to try to implement what you want and discuss the practical details.

masavini · 03-11-2022, 02:01 AM

Quote:

Originally Posted by pan64

I don't really understand the question. And also I don't understand why don't you try your idea. Would be better to try to implement what you want and discuss the practical details.

of course i tried my idea and it works, but i'm a total newbie and:
1) i don't want to reinvent the wheel
2) even if it seems to works, it doesn't mean it's the right way

pan64 · 03-11-2022, 02:30 AM

Quote:

Originally Posted by masavini

of course i tried my idea and it works, but i'm a total newbie and:
1) i don't want to reinvent the wheel

Actually I don't know which wheel is it

Quote:

Originally Posted by masavini

2) even if it seems to works, it doesn't mean it's the right way

there is no "right way", there are several different ways available to implement anything. So your solution can be acceptable especially because you understand how does that work.
From the other hand you can show us that code and we can help you to improve it.

masavini · 03-11-2022, 03:03 AM

Quote:

Originally Posted by pan64

Actually I don't know which wheel is it

well, i don't know either. but as i wrote i'm a newbie and i just know there are A LOT of wheels: i didn't want to miss an important one.

Quote:

Originally Posted by pan64

there is no "right way", there are several different ways available to implement anything. So your solution can be acceptable especially because you understand how does that work.
From the other hand you can show us that code and we can help you to improve it.

thanks, i usually post code samples to avoid bothering people with not relevant (frequently badly written) code...

boughtonp · 03-11-2022, 08:16 AM

Quote:

Originally Posted by masavini

of course i tried my idea and it works, but i'm a total newbie and:
1) i don't want to reinvent the wheel
2) even if it seems to works, it doesn't mean it's the right way

Both good attitudes to have, but as Pan says there isn't a single right answer, sometimes you have to make a choice and it'll either be the right one or you'll learn something.

Have a read of this article: https://en.wikipedia.org/wiki/Comparison_of_programming_paradigms

Different approaches can work better is different situations, but sometimes they work equally well and it doesn't matter which one you use - what's more important is consistency, correctness, and readability of the code.