Add LocalEventManager

[vdusek]

Ticket

• Implement LocalEventManager #28

Description

• Rewrote of LocalEventManager from https://github.com/apify/crawlee/blob/v3.7.3/packages/core/src/events/local_event_manager.ts.
  • Basically, it just schedules a periodic task for retrieving the CPU and memory-related data.
• I implemented the helper class RecurringTask as an alternative to betterSetInterval from @apify/utilities.
• CPU and memory data are retrieved using the psutil library.
• I put the default threshold values into Config, as it is done in TS implementation since they are probably going to be used in multiple places.

Test script

import asyncio
import logging
from datetime import timedelta

from crawlee import Config
from crawlee.events import LocalEventManager
from crawlee.events.types import Event, EventSystemInfoData
from crawlee.log_config import CrawleeLogFormatter

# Configuration of the root logger
handler = logging.StreamHandler()
handler.setFormatter(CrawleeLogFormatter(include_logger_name=True))
root_logger = logging.getLogger()
root_logger.setLevel(logging.INFO)
root_logger.addHandler(handler)

logger = logging.getLogger(__name__)


def system_info_handler(data: EventSystemInfoData) -> None:
    logger.info(f'data={data}')


async def main() -> None:
    logger.info('Starting the main...')
    config = Config(system_info_interval=timedelta(seconds=0.5))

    event_manager = LocalEventManager(config=config)
    await event_manager.init()
    event_manager.on(event=Event.SYSTEM_INFO, listener=system_info_handler)

    # Alocating some memory...
    await asyncio.sleep(1)
    a = list(range(1_000_000))
    await asyncio.sleep(1)
    b = list(range(1_000_000))
    await asyncio.sleep(1)
    c = list(range(1_000_000))

    logger.info('Waiting for all listeners to complete...')
    await event_manager.close(timeout=timedelta(seconds=1))
    logger.info('Done waiting for all listeners to complete.')


if __name__ == '__main__':
    asyncio.run(main())

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[janbuchar]

1. an explicit interval sounds like a good idea for cpu_percent
2. this won't include CPU usage of e.g. playwright-controlled browsers, right? Shouldn't we check child processes as well, like in the memory usage method?

[vdusek]

1 - sure thing, makes sense

2 - If I understand the doc of psutil correctly ... The psutil.cpu_percent function measures the system-wide CPU utilization, not just the CPU usage by your specific process or its child processes. So there should be no need to check child processes. I'd say in this case, it's what we want.

[janbuchar]

Oh, and could we write some tests?

[vdusek]

Oh, and could we write some tests?

Of course, as discussed previously, I'll prepare tests once the core components stabilize.

The recurring task, event manager, and local event manager can be considered in a solid state, I'll provide tests for them in the following PRs.

I will provide tests for the system status module later since I suppose there could be some changes when finishing the Snapshotter.

[vdusek]

Since Honza is sick, I am merging this, so that I can work on subsequent things. In case of further comments, I'll make another contribution.