[Documentation] Case Study: Use executorlib for task handling in a larger application

[jan-janssen]

excutorlib can be integrated in existing frameworks and in combination with the flux framework is can provide an efficient task scaling solution, using the available resources while not oversubscribing them. Given the background of executorlib these tasks can be distributed on a high performance computing (HPC) cluster:

• Explain the use of external cache key in the resource dictionary: resource_dict={"cache_key": "foo"}. This enables external applications to define the key to store a given calculation in.
• Implement a check status function based on from executorlib import get_future_from_file as implemented in [Feature] Implement get_future_from_file() #915 . The external application uses its cache key to check the status of a given job object.
• Terminate a task which is no longer required using the from executorlib import terminate_task_in_cache function as implemented in [Feature] Implement terminate_task_in_cache() #914 . The external application again uses its cache key to remove tasks from the backlog.

[jan-janssen]

@liamhuber I plan to include these features in 1.9.0. Basically with get_future_from_file() and terminate_task_in_cache() external libraries like pyiron_workflow can get the results for a previously submitted future object based on the cache_key or alternatively delete the corresponding task from the queuing system in case it is no longer required. Does this simplify your implementation in pyiron_workflow?

[liamhuber]

It's been a while since I looked at how the pyiron_workflow interface is implemented, but it definitely sounds like it would make things easier 🚀