并发编程-进程

进程创建启动

python提供了multiprocessing模块来支持多进程

• multiprocessing.Process(target=task, args=(arg,))用于创建进程

Process类相关方法

• start() 启动进程
• join() 等待进程结束

启动子线程

【注意】线程启动代码块要放在__name__ == __main__下

方式一：

import multiprocessing
import timedef task(name: str, count: int):print(f"{name} - step 1\n", end="")result = 0for i in range(count):result += i + 1time.sleep(5)print(f"{name}- end with {result}")def start_process1():name = "p1"count = 100process = multiprocessing.Process(target=task, args=(name, count))process.start()process.join()print("main process over")if __name__ == "__main__":start_process1()

运行结果：

PS F:\Code\python_code\high_python\python_advance\multiprocesses> python .\multiprocesses_demo.py
p1 - step 1
p1- end with 5050
main process over

查看进程信息：一条进程是python用来做资源监控的，所以有两条。

PS F:\Code\python_code\high_python> tasklist.exe|findstr "python"
python.exe                   19588 Console                    2     11,436 K
python.exe                   14300 Console                    2     11,644 K

方式二：

import multiprocessing
import timedef task(name: str, count: int):print(f"{name} - step 1\n", end="")result = 0for i in range(count):result += i + 1time.sleep(5)print(f"{name}- end with {result}")def statr_process2():args_list = [("a", 100), ("b", 99), ("c", 98)]processes = [multiprocessing.Process(target=task, args=(name, count)) for name, count in args_list]for p in processes:p.start()for p in processes:p.join()# 创建进程
if __name__ == "__main__":statr_process2()

运行结果：

PS F:\Code\python_code\high_python\python_advance\multiprocesses> python .\multiprocesses_demo.py
a - step 1
b - step 1
c - step 1
a- end with 5050
b- end with 4950
c- end with 4851

检查进程信息：

PS F:\Code\python_code\high_python> tasklist.exe|findstr "python"
python.exe                   15016 Console                    2     11,576 K
python.exe                   13820 Console                    2     11,520 K
python.exe                    3676 Console                    2     11,632 K
python.exe                    3080 Console                    2     11,624 K

进程池

• 进程的创建和销毁比起线程更加耗费资源
• 频繁的创建和销毁线程对系统的性能影响更大
• 进程池是python提供的便于进程管理和提高性能的工具

使用进程池

python提供了两个类concurrent.futures.ProcessPoolExecutor和Future

ProcessPoolExecutor相关方法

• submit() # 启动一个线程任务，返回一个Future类的对象
• map() # 相同任务，不同参数，启动多进程，返回可迭代对象，包含任务执行结果
• shutdown() # 停止进程

Future相关方法

• result() # 任务执行结果
• exception() # 执行出现的异常

下载图片案例

from concurrent.futures import ProcessPoolExecutor
from urllib.request import urlopen, Request
import osheaders = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36","content-type": "Content-Type: image/jpeg"
}
def download_img(url: str):site_url = Request(url, headers=headers)with urlopen(site_url) as web_file:img_data = web_file.read()if not img_data:raise Exception(f"Error: can not load the image from {url}")file_name = os.path.basename(url)with open(file_name, "wb") as f:f.write(img_data)return "Download image successfully, {}".format(url)urls = ["https://img0.bdstatic.com/static/searchresult/img/baseimg3_4f26a23.png",# "..."
]def main():with ProcessPoolExecutor() as ececutor:results = ececutor.map(download_img, urls)for r in results:print(r)
if __name__ == "__main__":main()

运行结果：

PS F:\Code\python_code\high_python\python_advance\multiprocesses> python .\multiprocess_pool.py
Download image successfully, https://img0.bdstatic.com/static/searchresult/img/baseimg3_4f26a23.png