鏈枃灏嗘繁鍏ヨВ鏋愭繁搴﹀己鍖栧涔犱腑鐨勫叧閿粍浠垛€斺€擜ctor-Critic绠楁硶銆傝繖涓€绠楁硶缁撳悎浜嗙瓥鐣ヨ凯浠ｄ笌浠峰€艰凯浠ｇ殑浼樼偣锛屾槸寮哄寲瀛︿範棰嗗煙涓殑涓€椤归噸瑕侀泦鎴愮瓥鐣ャ€傞€氳繃瀹炶返绀轰緥锛屾垜浠皢灞曠ず濡備綍浣跨敤Pytorch瀹炵幇涓€涓畬鏁寸殑Actor-Critic妯″瀷銆?/p>
鎴戜滑鏉ヤ簡瑙ｄ竴涓婣ctor-Critic绠楁硶鐨勫熀鏈師鐞嗗拰缁勬垚閮ㄥ垎銆傝绠楁硶鍖呭惈涓や釜閮ㄥ垎锛氳鍔ㄨ€咃紙Actor锛夊拰璇勮瀹讹紙Critic锛夈€傝鍔ㄨ€呰礋璐ｉ€夋嫨鍔ㄤ綔锛屾棬鍦ㄦ渶澶у寲绱Н濂栧姳锛涜€岃瘎璁哄鍒欒瘎浼板綋鍓嶇瓥鐣ョ殑濂藉潖锛岀粰鍑哄姩浣滅殑浠峰€硷紝骞堕€氳繃鍙嶉鎸囧琛屽姩鑰呮ā鍨嬬殑鏇存柊銆傞€氳繃杩欑鏂瑰紡锛孉ctor-Critic绠楁硶涓嶆柇浼樺寲绛栫暐浠ユ渶澶у寲绱Н濂栧姳銆?/p>
鎺ヤ笅鏉ワ紝鎴戜滑灏嗚缁嗕粙缁岮ctor鍜岃瘎璁哄鐨勮鑹插拰鐩爣銆傝鍔ㄨ€呰緭鍑哄姩浣滅殑姒傜巼鍒嗗竷鎴栬繛缁姩浣滃€硷紝浠ユ渶澶у寲绱Н濂栧姳涓虹洰鏍囷紱璇勮瀹跺垯杈撳嚭鐘舵€佷环鍊兼垨鍔ㄤ綔浠峰€硷紝浠ヨ瘎浼板綋鍓嶇瓥鐣ョ殑濂藉潖骞朵紭鍖栬鍔ㄨ€呯殑琛屼负銆?/p>
鍦ㄥ叕寮忔帹瀵奸儴鍒嗭紝鎴戜滑灏嗕粙缁嶇瓥鐣ヤ紭鍖栫洰鏍囧嚱鏁般€佽瘎璁哄鎹熷け鍑芥暟鍜孉ctor鎹熷け鍑芥暟鐨勫叧閿叕寮忋€傜瓥鐣ヤ紭鍖栫殑鐩爣鏄渶澶у寲绱Н濂栧姳锛岃瘎璁哄缃戠粶閫氬父閲囩敤鐘舵€佷环鍊煎嚱鏁版垨鍔ㄤ綔浠峰€煎嚱鏁扮殑褰㈠紡锛屽叾鎹熷け鍑芥暟鍙互琛ㄧず涓篢D璇樊鐨勫钩鏂瑰潎鍊笺€傝€孉ctor缃戠粶鐨勭洰鏍囧垯鏄€氳繃鏈€澶у寲绛栫暐鐨勫鏁颁技鐒舵潵浼樺寲绛栫暐鍙傛暟銆傛垜浠繕灏嗕粙缁岮ctor鍜岃瘎璁哄鐨勫叧閿弬鏁版洿鏂板叕寮忋€?/p>
涓轰簡鏇寸洿瑙傚湴灞曠ずActor-Critic绠楁硶鐨勫簲鐢紝鎴戜滑灏嗗熀浜嶱ytorch搴撹繘琛屾搷浣溿€傛垜浠皢灞曠ず濡備綍鏋勯€犵瓥鐣ョ綉缁滐紙PolicyNet锛変笌浠峰€肩綉缁滐紙ValueNet锛夛紝骞跺疄鐜版ā鍨嬬殑鏇存柊鍜岃缁冭繃绋嬨€傛垜浠繕灏嗗埄鐢∣penAI Gym鐨凜artPole-v1鐜锛屾紨绀哄浣曞皢Actor-Critic绠楁硶搴旂敤浜庡疄闄呴棶棰樹腑銆?/p>
鍦ㄨ繖涓ず渚嬩腑锛屾垜浠皢灞曠ず瀛︿範鏇茬嚎浠ュ強姣忓洖鍚堢殑鍥炴姤锛屼互渚挎洿濂藉湴浜嗚В妯″瀷鐨勮缁冭繃绋嬪拰鎬ц兘銆傞€氳繃瀹為檯搴旂敤鍜岀粨鏋滃睍绀猴紝鎴戜滑灏嗗府鍔╄鑰呮洿濂藉湴鐞嗚ВActor-Critic绠楁硶鐨勫師鐞嗗拰瀹炵幇鏂瑰紡锛屽苟浜嗚В鍏跺湪瀹為檯闂涓殑琛ㄧ幇銆?/p>
---
3. 瀹氫箟绛栫暐缃戠粶锛圥olicyNet锛変笌鍐掗櫓涔嬫梾鍚▼
涓轰簡鎺㈢储鏈煡鐨勭幆澧冿紝鎴戜滑闇€瑕佷竴涓瓥鐣ョ綉缁滐紙PolicyNet锛夋潵鎸囧紩鎴戜滑鐨勮剼姝ャ€傝鎴戜滑瀹氫箟杩欎釜缃戠粶鐨勬瀯閫犲拰鍓嶈鏂瑰紡銆?/p>
绛栫暐缃戠粶锛圥olicyNet锛夌殑鏋勫缓涓庡惎绋嬩唬鐮侊細
```python
import torch
from torch import nn
class PolicyNet(nn.Module):
def __init__(self, 杈撳叆缁村害, 闅愯棌灞傜淮搴? 杈撳嚭缁村害):
super(PolicyNet, self).__init__() 缁ф壙nn.Module鐨勫垵濮嬪寲鏂规硶
self.fc1 = nn.Linear(杈撳叆缁村害, 闅愯棌灞傜淮搴? 绗竴灞傚叏杩炴帴缃戠粶锛岃礋璐ｇ壒寰佹彁鍙?/p>
self.fc2 = nn.Linear(闅愯棌灞傜淮搴? 杈撳嚭缁村害) 绗簩灞傚叏杩炴帴缃戠粶锛岃礋璐ｇ瓥鐣ヨ緭鍑烘鐜囧垎甯冮娴?/p>
self.鍚▼ = torch.relu 浣跨敤ReLU婵€娲诲嚱鏁板紑鍚垜浠殑鍐掗櫓涔嬫梾
def forward(self, 鐘舵€?: 瀹氫箟缃戠粶鐨勫墠鍚戜紶鎾繃绋嬶紝鍗崇瓥鐣ュ喅绛栬繃绋?/p>
鐘舵€佺粡杩囨縺娲诲嚱鏁板鐞嗗悗鐨勭壒寰?= self.鍚▼(self.fc1(鐘舵€?) 鐗瑰緛鎻愬彇闃舵锛屽緱鍒版縺娲诲悗鐨勭壒寰佸悜閲?/p>
鍐崇瓥姒傜巼鍒嗗竷 = torch.softmax(self.fc2(鐗瑰緛鍚戦噺), dim=1) 绛栫暐鍐崇瓥闃舵锛岃緭鍑烘瘡涓姩浣滅殑棰勬祴姒傜巼鍒嗗竷
return 鍐崇瓥姒傜巼鍒嗗竷 杩斿洖棰勬祴鐨勬鐜囧垎甯冿紝鎸囧紩鎴戜滑鐨勫喅绛栨柟鍚?/p>
```
鎺ヤ笅鏉ワ紝鎴戜滑灏嗗畾涔変环鍊肩綉缁滐紙ValueNet锛夛紝鐢ㄤ簬璇勪及褰撳墠鐘舵€佺殑浠峰€笺€傜劧鍚庯紝鎴戜滑灏嗚繖涓や釜缃戠粶缁勫悎鎴愪竴涓己澶х殑ActorCritic绫汇€傛渶缁堬紝鎴戜滑灏嗗湪涓€涓狢artPole鐜涓娇鐢ㄨ繖涓狝ctorCritic绫昏繘琛岃缁冨苟鍙鍖栫粨鏋溿€傝鎴戜滑缁х画鎺㈢储杩欎釜鍐掗櫓涔嬫梾鍚э紒馃殌馃幃馃敆鉁ㄥ湪杩欐浠ｇ爜鎵€鏋勫缓鐨勬儏澧冧腑锛屾垜浠繍鐢ㄤ簡涓€绉嶅熀鏈殑Actor-Critic妯″瀷锛屽畠琚簿蹇冪紪缁囧湪CartPole-v1鐜涓€傝繖涓幆澧冨鍚屼竴涓姩鎬佺殑鑸炲彴锛屽皬杞﹀拰鏉嗗瓙鍦ㄥ叾涓笉鏂簰鍔紝鑰屾垜浠殑妯″瀷鍒欏姫鍔涘浼氬浣曟帶鍒跺皬杞︾殑琛屽姩锛屼互淇濇寔鏉嗗瓙鐨勭珫鐩翠笉鍊掋€傝繖涓涔犺繃绋嬩笉浠呮槸涓€椤规妧鏈殑鎸戞垬锛屾洿鏄櫤鑳藉拰绛栫暐鐨勭鎾炪€傛瘡涓€娆＄殑琛屽姩閫夋嫨銆佹瘡涓€娆＄殑濂栧姳鍙嶉閮藉湪鎺ㄥ姩鐫€妯″瀷鍚戝墠杩涙銆傞€氳繃杩欎竴绯诲垪鐨勫皾璇曞拰淇锛屾ā鍨嬮€愭鎺屾彙浜嗕竴绉嶆妧鑳斤細鍦–artPole鐜涓紝濡備綍浠ユ渶浼樼殑鏂瑰紡琛屽姩浠ヨ幏鍙栨渶澶х殑绱Н濂栧姳銆傝€屾垜浠墍缁樺埗鐨勬洸绾匡紝姝ｆ槸妯″瀷瀛︿範杩囩▼涓殑鈥滄垚缁╁崟鈥濓紝瀹冩竻鏅板湴灞曠ず浜嗛殢鐫€璁粌鐨勮繘琛岋紝妯″瀷鎵€鑾峰緱鐨勭疮绉鍔辨槸濡備綍鍙樺寲鐨勩€傛瘡涓€娆＄殑杩唬锛岄兘鏄竴娆℃柊鐨勫皾璇曞拰绐佺牬锛屾瘡涓€娆＄殑濂栧姳鎻愬崌锛岄兘鏄ā鍨嬭繘姝ョ殑璇佹嵁銆傝繖涓繃绋嬪厖婊′簡鎸戞垬鍜屾儕鍠滐紝灏卞鍚屾垜浠皬鏃跺€欏涔犳柊鎶€鑳戒竴鏍凤紝铏界劧鏇叉姌鍗村厖婊′簡鎴愬氨鎰熴€傝繖灏辨槸浜哄伐鏅鸿兘瀛︿範鐨勯瓍鍔涙墍鍦紝涔熸槸鎴戜滑涓嶆柇杩芥眰杩涙鐨勬簮娉夈€?/p> |
