{"id":1651,"date":"2025-02-03T21:38:00","date_gmt":"2025-02-03T13:38:00","guid":{"rendered":"https:\/\/www.forillusion.com\/?p=1651"},"modified":"2025-02-14T11:39:02","modified_gmt":"2025-02-14T03:39:02","slug":"3-8-mlp","status":"publish","type":"post","link":"https:\/\/www.forillusion.com\/index.php\/3-8-mlp\/","title":{"rendered":"3.8 \u591a\u5c42\u611f\u77e5\u673a"},"content":{"rendered":"\n<p><div class=\"has-toc have-toc\"><\/div><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u9690\u85cf\u5c42<\/h2>\n\n\n\n<p>\u591a\u5c42\u611f\u77e5\u673a\u5728\u5355\u5c42\u795e\u7ecf\u7f51\u7edc\u7684\u57fa\u7840\u4e0a\u5f15\u5165\u4e86\u4e00\u5230\u591a\u4e2a\u9690\u85cf\u5c42\uff08hidden layer\uff09\u3002\u9690\u85cf\u5c42\u4f4d\u4e8e\u8f93\u5165\u5c42\u548c\u8f93\u51fa\u5c42\u4e4b\u95f4\u3002<\/p>\n\n\n\n<p>\u5177\u4f53\u6765\u8bf4\uff0c\u7ed9\u5b9a\u4e00\u4e2a\u5c0f\u6279\u91cf\u6837\u672c$\\boldsymbol{X} \\in \\mathbb{R}^{n \\times d}$\uff0c\u5176\u6279\u91cf\u5927\u5c0f\u4e3a$n$\uff0c\u8f93\u5165\u4e2a\u6570\u4e3a$d$\u3002\u5047\u8bbe\u591a\u5c42\u611f\u77e5\u673a\u53ea\u6709\u4e00\u4e2a\u9690\u85cf\u5c42\uff0c\u5176\u4e2d\u9690\u85cf\u5355\u5143\u4e2a\u6570\u4e3a$h$\u3002\u8bb0\u9690\u85cf\u5c42\u7684\u8f93\u51fa\uff08\u4e5f\u79f0\u4e3a\u9690\u85cf\u5c42\u53d8\u91cf\u6216\u9690\u85cf\u53d8\u91cf\uff09\u4e3a$\\boldsymbol{H}$\uff0c\u6709$\\boldsymbol{H} \\in \\mathbb{R}^{n \\times h}$\u3002\u56e0\u4e3a\u9690\u85cf\u5c42\u548c\u8f93\u51fa\u5c42\u5747\u662f\u5168\u8fde\u63a5\u5c42\uff0c\u53ef\u4ee5\u8bbe\u9690\u85cf\u5c42\u7684\u6743\u91cd\u53c2\u6570\u548c\u504f\u5dee\u53c2\u6570\u5206\u522b\u4e3a$\\boldsymbol{W}_h \\in \\mathbb{R}^{d \\times h}$\u548c $\\boldsymbol{b}_h \\in \\mathbb{R}^{1 \\times h}$\uff0c\u8f93\u51fa\u5c42\u7684\u6743\u91cd\u548c\u504f\u5dee\u53c2\u6570\u5206\u522b\u4e3a$\\boldsymbol{W}_o \\in \\mathbb{R}^{h \\times q}$\u548c$\\boldsymbol{b}_o \\in \\mathbb{R}^{1 \\times q}$\u3002<\/p>\n\n\n\n<p>\u5148\u6765\u770b\u4e00\u79cd\u542b\u5355\u9690\u85cf\u5c42\u7684\u591a\u5c42\u611f\u77e5\u673a\u7684\u8bbe\u8ba1\u3002\u5176\u8f93\u51fa$\\boldsymbol{O} \\in \\mathbb{R}^{n \\times q}$\u7684\u8ba1\u7b97\u4e3a<\/p>\n\n\n\n<p>$$<br>\\begin{aligned}<br>\\boldsymbol{H} &amp;= \\boldsymbol{X} \\boldsymbol{W}_h + \\boldsymbol{b}_h,\\\\<br>\\boldsymbol{O} &amp;= \\boldsymbol{H} \\boldsymbol{W}_o + \\boldsymbol{b}_o,<br>\\end{aligned}<br>$$<\/p>\n\n\n\n<p>\u4e5f\u5c31\u662f\u5c06\u9690\u85cf\u5c42\u7684\u8f93\u51fa\u76f4\u63a5\u4f5c\u4e3a\u8f93\u51fa\u5c42\u7684\u8f93\u5165\u3002\u5982\u679c\u5c06\u4ee5\u4e0a\u4e24\u4e2a\u5f0f\u5b50\u8054\u7acb\u8d77\u6765\uff0c\u53ef\u4ee5\u5f97\u5230<\/p>\n\n\n\n<p>$$<br>\\boldsymbol{O} = (\\boldsymbol{X} \\boldsymbol{W}_h + \\boldsymbol{b}_h)\\boldsymbol{W}_o + \\boldsymbol{b}_o = \\boldsymbol{X} \\boldsymbol{W}_h\\boldsymbol{W}_o + \\boldsymbol{b}_h \\boldsymbol{W}_o + \\boldsymbol{b}_o.<br>$$<\/p>\n\n\n\n<p>\u4ece\u8054\u7acb\u540e\u7684\u5f0f\u5b50\u53ef\u4ee5\u770b\u51fa\uff0c\u867d\u7136\u795e\u7ecf\u7f51\u7edc\u5f15\u5165\u4e86\u9690\u85cf\u5c42\uff0c\u5374\u4f9d\u7136\u7b49\u4ef7\u4e8e\u4e00\u4e2a\u5355\u5c42\u795e\u7ecf\u7f51\u7edc\uff1a\u5176\u4e2d\u8f93\u51fa\u5c42\u6743\u91cd\u53c2\u6570\u4e3a$\\boldsymbol{W}_h\\boldsymbol{W}_o$\uff0c\u504f\u5dee\u53c2\u6570\u4e3a$\\boldsymbol{b}_h \\boldsymbol{W}_o + \\boldsymbol{b}_o$\u3002\u4e0d\u96be\u53d1\u73b0\uff0c\u5373\u4fbf\u518d\u6dfb\u52a0\u66f4\u591a\u7684\u9690\u85cf\u5c42\uff0c\u4ee5\u4e0a\u8bbe\u8ba1\u4f9d\u7136\u53ea\u80fd\u4e0e\u4ec5\u542b\u8f93\u51fa\u5c42\u7684\u5355\u5c42\u795e\u7ecf\u7f51\u7edc\u7b49\u4ef7\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u6fc0\u6d3b\u51fd\u6570<\/h2>\n\n\n\n<p>\u4e0a\u8ff0\u95ee\u9898\u7684\u6839\u6e90\u5728\u4e8e\u5168\u8fde\u63a5\u5c42\u53ea\u662f\u5bf9\u6570\u636e\u505a\u4eff\u5c04\u53d8\u6362\uff08affine transformation\uff09\uff0c\u800c\u591a\u4e2a\u4eff\u5c04\u53d8\u6362\u7684\u53e0\u52a0\u4ecd\u7136\u662f\u4e00\u4e2a\u4eff\u5c04\u53d8\u6362\u3002\u89e3\u51b3\u95ee\u9898\u7684\u4e00\u4e2a\u65b9\u6cd5\u662f\u5f15\u5165\u975e\u7ebf\u6027\u53d8\u6362\uff0c\u4f8b\u5982\u5bf9\u9690\u85cf\u53d8\u91cf\u4f7f\u7528\u6309\u5143\u7d20\u8fd0\u7b97\u7684\u975e\u7ebf\u6027\u51fd\u6570\u8fdb\u884c\u53d8\u6362\uff0c\u7136\u540e\u518d\u4f5c\u4e3a\u4e0b\u4e00\u4e2a\u5168\u8fde\u63a5\u5c42\u7684\u8f93\u5165\u3002\u8fd9\u4e2a\u975e\u7ebf\u6027\u51fd\u6570\u88ab\u79f0\u4e3a\u6fc0\u6d3b\u51fd\u6570\uff08activation function\uff09\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">ReLU\u51fd\u6570<\/h3>\n\n\n\n<p>ReLU\uff08rectified linear unit\uff09\u51fd\u6570\u63d0\u4f9b\u4e86\u4e00\u4e2a\u5f88\u7b80\u5355\u7684\u975e\u7ebf\u6027\u53d8\u6362\u3002\u7ed9\u5b9a\u5143\u7d20$x$\uff0c\u8be5\u51fd\u6570\u5b9a\u4e49\u4e3a<\/p>\n\n\n\n<p>$$<br>\\text{ReLU}(x) = \\max(x, 0).<br>$$<\/p>\n\n\n\n<p>\u53ef\u4ee5\u770b\u51fa\uff0cReLU\u51fd\u6570\u53ea\u4fdd\u7559\u6b63\u6570\u5143\u7d20\uff0c\u5e76\u5c06\u8d1f\u6570\u5143\u7d20\u6e05\u96f6\u3002\u51fd\u6570\u56fe\u50cf\uff1a<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\"   class=\"lazyload\" data-src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_relu.png\" src=\"https:\/\/cdn.forillusion.com\/moezx\/img\/svg\/loader\/trans.ajax-spinner-preloader.svg\" onerror=\"imgError(this)\"  alt=\"\"\/><\/figure >\n<noscript><img decoding=\"async\" src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_relu.png\" alt=\"\"\/><\/figure><\/noscript>\n\n\n\n<p>\u663e\u7136\uff0c\u5f53\u8f93\u5165\u4e3a\u8d1f\u6570\u65f6\uff0cReLU\u51fd\u6570\u7684\u5bfc\u6570\u4e3a0\uff1b\u5f53\u8f93\u5165\u4e3a\u6b63\u6570\u65f6\uff0cReLU\u51fd\u6570\u7684\u5bfc\u6570\u4e3a1\u3002\u5c3d\u7ba1\u8f93\u5165\u4e3a0\u65f6ReLU\u51fd\u6570\u4e0d\u53ef\u5bfc\uff0c\u4f46\u662f\u53ef\u4ee5\u53d6\u6b64\u5904\u7684\u5bfc\u6570\u4e3a0\u3002\u4e0b\u9762\u7ed8\u5236ReLU\u51fd\u6570\u7684\u5bfc\u6570\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\"   class=\"lazyload\" data-src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_relu_grad.png\" src=\"https:\/\/cdn.forillusion.com\/moezx\/img\/svg\/loader\/trans.ajax-spinner-preloader.svg\" onerror=\"imgError(this)\"  alt=\"\"\/><\/figure >\n<noscript><img decoding=\"async\" src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_relu_grad.png\" alt=\"\"\/><\/figure><\/noscript>\n\n\n\n<h3 class=\"wp-block-heading\">sigmoid\u51fd\u6570<\/h3>\n\n\n\n<p>sigmoid\u51fd\u6570\u53ef\u4ee5\u5c06\u5143\u7d20\u7684\u503c\u53d8\u6362\u52300\u548c1\u4e4b\u95f4\uff1a<\/p>\n\n\n\n<p>$$<br>\\text{sigmoid}(x) = \\frac{1}{1 + \\exp(-x)}.<br>$$<\/p>\n\n\n\n<p>sigmoid\u51fd\u6570\u5728\u65e9\u671f\u7684\u795e\u7ecf\u7f51\u7edc\u4e2d\u8f83\u4e3a\u666e\u904d\uff0c\u4f46\u5b83\u76ee\u524d\u9010\u6e10\u88ab\u66f4\u7b80\u5355\u7684ReLU\u51fd\u6570\u53d6\u4ee3\u3002\u4e0b\u9762\u7ed8\u5236\u4e86sigmoid\u51fd\u6570\u3002\u5f53\u8f93\u5165\u63a5\u8fd10\u65f6\uff0csigmoid\u51fd\u6570\u63a5\u8fd1\u7ebf\u6027\u53d8\u6362\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\"   class=\"lazyload\" data-src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_sigmoid.png\" src=\"https:\/\/cdn.forillusion.com\/moezx\/img\/svg\/loader\/trans.ajax-spinner-preloader.svg\" onerror=\"imgError(this)\"  alt=\"\"\/><\/figure >\n<noscript><img decoding=\"async\" src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_sigmoid.png\" alt=\"\"\/><\/figure><\/noscript>\n\n\n\n<p>\u4f9d\u636e\u94fe\u5f0f\u6cd5\u5219\uff0csigmoid\u51fd\u6570\u7684\u5bfc\u6570<\/p>\n\n\n\n<p>$$<br>\\text{sigmoid}'(x) = \\text{sigmoid}(x)\\left(1-\\text{sigmoid}(x)\\right).<br>$$<\/p>\n\n\n\n<p>\u4e0b\u9762\u7ed8\u5236\u4e86sigmoid\u51fd\u6570\u7684\u5bfc\u6570\u3002\u5f53\u8f93\u5165\u4e3a0\u65f6\uff0csigmoid\u51fd\u6570\u7684\u5bfc\u6570\u8fbe\u5230\u6700\u5927\u503c0.25\uff1b\u5f53\u8f93\u5165\u8d8a\u504f\u79bb0\u65f6\uff0csigmoid\u51fd\u6570\u7684\u5bfc\u6570\u8d8a\u63a5\u8fd10\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\"   class=\"lazyload\" data-src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_sigmoid_grad.png\" src=\"https:\/\/cdn.forillusion.com\/moezx\/img\/svg\/loader\/trans.ajax-spinner-preloader.svg\" onerror=\"imgError(this)\"  alt=\"\"\/><\/figure >\n<noscript><img decoding=\"async\" src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_sigmoid_grad.png\" alt=\"\"\/><\/figure><\/noscript>\n\n\n\n<h3 class=\"wp-block-heading\">tanh\u51fd\u6570<\/h3>\n\n\n\n<p>tanh\uff08\u53cc\u66f2\u6b63\u5207\uff09\u51fd\u6570\u53ef\u4ee5\u5c06\u5143\u7d20\u7684\u503c\u53d8\u6362\u5230-1\u548c1\u4e4b\u95f4\uff1a<\/p>\n\n\n\n<p>$$<br>\\text{tanh}(x) = \\frac{1 - \\exp(-2x)}{1 + \\exp(-2x)}.<br>$$<\/p>\n\n\n\n<p>\u5f53\u8f93\u5165\u63a5\u8fd10\u65f6\uff0ctanh\u51fd\u6570\u63a5\u8fd1\u7ebf\u6027\u53d8\u6362\u3002\u867d\u7136\u8be5\u51fd\u6570\u7684\u5f62\u72b6\u548csigmoid\u51fd\u6570\u7684\u5f62\u72b6\u5f88\u50cf\uff0c\u4f46tanh\u51fd\u6570\u5728\u5750\u6807\u7cfb\u7684\u539f\u70b9\u4e0a\u5bf9\u79f0\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\"   class=\"lazyload\" data-src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_tanh.png\" src=\"https:\/\/cdn.forillusion.com\/moezx\/img\/svg\/loader\/trans.ajax-spinner-preloader.svg\" onerror=\"imgError(this)\"  alt=\"\"\/><\/figure >\n<noscript><img decoding=\"async\" src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_tanh.png\" alt=\"\"\/><\/figure><\/noscript>\n\n\n\n<p>\u4f9d\u636e\u94fe\u5f0f\u6cd5\u5219\uff0ctanh\u51fd\u6570\u7684\u5bfc\u6570<\/p>\n\n\n\n<p>$$<br>\\text{tanh}'(x) = 1 - \\text{tanh}^2(x).<br>$$<\/p>\n\n\n\n<p>\u4e0b\u9762\u7ed8\u5236\u4e86tanh\u51fd\u6570\u7684\u5bfc\u6570\u3002\u5f53\u8f93\u5165\u4e3a0\u65f6\uff0ctanh\u51fd\u6570\u7684\u5bfc\u6570\u8fbe\u5230\u6700\u5927\u503c1\uff1b\u5f53\u8f93\u5165\u8d8a\u504f\u79bb0\u65f6\uff0ctanh\u51fd\u6570\u7684\u5bfc\u6570\u8d8a\u63a5\u8fd10\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\"   class=\"lazyload\" data-src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_tanh_grad.png\" src=\"https:\/\/cdn.forillusion.com\/moezx\/img\/svg\/loader\/trans.ajax-spinner-preloader.svg\" onerror=\"imgError(this)\"  alt=\"\"\/><\/figure >\n<noscript><img decoding=\"async\" src=\"https:\/\/cos.forillusion.top\/wp-content\/uploads\/2025\/02\/3.8_tanh_grad.png\" alt=\"\"\/><\/figure><\/noscript>\n\n\n\n<h2 class=\"wp-block-heading\">\u591a\u5c42\u611f\u77e5\u673a<\/h2>\n\n\n\n<p>\u591a\u5c42\u611f\u77e5\u673a\u5c31\u662f\u542b\u6709\u81f3\u5c11\u4e00\u4e2a\u9690\u85cf\u5c42\u7684\u7531\u5168\u8fde\u63a5\u5c42\u7ec4\u6210\u7684\u795e\u7ecf\u7f51\u7edc\uff0c\u4e14\u6bcf\u4e2a\u9690\u85cf\u5c42\u7684\u8f93\u51fa\u901a\u8fc7\u6fc0\u6d3b\u51fd\u6570\u8fdb\u884c\u53d8\u6362\u3002\u591a\u5c42\u611f\u77e5\u673a\u7684\u5c42\u6570\u548c\u5404\u9690\u85cf\u5c42\u4e2d\u9690\u85cf\u5355\u5143\u4e2a\u6570\u90fd\u662f\u8d85\u53c2\u6570\u3002\u4ee5\u5355\u9690\u85cf\u5c42\u4e3a\u4f8b\u5e76\u6cbf\u7528\u672c\u8282\u4e4b\u524d\u5b9a\u4e49\u7684\u7b26\u53f7\uff0c\u591a\u5c42\u611f\u77e5\u673a\u6309\u4ee5\u4e0b\u65b9\u5f0f\u8ba1\u7b97\u8f93\u51fa\uff1a<\/p>\n\n\n\n<p>$$<br>\\begin{aligned}<br>\\boldsymbol{H} &amp;= \\phi(\\boldsymbol{X} \\boldsymbol{W}_h + \\boldsymbol{b}_h),\\\\<br>\\boldsymbol{O} &amp;= \\boldsymbol{H} \\boldsymbol{W}_o + \\boldsymbol{b}_o,<br>\\end{aligned}<br>$$<\/p>\n\n\n\n<p>\u5176\u4e2d$\\phi$\u8868\u793a\u6fc0\u6d3b\u51fd\u6570\u3002\u5728\u5206\u7c7b\u95ee\u9898\u4e2d\uff0c\u6211\u4eec\u53ef\u4ee5\u5bf9\u8f93\u51fa$\\boldsymbol{O}$\u505asoftmax\u8fd0\u7b97\uff0c\u5e76\u4f7f\u7528softmax\u56de\u5f52\u4e2d\u7684\u4ea4\u53c9\u71b5\u635f\u5931\u51fd\u6570\u3002<br>\u5728\u56de\u5f52\u95ee\u9898\u4e2d\uff0c\u6211\u4eec\u5c06\u8f93\u51fa\u5c42\u7684\u8f93\u51fa\u4e2a\u6570\u8bbe\u4e3a1\uff0c\u5e76\u5c06\u8f93\u51fa$\\boldsymbol{O}$\u76f4\u63a5\u63d0\u4f9b\u7ed9\u7ebf\u6027\u56de\u5f52\u4e2d\u4f7f\u7528\u7684\u5e73\u65b9\u635f\u5931\u51fd\u6570\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u9690\u85cf\u5c42 \u591a\u5c42\u611f\u77e5\u673a\u5728\u5355\u5c42\u795e\u7ecf\u7f51\u7edc\u7684\u57fa\u7840\u4e0a\u5f15\u5165\u4e86\u4e00\u5230\u591a\u4e2a\u9690\u85cf\u5c42\uff08hidden layer\uff09\u3002\u9690\u85cf\u5c42\u4f4d\u4e8e\u8f93\u5165\u5c42\u548c\u8f93\u51fa\u5c42\u4e4b\u95f4\u3002 \u5177\u4f53\u6765\u8bf4 &#8230;<\/p>","protected":false},"author":1,"featured_media":1654,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[46,3],"tags":[45,44,12,22],"class_list":["post-1651","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-46","category-3","tag-45","tag-44","tag-12","tag-22"],"_links":{"self":[{"href":"https:\/\/www.forillusion.com\/index.php\/wp-json\/wp\/v2\/posts\/1651","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.forillusion.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.forillusion.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.forillusion.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.forillusion.com\/index.php\/wp-json\/wp\/v2\/comments?post=1651"}],"version-history":[{"count":1,"href":"https:\/\/www.forillusion.com\/index.php\/wp-json\/wp\/v2\/posts\/1651\/revisions"}],"predecessor-version":[{"id":1714,"href":"https:\/\/www.forillusion.com\/index.php\/wp-json\/wp\/v2\/posts\/1651\/revisions\/1714"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.forillusion.com\/index.php\/wp-json\/wp\/v2\/media\/1654"}],"wp:attachment":[{"href":"https:\/\/www.forillusion.com\/index.php\/wp-json\/wp\/v2\/media?parent=1651"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.forillusion.com\/index.php\/wp-json\/wp\/v2\/categories?post=1651"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.forillusion.com\/index.php\/wp-json\/wp\/v2\/tags?post=1651"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}