pytorch自定義不可導(dǎo)激活函數(shù)的操作
pytorch自定義不可導(dǎo)激活函數(shù)
今天自定義不可導(dǎo)函數(shù)的時(shí)候遇到了一個(gè)大坑。
首先我需要自定義一個(gè)函數(shù)：sign_f
import torch from torch.autograd import Function import torch.nn as nn class sign_f(Function): @staticmethod def forward(ctx, inputs): output = inputs.new(inputs.size()) output[inputs >= 0.] = 1 output[inputs < 0.] = -1 ctx.save_for_backward(inputs) return output @staticmethod def backward(ctx, grad_output): input_, = ctx.saved_tensors grad_output[input_>1.] = 0 grad_output[input_<-1.] = 0 return grad_output
然后我需要把它封裝為一個(gè)module 類型，就像 nn.Conv2d 模塊 封裝 f.conv2d 一樣，于是
import torch from torch.autograd import Function import torch.nn as nn class sign_(nn.Module): # 我需要的module def __init__(self, *kargs, **kwargs): super(sign_, self).__init__(*kargs, **kwargs) def forward(self, inputs): # 使用自定義函數(shù) outs = sign_f(inputs) return outs class sign_f(Function): @staticmethod def forward(ctx, inputs): output = inputs.new(inputs.size()) output[inputs >= 0.] = 1 output[inputs < 0.] = -1 ctx.save_for_backward(inputs) return output @staticmethod def backward(ctx, grad_output): input_, = ctx.saved_tensors grad_output[input_>1.] = 0 grad_output[input_<-1.] = 0 return grad_output
結(jié)果報(bào)錯(cuò)
TypeError: backward() missing 2 required positional arguments: 'ctx' and 'grad_output'
我試了半天，發(fā)現(xiàn)自定義函數(shù)后面要加 apply ，詳細(xì)見下面
import torch from torch.autograd import Function import torch.nn as nn class sign_(nn.Module): def __init__(self, *kargs, **kwargs): super(sign_, self).__init__(*kargs, **kwargs) self.r = sign_f.apply ### <-----注意此處 def forward(self, inputs): outs = self.r(inputs) return outs class sign_f(Function): @staticmethod def forward(ctx, inputs): output = inputs.new(inputs.size()) output[inputs >= 0.] = 1 output[inputs < 0.] = -1 ctx.save_for_backward(inputs) return output @staticmethod def backward(ctx, grad_output): input_, = ctx.saved_tensors grad_output[input_>1.] = 0 grad_output[input_<-1.] = 0 return grad_output
問題解決了！
PyTorch自定義帶學(xué)習(xí)參數(shù)的激活函數(shù)（如sigmoid）
有的時(shí)候我們需要給損失函數(shù)設(shè)一個(gè)超參數(shù)但是又不想設(shè)固定閾值想和網(wǎng)絡(luò)一起自動(dòng)學(xué)習(xí)，例如給Sigmoid一個(gè)參數(shù)alpha進(jìn)行調(diào)節(jié)
函數(shù)如下：
import torch.nn as nn import torch class LearnableSigmoid(nn.Module): def __init__(self, ): super(LearnableSigmoid, self).__init__() self.weight = torch.nn.Parameter(torch.FloatTensor(1), requires_grad=True) self.reset_parameters() def reset_parameters(self): self.weight.data.fill_(1.0) def forward(self, input): return 1/(1 + torch.exp(-self.weight*input))
驗(yàn)證和Sigmoid的一致性
class LearnableSigmoid(nn.Module): def __init__(self, ): super(LearnableSigmoid, self).__init__() self.weight = torch.nn.Parameter(torch.FloatTensor(1), requires_grad=True) self.reset_parameters() def reset_parameters(self): self.weight.data.fill_(1.0) def forward(self, input): return 1/(1 + torch.exp(-self.weight*input)) Sigmoid = nn.Sigmoid() LearnSigmoid = LearnableSigmoid() input = torch.tensor([[0.5289, 0.1338, 0.3513], [0.4379, 0.1828, 0.4629], [0.4302, 0.1358, 0.4180]]) print(Sigmoid(input)) print(LearnSigmoid(input))
輸出結(jié)果
tensor([[0.6292, 0.5334, 0.5869],
[0.6078, 0.5456, 0.6137],
[0.6059, 0.5339, 0.6030]])
tensor([[0.6292, 0.5334, 0.5869],
[0.6078, 0.5456, 0.6137],
[0.6059, 0.5339, 0.6030]], grad_fn=<MulBackward0>)
驗(yàn)證權(quán)重是不是會(huì)更新
import torch.nn as nn import torch import torch.optim as optim class LearnableSigmoid(nn.Module): def __init__(self, ): super(LearnableSigmoid, self).__init__() self.weight = torch.nn.Parameter(torch.FloatTensor(1), requires_grad=True) self.reset_parameters() def reset_parameters(self): self.weight.data.fill_(1.0) def forward(self, input): return 1/(1 + torch.exp(-self.weight*input)) class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.LSigmoid = LearnableSigmoid() def forward(self, x): x = self.LSigmoid(x) return x net = Net() print(list(net.parameters())) optimizer = optim.SGD(net.parameters(), lr=0.01) learning_rate=0.001 input_data=torch.randn(10,2) target=torch.FloatTensor(10, 2).random_(8) criterion = torch.nn.MSELoss(reduce=True, size_average=True) for i in range(2): optimizer.zero_grad() output = net(input_data) loss = criterion(output, target) loss.backward() optimizer.step() print(list(net.parameters()))
輸出結(jié)果
tensor([1.], requires_grad=True)]
[Parameter containing:
tensor([0.9979], requires_grad=True)]
[Parameter containing:
tensor([0.9958], requires_grad=True)]
會(huì)更新~
以上為個(gè)人經(jīng)驗(yàn)，希望能給大家一個(gè)參考，也希望大家多多支持本站。
版權(quán)聲明：本站文章來源標(biāo)注為YINGSOO的內(nèi)容版權(quán)均為本站所有，歡迎引用、轉(zhuǎn)載，請(qǐng)保持原文完整并注明來源及原文鏈接。禁止復(fù)制或仿造本網(wǎng)站，禁止在非maisonbaluchon.cn所屬的服務(wù)器上建立鏡像，否則將依法追究法律責(zé)任。本站部分內(nèi)容來源于網(wǎng)友推薦、互聯(lián)網(wǎng)收集整理而來，僅供學(xué)習(xí)參考，不代表本站立場(chǎng)，如有內(nèi)容涉嫌侵權(quán)，請(qǐng)聯(lián)系alex-e#qq.com處理。
關(guān)注官方微信