PM Modi was also mum during Wrestlers' protests: Gogoi