To "try your hand" at something means to do an activity for the first time in order to find out whether one likes it or is good at it. Michael is a dominant, and so is Elijah, BDSM is something that is not new to either of them, but being in a relationship with another dominant is. They have no idea how this is going to work.