Every CISO already knows that penetration testing is needed in order to properly determine exploitable weaknesses in systems, applications, people (in the case of social engineering tests), and physical safety. Penetration tests are also required by many regulations and industry compliance standards.

The question, however, is should you pentest?  Should CISOs have personal experience with penetration testing? In most cases, the answer is that it's not necessary, as long as there are good pentesters (and enough of them) on staff or the work has been contracted to a reputable third party.

Of course, having that hands on experience will set you apart. Understanding the intricacies of infiltration will help you turn theoretical book-knowledge into practical decision making, and only by getting your hands dirty can you truly understand how your adversaries work.  This will enable you to develop stronger, more relevant defense strategies and, ironically, it will help you better illustrate the message in the board room.

My day job doesn't really afford me any time to pentest, but I do try to occasionally participate in "capture the flag" and other hacking exercises, just to stay hack-fit. Look out for the CTF and hacking tags for write ups on some of these.