Every CISO already knows that penetration testing is needed in order to properly determine exploitable weaknesses in systems, applications, people (in the case of social engineering tests), and physical safety. Penetration tests are also required by many regulations and industry compliance standards.
The question is should you pentest? Should CISOs have personal experience with penetration testing? In most cases, the answer is that it's not necessary, as long as there are good pentesters (and enough of them) on staff or the work has been contracted to a reputable third party.
Of course, having that hands on experience will set you apart. Understanding the intricacies of infiltration will help you turn theoretical book-knowledge into practical decision making, and only by getting your hands "dirty" can you truly understand how your adversaries work. This will enable you to develop stronger, more relevant defense strategies and, ironically, it will help you better illustrate the message in the board room.
The CISO day job doesn't really afford time for personal pen-testing, but its not a bad idea to try an occasional "capture the flag" or other hacking exercise during spare time, just to stay hack-fit.
Look out for the CTF and hacking tags on this blog to discover CTF write ups along with hacking tricks and tips.