Feb 10, 2014

Penetration Testers - The Good, The Bad, The Kiddie

As a company you may want to conduct a penetration test once in a while to test your security posture. Nowadays there are many companies in the market offering penetration testing services. Suddenly you find yourself in the position to evaluate different penetration testing companies. You'll probably have to check which penetration testing company best suites your needs and you'll have to distinguish between those who are competent and those who are not. But how can you recognize the good pentesters and what are signs that unmask low-skilled pentesters?

In this article I will introduce you to factors that differentiate good penetration testers from bad ones or even worse - from Skriptkiddies. I will talk about  (1) obvious stuff like things to look out for on a CV or website, (2) certifications and (3) things to ask in an interview or company review.
You may also find this article interesting when you think about hiring a penetration tester for your team and are not sure how to assess his qualifications and skillset.



The obvious stuff

There are some basic things you should check out. You don't need to have a lot of IT-Security experience to be able to assess the following items. If the following things are well done, you can progress to the other factors further down on our list. If they are lousily done, you should think about the pentester's or company's attitude towards quality:

  • The pentester's CV or the pentesting company's website
  • If there is a profile on Crossing / LinkedIn, take a look at it
  • Are there any recommendations from former employers, colleagues or clients?
  • A list of former employers or clients
  • Security clearances if it applies to your industry (e.g. work for the government / public sector)

Certifications are important... right?

So while checking on the things above you discovered some cryptic acronyms on the pentester's CV or at the company's website? Most probably the acronyms were security certifications like Certified Information Systems Security Professional (CISSP), Offensive Security Certified Professional (OSCP) or Certified Ethical Hacker (CEH). 
There are plenty of other security certifications out in the market and there is at least an equally big amount of opinions about certifications out there.
A lot of security certifications require only theoretical knowledge and taking a written exam of some sort as a final test. This is in some cases an extended multiple-choice test. You can pass some of the exams if you are a good learner, even if you don't know a thing about IT-Security.
This does not mean that every certificate holder has no idea about security. A certificate proves that the certificate holder spent a certain amount of time on preparations for the certificate, nothing more and nothing less.
Meeting a pentester who has no certificates at all does not necessarily mean he does not know his craft. On the other hand, meeting a pentester who is holding 5 certificates does not necessarily mean that he knows his craft. I think you see the point I'm trying to make here:
Certificates can be a good indicator for skill but don't rely on them as your single factor of assessing an individual's or organization's skill level.


Skillset and practical experience

Practical experience and in-depths knowledge of penetration testing and ethical hacking is what differentiates the real expert from the Scriptkiddie. But how can you recognize inexperienced pentesters if you don't know a thing about pentesting yourself?
Ask for the following things and based on the answers and reactions you'll receive you will rapidly get a solid impression of whether your potential security professional is experienced or if he is just a tool-jockey.

  • Ask for an example report and ask to be walked through by the pentester / company:
    • Ask questions! How well can the findings of a report be explained?
    • Does the report only contain raw tool output or are there also descriptions and explanations around it? If it is just tool output, how much deeper does the verbal explanation go than "Nessus says you have a critical Vulnerability on machine XYZ"?
  • Ask for practical experience:
    • On which projects did the pentester / the company work before? Are these projects similar to the ones you are planning to conduct? If you are planning to do work for the public health sector or if you are public sector yourself, ask yourself if the pentester / company doing a lot of work only for small businesses is the best choice for your intents
  • How broad is the skillset (horizontal skillset)? 
    • How many domains does the pentester know and is experienced in? 
    • Does he only know about web-application testing or does he also have experience in mobile testing and Social Engineering? Usually, the more domains a pentester has experience in, the better it is for his overall understanding
  • How deep is the skillset (vertical skillset)? 
    • When the talk comes to a certain domain of ethical hacking (e.g. web-app testing), how well can the pentester explain the underlying concepts of vulnerabilities, exploits or how a certain tools works?
    • Can he abstract so much that even you as a non-security person can well understand what he is talking about?
    • Can he explain defense mechanisms, related hacking techniques and remediation actions? Asking for these things can quickly show you if the pentester is really experienced or if he is just pretending to be an expert
  • Present the pentester / company with a vulnerable machine or a map of a vulnerable target network or a web-app and let them demonstrate their skills. Rather than assessing if they manage to breach the target you should look at the approach:
    • How is the task tackled?
    • Is a structured approach used, a methodology or is it  rather randomly poking at the target system?
    • Are they manually checking on the findings or are they relying on tools very heavily?
  • Carefully listen when the pentester / company talks:
    • Are the explanations tool-centric? Is there a focus on output and results of tools? A good pentester can abstract from tool output and process results in such a way that they are understandable for non-security people
  • When evaluating an individual pentester, look at his hobbies:
    • Some hackers tend to have a broad range of hobbies, reflecting their burning interest for knowledge and growth. This is usually a good sign for interest but can easily be mistaken as an indicator for someone who tries to do everything at once, but does nothing really in depth

All in all you should try to dig deep. Challenge your potential candidates, be it an individual or a penetration testing company. Hackers usually like to be challenged, so go ahead and challenge them.


TL;DR: If you want to hire a penetration tester or a penetration testing company, go beyond the usual CV-checks and meet them on their field of expertise. Ask them to explain technical findings to you and listen carefully when they start talking. Value provable practical experience over certifications.

No comments:

Post a Comment