Web application security: Complete beginner’s guide

Advances in web applications, web services, and other web technologies have changed the way we do business and access and share information. Many businesses have shifted most of their operations online so employees from remote offices and business partners from different countries can share sensitive data in real time and collaborate towards common goals.

With the introduction of modern HTML5 web applications and JavaScript frameworks, customer and user demands have changed: now everyone expects 24/7 access to all the data and functionality they need. Such demands are also pushing businesses into making such data available online via web applications. A perfect example of this are online banking systems and online shopping websites.

All these advancements in web applications have also attracted malicious hackers and scammers, who are always coming up with new attack vectors because – like in any other industry – there is money to be gained illegally. This has led to the birth of a separate industry: web application security.

This article explains the basics and myths of web application security and shows how businesses can improve the security of their websites and web applications to keep malicious hackers at bay.

Table of contents

We have a secure network firewall

Most probably this is the most common web application security myths. Many think that the network firewall they have in place to secure their network will also protect the websites and web applications sitting behind it.

Network security differs from web application security. In network security perimeter defences such as firewalls are used to block the bad guys out and allow the good guys in. For example, administrators can configure firewalls to allow specific IP addresses or users to access specific services and block the rest.

But perimeter network defences are not suitable to protect web applications from malicious attacks. Business websites and web applications have to be accessed by everyone, therefore administrators have to allow all incoming traffic on port 80 (HTTP) and 443 (HTPS) and hope that everyone plays by the rules.

Network firewalls cannot analyze web traffic sent to and from the web applications, therefore it can never block any malicious requests sent by someone trying to exploit a vulnerability such as an SQL injection or Cross-site Scripting.

We scan our servers and networks with a network security scanner

Network security scanners are designed to identify insecure server and network device configurations and security vulnerabilities and not web application vulnerabilities (like SQL Injection). For example if an FTP server allows anonymous users to write to the server, a network scanner will identify such problem as a security threat. Network security scanners can also be used to check if all of the scanned components, mainly servers and network servers such as FTP, DNS, SMTP etc are fully patched.

What about a web application firewall?

A web application firewall, also known as WAF, does analyse both HTTP and HTTPS web traffic, hence it can identify malicious hacker attacks because it works at the application layer. For example, if the attacker is trying to exploit a number of known web application vulnerabilities in a website, it can block such connection thus stopping the attacker from successfully hacking the website. But such an approach has a number of shortcomings:

Only detects known security vulnerabilities

A web application firewall can determine if a request is malicious or not by matching the request’s pattern to an already preconfigured pattern. Therefore most of the time web application firewall cannot protect you against new zero day vulnerabilities and attack vectors. However, some of them can protect you against denial of service attacks.

As good as the administrator

A web application firewall is a user configurable software or appliance, which means it depends on one of the weakest links in the web application security chain, the user. Therefore if not configured properly, the web application firewall will not fully protect the web application.

Does not fix security holes in web applications

A web application security firewall does not fix and close the security holes in a web application, it only hides them from the attacker by blocking the requests trying to exploit them. Therefore if the web application firewall has a security issue and can be bypassed as seen in the next point, the web application vulnerability will also be exploited.

A WAF is a normal application that can have vulnerabilities

A web application firewall is a normal software application that can have its own vulnerabilities and security issues. Over time many security researchers identified several vulnerabilities in web application firewalls that allow hackers to gain access to the firewall’s admin console, switch off the firewall and even bypass the firewall.

Overall web application firewalls are an extra defence layer but are not a solution to the problem. In other words, if the budget permits it is of good practise to add a WAF after auditing a web application with a web vulnerability scanner. Additional layers of security should be always welcome!

Web application vulnerabilities should be treated as normal functionality bugs, therefore, should always be fixed, irrelevant if there is a firewall or any other type of defence mechanism in front of the application. In fact, web application security testing should be part of the normal QA tests.

How can I secure my web applications?

To ensure that a web application is secure you have to identify all security issues and vulnerabilities within the web application itself before a malicious hacker identifies and exploits them. That is why it is very important that the web application vulnerabilities detection process is done throughout all of the SDLC stages, rather than once the web application is live.

There are several different ways to detect vulnerabilities in web applications. You can scan the web application with a black-box scanner, do a manual source code audit, use an automated white box scanner to identify coding problems, or do a manual security audit and penetration test.

Which is the best method? There is no single bulletproof method that you can use to identify all vulnerabilities in a web application. Each of the methods mentioned above has its own pros and cons.

For example, while an automated tool will discover almost all technical vulnerabilities, more than a seasoned penetration tester can, it cannot identify logical vulnerabilities. Logical vulnerabilities can only be identified with a manual audit. On the other hand, a manual audit is not efficient and can take a considerable amount of time and cost a fortune. With a manual audit, there are also the risks of leaving unidentified vulnerabilities. White box testing will complicate the development procedures and can only be done by the developers who have access to the code.

If budget and time permit it is recommended to use a variety of all available tools and testing methodologies, but in reality no one has the time and budget to permit it. Therefore one has to choose the most cost effective solution that can realistically emulate a malicious hacker trying to hack a website; use a black box scanner, also known as web application security scanner or web vulnerability scanner. Of course, an automated web application security scan should always be accompanied by a manual audit. Only by using both methodologies you can identify all types of vulnerabilities, i.e. logical and technical vulnerabilities.

Web vulnerability scanners

A black-box web vulnerability scanner, also known as a web application security scanner is a software that can automatically scan websites and web applications and identify vulnerabilities and security issues within them. Web application security scanners have become really popular because they automate most of the vulnerability detection process and are typically very easy to use. For example to use a white box scanner one has to be a developer and needs access to the source code, while a black box scanner can be used by almost any member of the technical teams, such as QA team members, software testers, product and project managers etc.

Choosing the right web vulnerability scanner

There are several commercial and non commercial web vulnerability scanners available on the internet and choosing the one that meets all your requirements is not an easy task. The best way to find out which one is the best scanner for you is to test them all. Below are some guidelines to help you plan your testing and identify the right web application security scanner.

Commercial vs open-source web vulnerability scanners

There are many factors which will affect your decision when choosing a web application security scanner. The first obvious one is; should I use a commercial software or use a free, non-commercial solution? I recommend and always preferred commercial software. There are several reasons why, such as frequent updates of the software itself and the web security checks, ease of use, professional support and several others. For more information and detailed explanation of the advantages of using a commercial solution as opposed to a free one, refer to the article Should you pay for a web application security scanner?

How to test a web vulnerability scanner

Will you be scanning a custom web application built with .NET or a well known web application built in PHP, such as WordPress? Whichever web application you will be scanning, the security scanner you will be choosing should be able to crawl and scan your website. Although this sounds like the obvious, in practice it seems not.

For example, many choose a web vulnerability scanner based on the results of a number of comparison reports released over a number of years, or based on what the web security evangelists say. Although such information can be of an indication of who are the major players, your purchasing decision should not be totally based on it.

Many others take another wrong testing approach when comparing web vulnerability scanners; they scan popular vulnerable web applications, such as DVWA, bWAPP or other applications from the OWASP’s Broken Web Applications Project. It is a wrong approach because unless the web applications you want to scan are identical (in terms of coding and technology) to these broken web applications, which I really doubt, you are just wasting your time. Such vulnerable web applications are built for educational purposes and are not in any way similar to a real live web application.

The best approach to identify the right web application security scanner is to launch several security scans using different scanners against a web application, or a number of web applications that your business uses. Note that it is recommended to launch web security scans against staging and testing web applications, unless you really know what you are doing.

Ability to identify web application attack surfaces

During test scans verify which of the automated black box scanners has the best crawler; the component that is used to identify all entry points and attack surfaces in a web application prior to start attacking it. The crawler is most probably the most important component because a vulnerability cannot be detected unless the vulnerable entry point on a web application is identified by the crawler.

To identify the scanner which has the ability to identify all attack surfaces compare the list of pages, directories, files and input parameters each crawler identified and see which of them identified the most or ideally all parameters. If a particular scanner was unable to crawl the web application properly, it might also mean that it might need to be configured, which brings us to the next point; easy to use software.

Easy-to-use web vulnerability scanner

While some black box scanners can automatically crawl almost any type of website using an out of the box configuration, some others might need to be configured before launching a scan.

Because web application security is a niche industry, not all businesses will have web security specialists who are able to understand and configure a web application security scanner. Therefore go for an easy to use scanner that can automatically detect and adapt to most of the common scenarios, such as custom 404 error pages, anti-CSRF protection on website, URL rewrite rules etc.

Easy to use web application security scanners will have a better return on investment because you do not have to hire specialists, or train team members to use them.

Ability to identify web application vulnerabilities

The next factor used in comparing web application security scanner is which of the scanners can identify the most vulnerabilities, which of course are not false positives. I have seen vulnerability scanners identified hundreds of vulnerabilities on a website, but more than 70% of them were false positives.

If a scanner reports a lot of false positives, developers, QA people and security professionals will spend more time verifying the findings rather than focusing on remediations, hence try to avoid it. For more more information about false positives and their negative effect on web application security refer to the article The Problem of False Positives in Web Application Security and How to Tackle Them.

Automating web application security

The more a web application security scanner can automate, the better it is. For example imagine a web application with 100 visible input fields, which by today’s standards is a small application. If a penetration tester had to manually test each input on the web application for all known variants of cross-site scripting (XSS) vulnerabilities, he would need to launch around 800 different tests.

If each test takes around 2 minutes to complete, and if all works smoothly such a test would take around 12 days should the penetration tester work 24 hours a day. And this is just about the visible parameters. And what about the under the hood parameters?

Typically there is much more going on in a web application hidden under the hood rather than what can be seen. Therefore it is difficult for a penetration tester to rapidly identify all attack surfaces of a web application, while an automated web application security scanner can do the same test and identify all “invisible” parameters in around 2 or 3 hours.

But it is not just about time and money. When hiring a security professional for a web application penetration test, it will be limited to the professional’s knowledge, while on the other hand, a typical commercial web application security scanner contains large numbers of security checks and variants backed by years of research and experience.

Therefore automation is another important feature to look for. By automating the security test will cost less and is done more efficiently. For more information about the advantages of automating web application vulnerability detection, refer to Why Web Vulnerability Testing Needs to be Automated.

When to use a web vulnerability scanner

Web application security is something that should be catered for during every stage of the development and design of a web application. The earlier web application security is included in the project, the more secure the web application will be and the cheaper and easier it would be to fix identified issues at a later stage.

For example, an automated web application security scanner can be used throughout every stage of the software development lifecycle (SDLC). Even when the web application is in its early stages of development when it just has a couple of non visible inputs. Testing in the early stages of development is of utmost importance because if such inputs are the base of all other inputs, later on it would be very difficult if not impossible to secure them unless the whole web application is rewritten.

There are also several other advantages to using a vulnerability scanner throughout every stage of the SDLC. For example developers are automatically trained in writing more secure code because apart from just identifying vulnerabilities, most commercial scanners also provide a practical solution to how to fix the vulnerability. This helps developers understand and get to know more about web application security.

A complete guide to securing the web application environment

Scanning a web application with an automated web application security scanner will help you identify technical vulnerabilities and secure parts of the web application itself. But what about the logical vulnerabilities and all the other components that make up a web application environment?

Identifying logical vulnerabilities

Web application security scanners can only identify technical vulnerabilities, such as SQL Injection, Cross-Site Scripting, Remote Code execution etc. Therefore an automated web application security scan should always be accompanied by manual audit to identify logical vulnerabilities.

Logical vulnerabilities could also have a major impact on business operations therefore, it is very important to do a manual analysis of the web application by testing several combinations and ensure that the web application works as it was meant to be.

Example of a logical vulnerability

Imagine a shopping cart that has the price specified in the URL as per the example below:

/shoppingcart/index.php?price=250

What happens if the user changes the price from $250 to $30 in the URL? Will the user be able to proceed with the checkout and pay just $30 for an item that costs $250? If yes then that is a logical vulnerability that could seriously impact your business.

These types of vulnerabilities can never be identified by an automated tool because tools do not have the intelligence that allows them to determine the effect such a parameter could have on the operations of the business.

Securing the web server and other components

There are several other components in a web application farm that make the hosting and running of a web application possible. In a very basic environment at least there is the web server software (such as Apache or IIS), web server operating system (such as Windows or Linux), database server (such as MySQL or MS SQL) and a network based service that allows the administrators to update the website, such as FTP or SFTP.

All of these components that make up a web server also need to be secure because if any of them is broken into, the malicious attackers can still gain access to the web application and retrieve data from the database or tamper it. Therefore it is recommended that you to refer to the security guidelines and best practises documentation for the software you are using on your web server. Below are also some basic security guidelines which could be applied to any type of server and network based service:

Switch off unnecessary functionality

The more functionality a network service or operating system has, the bigger the chances are of having an exploitable entry point. Therefore switch off and disable any functionality, services or daemons which are not used by your web application environment. For example typically a web server operating system has an SMTP service running. If you are not using such service switch it off and ensure that it is permanently disabled.

Limit and secure remote access

Ideally administrators should be able to login to the web server locally. If not possible though ensure that any type of remote access traffic such as RDP and SSH is tunnelled and encrypted. It would also be beneficial if you can limit the remote access to a specific number of IP addresses, such as those of the office.

Use accounts with limited privileges

Administrators do not typically like any type of restriction on their own accounts because sometimes limited privileges can be a little bit cumbersome to complete a specific task. Therefore if you work towards finding the right balance between security and practicality, you can have a secure web server while administrators can still do their job. For example, an administrator can have different accounts to do different tasks; an account which is specifically used for backups, an account which is used for generic operations such as pruning of log files, an account which is used solely to change the configuration of services such as FTP, DNS, SMTP etc.

By using such an approach, you are limiting the damage that could be done if one of the administrator’s account is hijacked by a malicious attacker.

Permissions and privileges

Complementing with user accounts, the same applies to every other type of service and application. For example, most of the time the database user your web application is using to connect to the database only needs to read and write data to and from the database and does not need privileges to create or drop tables. But yet most of the time most administrators give an account all possible privileges because it “will always work”.

Another typical scenario for this type of problems are FTP users. FTP users who are used to update the files of a web application should only have access to those files and nothing else. Take the time to analyse every application, service and web application you are running and ensure the least possible privileges are given to the user, application and service.

Segregate development, testing, and live environments

It is of utmost importance to always segregate live environments from development and testing environments. By mixing such environments you are inviting hackers into your web application.

When developing or troubleshooting a web application developers leave traces behind them that could help a malicious hacker to craft an attack against the web application. For example debug, which could be used to expose sensitive information about the environment of the web application is left enabled. Log files containing sensitive information about the database setup can be left on the website and could be accessed by malicious users.

Hence why it is important that any development and troubleshooting is done in a staging environment. Once the development and testing of a web application is finished, the administrator should apply the changes to the live environment and also ensure that any of the applied changes do not pose any security risks and that no files, such as log files or source code files with sensitive technical comments are uploaded to the server.

Segregate data

Similar to the above, the same applies to the data itself. Do not keep non related information in the same database, such as customers credit card numbers and website user activity. Store such data into different databases using different database users.

Apply the same segregation concept on the operating system and web application files. Ideally, web application files, i.e. the directory which is published on the web server should be on a separate drive from the operating system and log files. By doing so you are not exposing operating system files to the malicious attacker in case he or she exploits a vulnerability on the web server.

Always install security patches

Even though this is one of the most important steps in any type of security, unfortunately, this is still the most overlooked task. It cannot be stressed enough how important it is to always use the latest and most recent version of a particular software you are using and to always apply the vendor’s security patches. By doing so you ensure that malicious hackers cannot find and exploit any known security vulnerability in the software you use.

Monitor and audit the servers and logs

As the name implies, log files are used to keep a log of everything that is happening on the server and not simply to consume an infinite amount of hard disk space. From time to time every administrator should analyse the server log files. By doing so administrators can uncover a lot of information, such as suspicious behaviour on the server and therefore can better protect the web server better, or in case of an attack, can easily trace back what happened and what was exploited during the attack.

Use security tools

Apart from a web application security scanner, you should also use a network security scanner and other relevant tools to scan the web server and ensure that all services running on the server are secure. Security tools should be included in every administrator’s toolbox.

Stay informed

Last but not least, stay informed! Nowadays, you can find a lot of information for free on the Internet from a number of web application security blogs and websites, not least this very blog (to which you can also subscribe so you don’t miss anything). By keeping yourself informed on what is happening in the web application security and the cybersecurity industry in general, you are arming and educating yourself, so you’ll be able to better protect and secure web servers and web applications.

Complete beginner’s guide to web application security

We have a secure network firewall

We scan our servers and networks with a network security scanner

What about a web application firewall?