How to Scan REST APIs and Web Services with Invicti

What are Web APIs?

Application Programming Interfaces (APIs) can enable systems to talk to each other or control communication between systems. The interfaces allow easy and isolated access to an application’s functions and can greatly simplify management. By exposing APIs, developers can share specific application functionality.

In a web context, APIs are called web services. The use of web services has dramatically changed web application architecture. Traditionally, each web application executed all the tasks it needed, such as logic processing, data storage, and presentation. With the widespread use of web services, web applications mostly take care of presentation, while data processing and business logic are handled via APIs calls.

API requests constitute majority of modern web traffic. According to the Akamai report on the State of the Internet, communication over APIs accounts for 83% of all web traffic, with most of the usage coming from web services and mobile applications. In spite of this, web API security is mostly overlooked. This neglect results from the continued belief in security through obscurity – after all, web services perform their jobs in the background, so they shouldn’t attract the attention of attackers, right? Unfortunately, research from Gartner suggests otherwise. 

According to Gartner’s analysts, APIs are set to become a major attack vector for malicious hackers. Gartner’s report How to Build an Effective API Security Strategy predicts that APIs will become the number one attack vector in 2022, while the API Strategy Maturity Model from October 2019, again from Gartner, expects APIs to have a wide attack surface by 2021. These predictions have prompted many people to pay more attention to securing their web services, APIs, and applications. 

To help you secure these assets, this article will walk you step by step through scanning your web services with Invicti and choosing the best configuration for complete and accurate results.  

Discovering API Endpoints for Web Services

Invicti can easily scan applications that use web APIs and services. Invicti can detect web service declaration files during crawling, including WSDL and OpenAPI (Swagger), and import endpoint URLs to test the security controls of API endpoints covered by the scan.

In any security test, the first phase consists of collecting of resources through URLs and input points where security checks will be carried out. When web service declarations are provided, attackers have an advantage in determining the application attack surface. 

Generally, web service declarations are very detailed. For example, they specify which function or endpoint takes which parameter and what its response and function will be. However, to protect against attacks, service owners may prefer to share API declarations and documentation offline, without publishing them online. Invicti will not be able to detect such declarations on the target, so you can use the scanner’s interface to import WSDL and OpenAPI files.

Detecting Endpoints Used by Mobile Applications

Mobile applications present another challenge in detecting declarations as they use endpoints embedded into the application. It is important to keep in mind that mobile applications have a very different architecture to web applications. In mobile applications, mobile devices handle the presentation while web services take care of the data and business logic. Fortunately, these web services are very similar to those used in web applications, so a web application scanner can also scan web services that provide the backend for mobile applications.

When mobile applications access web services, they use endpoints hard-coded into the application. This means that the service declarations of these endpoints are not shared, which requires a different approach to detecting sources during the first phase of a security test. In this case, web traffic generated by mobile applications can be monitored via a proxy to detect API endpoints and input points.

At this stage, you can use Invicti’s Internal Proxy feature. When this is enabled, Invicti detects all requests, directs them to the target, and records them for use in the attack phase. In Invicti, you can enable the proxy to listen to connections from other devices. To do this, activate the Allow Remote Connections feature in the Options window, as shown below.

Once you enable this option to direct traffic from the mobile device to Invicti, you can go to Proxy settings in the connection settings of your mobile device and enter the IP of the computer where Invicti is running, with port number 10010.

Now start a Invicti scan using Manual Scan (Proxy Mode) so the scanner can listen to the mobile application’s traffic. Invicti will prompt you for a target URL in the New Scan window. As you don’t know the endpoints used by the mobile application, enter a domain that is valid for your license. After making some requests to the target URL using Manual Scan (Proxy Mode), you will be able to feed the crawl list by navigating to suitable URLs in a browser or mobile application, so that Invicti will record and analyze these requests.

Once the scan has started, you can view requests detected by Invicti under Knowledge Base > Out of Scope Links via the mobile application, as shown below.

Based on your sites to scan, you can start a scan for web service endpoints that are reported as out of scope. Alternatively, you can explore API endpoints used by mobile applications using a debugging proxy like Fiddler.  

Meet Invicti’s REST API Test Site: rest.testinvicti.com

Representational State Transfer, or REST, is an architectural style that can be used to communicate with web services. REST can serve as a communication mechanism between two applications or between an application and an online service. Many mobile applications communicate with online services through a RESTful API at the backend.

We will scan rest.testinvicti.com to showcase Invicti’s capabilities in the security scanning of web services. Then we will go through the exact settings that Invicti uses for this scan. 

Prepared by Invicti security engineers, rest.testinvicti.com exposes a vulnerable REST API service. The test environment is based on a content management system scenario and you can use it to test Invicti’s crawling and attacking capabilities. 

For best results when using Invicti to scan test environments prepared by our security engineers, you can use preconfigured Scan Profile and Policy files. You can import these into Invicti manually, but if you enter the vulnerable REST API site as a target URL, Invicti will prompt you to use the scan policy and profile configured for the test application.

Like our other test sites, rest.testinvicti.com is preconfigured with the best policy and profile to test this environment. You can instruct the scanner to download and use this policy and profile and to start the scan immediately.  

The policy wizard will ask you some questions to determine which configuration you want to use while scanning the REST API test site.

In the wizard, select Basic Authentication (JSON+XML). As you can see, the REST API test site also supports JWT and OAuth authentication.  

After completing the wizard, Invicti Standard will load policy and profile files for the relevant test website. The wizard will have the following Policy and Profile based on your selection:

Setting a Profile for Basic Authentication (XML+JSON Body)

Invicti scan profiles allow you to save the target, scope, authentication, and policy settings as a named scan profile that you use to run a scan later. When you view your current profile, you can see that the target URL is http://rest.testinvicti.com/basic_authentication/api/. Because you selected basic authentication, the path of this URL is /basic_authentication/api/. For JWT Authentication, the path would be /jwt/api/, and for OAuth2 it would be /jwt/api/.

Looking at the profile settings, you will see that 35 REST API resource URLs have been imported automatically with the profile file:

While scanning the REST API test site, there might be no responses that Invicti can parse and add to the site map. One of the first steps of scanning REST services is to import resource URLs into the scanner. As mentioned earlier, Invicti can parse web service declarations and import endpoint URLs from web service declaration files.

Another important part of a scan profile is entering URL rewrite rules. Instead of testing multiple numbered resources that could point to the same backend functionality, Invicti will test one page to obtain the best results in less time. For example, let’s look at the following resource in the URL Rewrite list:

http://rest.testinvicti.com/basic_authentication/api/posts/{postid}

This rule would cover all of the following URLs: 

http://rest.testinvicti.com/basic_authentication/api/posts/1
http://rest.testinvicti.com/basic_authentication/api/posts/2
http://rest.testinvicti.com/basic_authentication/api/posts/3
http://rest.testinvicti.com/basic_authentication/api/posts/4
http://rest.testinvicti.com/basic_authentication/api/posts/5
...
http://rest.testinvicti.com/basic_authentication/api/posts/100

A request to attack this endpoint takes the integer parameter postid, enabling you to use just one request to obtain the best results for multiple resources that meet this pattern. Normally, Invicti collects a certain number of samples to decide on URL rewrite patterns. By importing this rule manually, you can save time, but you can see that the heuristic detection option is enabled. Invicti will collect and process samples to decide on any patterns that you did not import manually. 

As you can see, Invicti provides a way to get full coverage without having to collect any samples in the scanner. With the URL rewrite detection feature, Invicti completes scanning in the shortest time possible and with the best results and coverage. 

Basic Authentication in the Authorization Header

We already know that the REST API test site supports Basic Authentication, OAuth2, and JWT Authentication. Since you selected Basic Authentication in the profile options, the setting is created as shown below.

When the scanner makes requests to REST sources, it will send these credentials in the Authorization header.

After you select Start Scan and scanning begins, Invicti will crawl and scan all endpoints at the REST API test site and list all its findings. It will test all input points in all JSON and XML type requests provided on REST API test site and can carry out attacks on all these input points.

For more detailed information, see the REST API test site documentation.

Conclusion

Industry experts agree that web services and APIs will increasingly come under attack in the coming years. Recent events, such as the Facebook data leak caused by an API vulnerability and the discovery of API-related vulnerabilities in SoundCloud and Lime Scotter, underline the importance of API security. In 2019, OWASP even added a new Top 10 list: the API Security Top Ten.

As you can see from this demonstration of the test environment at rest.testinvicti.com, Invicti can extensively analyze security vulnerabilities in web services and APIs – a vital component of modern web and mobile applications. With Invicti, you can easily automate web service security testing, while its proprietary Proof-Based Scanning technology ensures that results are always accurate and trustworthy.

Original article written by Ziyahan Albeniz