Skip to main content Link Menu Expand (external link) Document Search Copy Copied
Test-PCLP4-GH-Pages

Project: Website for security analysis

For the project you will need to fully implement a website that tracks down whether a link or file is malicious or not, similar to virustotal.

Description

The website that we will be implemeting has a landing page similar to the one of virustotal. It offers the possibility to upload a file or a post link. The input will then be checked from a security perspective and a result will be presented (whether the website or file is malicious or not). Alternatively, the database can be queried if it contains information regarding a file or a URL.

The website is comprised of 3 major components:

  • The frontend which is essentially the user facing interface, the actual website. When a specific action is taken (for example, a button is pressed), the frontend intercepts the action and sends a request to the middleware.
  • The middleware receives raw requests from the frontend and transforms them into calls to functions that are implemented in the backend.
  • The backend does the actual heavy lifting in terms of implementing the logic: it stores information in a database and computes whatever values are requested.

Each of the above components will have to be implemented for each of the 3 milestones of the projects. Essentially, each component represents an assignment.

Milestone 1 - Implementing the backend

The backend implements a database where all of the information is stored. We want to track down information about users, files and URLs. As a consequence, we will be using a database where we will define 3 collections:

  1. The users collection which stores the following fields:
    • an email address - a string that represents the unique identifier of user
    • a username - a string that represents the actual name that is publicly posted for a user
    • a password - a string that represents the encrypted password for a user
    • a name - an optional string field that represents the real name of the user
    • a description - an optional string field that represents the description of the user (hobbies, passions etc.)
  2. The files collection which stores the following fields:
    • a file id - a number that represents the unique identifier for an entry in this collection
    • a user id - a string that contains the email address of the user that added this file
    • the file contents - a ubyte[] that stores the bytes of the file
    • a hash of the file - a string that contains the result of applying a hash function to the file contents
    • threat level - a number representing the degree of maliciousness of the file
  3. The URLs collection which stores the following fields:
    • a URL id - a string that represents the unique identifier for an entry in this collection
    • a user id - a string that contains the email address of the user that added this URL
    • an address - a string that contains the actual URL (e.g. “www.google.com”)
    • a security level - a number representing the degree ofm maliciousness of the URL
    • a list of aliases - a string[] that contains different aliases for this website

The database is implemented using mongo-db. On top of mongo-db we will be using the vibe-d framework, which is a a high-performance asynchronous I/O, concurrency and web application toolkit written in D. By using vibe-d we will be able to both implement the database and create the server (for milestone 2).

Setup

The initial step is to install docker. We use docker to host the database.

On a Debian system we install docker by running the following command:

curl -fsSL test.docker.com -o get-docker.sh && sh get-docker.sh
sudo apt-get install -y docker-compose

After you have installed a working version of docker on your system, you will need to run the provided makefile to set up the database.

cd docker && make start

The assignment skeleton contains 2 directories:

  • docker: which contains the necessary files to configure the mondo database
  • db-conn: which contains the starting point for this milestone

You will implement this milestone in the db-conn/source/app.d file.

Working with mongo-db from D

To be able to access mongo from D we use the mongodb library from the vibe-d framework.

There is no need to install anything to use the vibe-d framework as thisis taken care of by the D package manager: dub.

To compile and run the application we need to navigate to the db-conn/ directory and run:

dub run

Do not mind the deprecation messages issued by dub when running the command above with the skeleton. They come from the vibe-d library and there’s nothing you can do about them.

To be able to interact with the database, we need to create a MongoClient object that will represent the connection to the database:

MongoClient client = connectMongoDB("mongodb://root:example@127.0.0.1:27017/");

The connectMongoDB function expects an URL, in the form of a string, and it establishes a connection that it. In our case, we use the localhost instance that was set up by docker using the credentials in the docker/docker-compose.yml: user - root; pass - example.

Now that we have a MongoClient instance we can use it to issue queries to the database. For example, we can get all of the entries for a particular collection:

MongoCollection users = client.getCollection("testing.users");

We can now use the users object to alter our collection:

users.insert(["_id": "unique_id", "key1": "demo", "key2": "test"]);

Mongo uses JSON to represent objects. In short: a glorified version of a key-value dictionary. The internal representation of a JSON in Mongo is called a BSON (Binary JSON) type. The mongodb library automatically converts D associative arrays into BSONs.

Coming back to the above example: it will insert in the users collection a document that consists of 3 key-value pairs.

Note that each mongo document must contain an _id field, otherwise mongo will automatically generate one.

To query the database for a single result we use the find function:

// "non-existent" does no exist in the database so a null Bson is returned
auto oneResult = users.findOne(["key1": "demo", "key2": "non-existent"]);
assert(oneResult == Bson(null));

// will return the first object that has key="key1" and value="demo"
auto oneResult = users.findOne(["key1": "demo"]);
assert(oneResult != Bson(null));

A more complex version of querying that returns all the results that match the query constraint is:

auto result = users.find(["key1": "demo", "key2": "test"]);

// find returns a MongoCursor, which is a range
assert(!result.empty);
foreach(r; result)
{
    writeln(r);
}

// foreach will consume the range by calling the popFront method, so now result will be an empty array []
writeln(result);

find returns all of the entries that contain the provided key-value pairings. The result may be iterated over with a foreach range. Be careful! Iterating over the result consumes the data! Similarly, you can use the remove and update functions:

// removes entry that has the key-value pairing provided
users.remove(["_id": "unique_id"]);

users.update(["_id" : "unique_id"],                                               // search criteria
             ["_id": "unique_id", "key1": "demo", "key2": "this_is_not_a_test"]); // updated entry

Tasks

You will have to implement the functions marked with TODO in the Assignment/m1/db_conn/source/app.d file.

To test your implementation simply run:

dub run

That will automatically compile and run the checker.

Sending the Assignment

On VMchcke, more details will follow soon…

Milestone 2

Milestone 3