Install Malcolm Network Traffic Analysis Tool on Ubuntu 22.04

0
247

Follow through this tutorial to learn how to install Malcolm network traffic analysis tool on Ubuntu 22.04. Malcolm is a network traffic analysis tool suite for full packet capture artifacts (PCAP files) and Zeek logs. The PCAP files or Zeek logs can be uploaded to Malcolm via browser, forwarded via the forwarders or can capture live traffic, parses and normalize the traffic for visualization via OpenSearch dashboards or Arkime.

Read more about Malcolm network traffic analysis tool and its features on their page.

Install Malcolm Network Traffic Analysis Tool on Ubuntu 22.04

Malcolm provides scripts that can be used to easy deploy them on a Linux system.

There are two ways in which you can obtain the Malcolm installation script.

  1. Either by downloading the install.py and the malcolm_YYYYMMDD_HHNNSS_xxxxxxx.tar.gz from the Malcolm Github releases page.
  2. Or by simply cloning Malcolm Github repository.

In this tutorial, we will use the second method to install Malcolm network traffic analysis tool on Ubuntu 22.04.

Thus, to install Malcolm network traffic analysis tool on Ubuntu 22.04;

System Requirements

Check Malcolm deployment system requirements.

In our environment, these are the system resources we are using;

CPU cores;

sudo dmidecode -t 4 | grep -i 'core count'
	Core Count: 4

RAM;

free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi        14Gi       153Mi       7.0Mi       723Mi       372Mi
Swap:             0B          0B          0B

Disk Space;

df -hT -P /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      ext4  150G   14G  131G  10% /

Create Malcolm System Account

Create an account to run Malcolm with;

useradd -m -d /opt/malcolm -s /bin/bash -G sudo malcolm
passwd malcolm

Get the ID of the user as this is required while configuring Malcolm below.

id malcolm

Sample output;

uid=1001(malcolm) gid=1001(malcolm) groups=1001(malcolm),27(sudo)

Run system update;

Ensure system package cache is up-to-date.

sudo apt update

Install Git package on Ubuntu 22.04;

You will need git to clone the Malcolm Github repository

sudo apt install git

Clone Malcolm Github Repository

Run the command below to clone Malcolm Github repository;

su - malcolm
git clone https://github.com/idaholab/Malcolm

Install Required Malcolm Packages and Setup System Tunables

Next, change into cloned Malcolm Github repository directory;

cd Malcolm

Malcolm installation scripts are located under scripts directory. Thus, to launch the installation, execute the command below.

sudo ./scripts/install.py

Malcolm runs atop Docker. Thus, when the script runs;

  • You will be prompted to install required Docker packages. When prompted, accept to proceed with installation.
  • Once the installation of Docker packages is done, you are prompted on whether to add a non-root user account to Docker group. And by all means, please add your standard user account into the group!
  • Next, install Docker compose.
Installing required packages: ['apache2-utils', 'make', 'openssl', 'python3-dialog']

"docker info" failed, attempt to install Docker? (Y/n): y

Attempt to install Docker using official repositories? (Y/n): y
Installing required packages: ['apt-transport-https', 'ca-certificates', 'curl', 'gnupg-agent', 'software-properties-common']
Installing docker packages: ['docker-ce', 'docker-ce-cli', 'containerd.io']
Installation of docker packages apparently succeeded

Add a non-root user to the "docker" group?: y

Enter user account: kifarunix 

Add another non-root user to the "docker" group?: no

"docker-compose version" failed, attempt to install docker-compose? (Y/n): y

Install docker-compose directly from docker github? (Y/n): y
Download and installation of docker-compose apparently succeeded
...
  • Next, you will be prompted to update some kernel tunables. Accept for all prompts.
fs.file-max increases allowed maximum for file handles
fs.file-max= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y


fs.inotify.max_user_watches increases allowed maximum for monitored files
fs.inotify.max_user_watches= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y


fs.inotify.max_queued_events increases queue size for monitored files
fs.inotify.max_queued_events= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y


fs.inotify.max_user_instances increases allowed maximum monitor file watchers
fs.inotify.max_user_instances= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y


vm.max_map_count increases allowed maximum for memory segments
vm.max_map_count= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y


net.core.somaxconn increases allowed maximum for socket connections
net.core.somaxconn= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y


vm.swappiness adjusts the preference of the system to swap vs. drop runtime memory pages
vm.swappiness= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y


vm.dirty_background_ratio defines the percentage of system memory fillable with "dirty" pages before flushing
vm.dirty_background_ratio= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y


vm.dirty_ratio defines the maximum percentage of dirty system memory before committing everything
vm.dirty_ratio= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y


/etc/security/limits.d/limits.conf increases the allowed maximums for file handles and memlocked segments
/etc/security/limits.d/limits.conf does not exist, create it? (Y/n): y
  • Once that is done, the script will exit.

Run Malcolm System Configurations

Once the install script above completes, proceed to configure Malcolm;

sudo ./scripts/install.py --configure

When configuration starts;

  • Choose a user with which to run Malcolm as. We created malcolm user account above, whose UID and GID is 1001. Hence, say no to user with UID/GID 1000.
Malcolm processes will run as UID 1000 and GID 1000. Is this OK? (Y/n): n
  • Enter the user ID and group ID of malcolm system account.
Enter user ID (UID) for running non-root Malcolm processes; 1001
Enter group ID (GID) for running non-root Malcolm processes; 1001

Confirm;

Malcolm processes will run as UID 998 and GID 998. Is this OK? yes
  • OpenSearch and logstash settings;
Setting 10g for OpenSearch and 3g for Logstash. Is this OK? yes
Setting 3 workers for Logstash pipelines. Is this OK? (Y/n): yes
  • Restart Malcolm when system or Docker restarts;
Restart Malcolm upon system or Docker daemon restart? Yes

Choose when to restart Malcolm. You have four options here; no, on-failure, always, unless-stopped. Choose the default option, unless-stopped.

  • Choose whether to setup Malcolm with HTTPS. We select yes (SSL certificates will be generated at a later step)
  • Choose whether Malcolm will run behind any proxy. You are given some options like Caddy, Traefik..We choose No.
  • On Default networking, press enter to accept the defaults.
  • If prompted whether to use LDAP, select no for now.
  • Store OpenSearch index snapshosts locally in /opt/malcolm/Malcom/opensearch-backup? You can choose to accept the default or just enter the new path, /opt/malcolm/.
  • Choose to Compress OpenSearch index snapshots (yes)
  • When prompted to delete the oldest indices when the database exceeds a certain size choose no to avoid loosing data.
  • Automatically analyze all PCAP files with Suricata: yes
  • Automatically analyze all PCAP files with Zeek: yes
  • Disable reverse DNS lookup locally for source and destination IP addresses in logs.
  • Enable hardware vendor OUI lookups for MAC addresses.
  • Perform string randomness scoring on some fields: yes
  • Expose OpenSearch port to external hosts: no
  • Expose Logstash port to external hosts: no
  • Forward Logstash logs to external OpenSearch instance: no
  • Expose Filebeat TCP port to external hosts: no
  • Enable file extraction with Zeek: yes
  • choose interesting as the extraction behavior.
  • Select quarantine as the file preservation method.
  • Scan extracted files/PE files with ClamAV: yes
  • Scan extracted files/PE files with Yara: yes
  • Scan extracted files/PE files with Capa: yes
  • Lookup extracted file hashes with VirusTotal: no
  • Download updated scanner signatures periodically: yes
  • Should Malcolm capture network traffic to PCAP files: yes
  • Specify capture interface(s) (comma-separated) on which Malcolm will use to network traffic: enp0s8
  • Capture packets using netsniff-ng? (Y/n): yes
  • Capture packets using tcpdump? (y/N): no
  • Capture filter (tcpdump-like filter expression; leave blank to capture all traffic) (): You can disable traffic related to Elasticsearch (port 9200), Logstash (5044), Arkime(8005): not port 9200 and not port 5044 and not port 8005.
  • Disable capture interface hardware offloading and adjust ring buffer sizes? (y/N): n

Malcolm has now been installed to /home/$USER/Malcolm.

Scripts for starting and stopping Malcolm and changing authentication-related settings can be found in /home/$USER/Malcolm/scripts.

Run system reboot to apply some of the system changes;

sudo systemctl reboot -i

Create Malcolm User Account

Navigate back to the Malcolm Github repository directory;

su - malcolm
cd ~/Malcolm

Run the authentication setup script;

./scripts/auth_setup
  • Store administrator username/password for local Malcolm access? yes
  • Set administrator password and username.
  • (Re)generate self-signed SSL certs for web traffic HTTPS: yes
  • (Re)generate self-signed certificates for a remote log forwarder: yes
  • Store username/password for forwarding Logstash events to a secondary, external OpenSearch instance: no.
  • Store username/password for email alert sender account (see https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#authenticate-sender-account): no

Download Malcolm Docker Images

Next, run the command below to download Malcolm Docker images from the Docker hub.

cd ~/Malcolm
docker-compose pull

Depending on your internet speed, this may take a few mins to complete pulling the images.

Once all the required images are downloaded, you can list them as follow;

docker images

Sample output;

REPOSITORY                        TAG       IMAGE ID       CREATED       SIZE
malcolmnetsec/zeek                6.1.0     f866620ebd58   3 weeks ago   1.29GB
malcolmnetsec/dashboards          6.1.0     13775018c809   3 weeks ago   1.08GB
malcolmnetsec/arkime              6.1.0     5b7f6018db22   3 weeks ago   794MB
malcolmnetsec/logstash-oss        6.1.0     a8b93b43dbb3   3 weeks ago   1.46GB
malcolmnetsec/suricata            6.1.0     fa25601e29c4   3 weeks ago   285MB
malcolmnetsec/filebeat-oss        6.1.0     b4d1c83bf3de   3 weeks ago   648MB
malcolmnetsec/file-monitor        6.1.0     fa9438234b42   3 weeks ago   589MB
malcolmnetsec/nginx-proxy         6.1.0     bb9720d9d456   3 weeks ago   124MB
malcolmnetsec/file-upload         6.1.0     fdbd62ce5a92   3 weeks ago   259MB
malcolmnetsec/htadmin             6.1.0     623e193fd419   3 weeks ago   242MB
malcolmnetsec/opensearch          6.1.0     a1bbcaae7647   3 weeks ago   1.21GB
malcolmnetsec/pcap-monitor        6.1.0     0e0cb91f32a1   3 weeks ago   214MB
malcolmnetsec/api                 6.1.0     54a99d983b37   3 weeks ago   173MB
malcolmnetsec/pcap-capture        6.1.0     9a1664dc488f   3 weeks ago   121MB
malcolmnetsec/freq                6.1.0     e4691604218f   3 weeks ago   131MB
malcolmnetsec/dashboards-helper   6.1.0     ef14087c003d   3 weeks ago   169MB
malcolmnetsec/name-map-ui         6.1.0     90f49b94bfe3   3 weeks ago   123MB

Running Malcolm Service

You can now start Malcolm services using the startup script, ./scripts/start.

The script can be used to start, stop, restart, wipe Malcolm data, etc.

./scripts/start --help
usage: start 
Malcolm control script
options:
  -v [DEBUG], --verbose [DEBUG]
                        Verbose output
  -f , --file 
                        docker-compose YML file
  -s , --service 
                        docker-compose service (only for status and logs operations)
  -l [CMDLOGS], --logs [CMDLOGS]
                        Tail Malcolm logs
  --start [CMDSTART]    Start Malcolm
  --restart [CMDRESTART]
                        Stop and restart Malcolm
  --stop [CMDSTOP]      Stop Malcolm
  --wipe [CMDWIPE]      Stop Malcolm and delete all data
  --auth [CMDAUTHSETUP]
                        Configure Malcolm authentication
  --status [CMDSTATUS]  Display status of Malcolm components

You can simply start Malcolm by executing the start script with no arguments.

./scripts/start

It will take a few minutes to start all necessary Malcolm services.

The start script will launch Malcolm in the foreground and populate debug messages to the console.

Started Malcolm


In a few minutes, Malcolm services will be accessible via the following URLs:
------------------------------------------------------------------------------
  - Arkime: https://localhost/
  - OpenSearch Dashboards: https://localhost/dashboards/
  - PCAP upload (web): https://localhost/upload/
  - PCAP upload (sftp): sftp://[email protected]:8022/files/
  - Host and subnet name mapping editor: https://localhost/name-map-ui/

  - Account management: https://localhost:488/

NAME                          COMMAND                  SERVICE             STATUS               PORTS
malcolm-api-1                 "/usr/local/bin/dock…"   api                 running (starting)   5000/tcp
malcolm-arkime-1              "/usr/local/bin/dock…"   arkime              running (starting)   8081/tcp
malcolm-dashboards-1          "/usr/local/bin/dock…"   dashboards          running (starting)   5601/tcp
malcolm-dashboards-helper-1   "/usr/local/bin/dock…"   dashboards-helper   running (starting)   28991/tcp
malcolm-file-monitor-1        "/usr/local/bin/dock…"   file-monitor        running (starting)   8440/tcp
malcolm-filebeat-1            "/usr/local/bin/dock…"   filebeat            running (starting)   127.0.0.1:5045->5045/tcp
malcolm-freq-1                "/usr/local/bin/dock…"   freq                running (starting)   10004/tcp
malcolm-htadmin-1             "/usr/local/bin/dock…"   htadmin             running (starting)   80/tcp
malcolm-logstash-1            "/usr/local/bin/dock…"   logstash            running (starting)   127.0.0.1:5044->5044/tcp
malcolm-name-map-ui-1         "/usr/local/bin/dock…"   name-map-ui         running (starting)   8080/tcp
malcolm-nginx-proxy-1         "/usr/local/bin/dock…"   nginx-proxy         running (starting)   0.0.0.0:443->443/tcp, 0.0.0.0:488->488/tcp, 127.0.0.1:5601->5601/tcp, 127.0.0.1:9200->9200/tcp
malcolm-opensearch-1          "/usr/local/bin/dock…"   opensearch          running (starting)   9650/tcp
malcolm-pcap-capture-1        "/usr/local/bin/dock…"   pcap-capture        running              
malcolm-pcap-monitor-1        "/usr/local/bin/dock…"   pcap-monitor        running (starting)   30441/tcp
malcolm-suricata-1            "/usr/local/bin/dock…"   suricata            running (starting)   8711/tcp
malcolm-upload-1              "/usr/local/bin/dock…"   upload              running (starting)   127.0.0.1:8022->22/tcp
malcolm-zeek-1                "/usr/local/bin/dock…"   zeek                running (starting) 
....

Press Ctrl+z to run Malcolm in the background. You can always bring it to foreground by just typing fg.

Check the status of Malcolm containers;

docker ps -a
CONTAINER ID   IMAGE                                   COMMAND                  CREATED         STATUS                           PORTS                                                                                            NAMES
1a164697dc9d   malcolmnetsec/nginx-proxy:6.1.0         "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           0.0.0.0:443->443/tcp, 127.0.0.1:5601->5601/tcp, 0.0.0.0:488->488/tcp, 127.0.0.1:9200->9200/tcp   malcolm-nginx-proxy-1
d94fdbdc5edc   malcolmnetsec/filebeat-oss:6.1.0        "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           127.0.0.1:5045->5045/tcp                                                                         malcolm-filebeat-1
41a1664e9863   malcolmnetsec/suricata:6.1.0            "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           8711/tcp                                                                                         malcolm-suricata-1
18ddca5a001a   malcolmnetsec/dashboards:6.1.0          "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           5601/tcp                                                                                         malcolm-dashboards-1
6cc30e2dc8ea   malcolmnetsec/file-upload:6.1.0         "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           80/tcp, 127.0.0.1:8022->22/tcp                                                                   malcolm-upload-1
21de5caca01e   malcolmnetsec/zeek:6.1.0                "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)                                                                                                            malcolm-zeek-1
c7d9b3da48e8   malcolmnetsec/dashboards-helper:6.1.0   "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           28991/tcp                                                                                        malcolm-dashboards-helper-1
d41224f2d9fc   malcolmnetsec/logstash-oss:6.1.0        "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           9001/tcp, 127.0.0.1:5044->5044/tcp, 9600/tcp                                                     malcolm-logstash-1
3de2293aac1b   malcolmnetsec/arkime:6.1.0              "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           8000/tcp, 8005/tcp, 8081/tcp                                                                     malcolm-arkime-1
286a51eff770   malcolmnetsec/pcap-monitor:6.1.0        "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           30441/tcp                                                                                        malcolm-pcap-monitor-1
9372ec08a576   malcolmnetsec/file-monitor:6.1.0        "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           3310/tcp, 8440/tcp                                                                               malcolm-file-monitor-1
13cb9b77965d   malcolmnetsec/api:6.1.0                 "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           5000/tcp                                                                                         malcolm-api-1
dd079184f941   malcolmnetsec/pcap-capture:6.1.0        "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes                                                                                                                      malcolm-pcap-capture-1
a00f79a8e1b9   malcolmnetsec/htadmin:6.1.0             "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           80/tcp                                                                                           malcolm-htadmin-1
17750ddd1a72   malcolmnetsec/name-map-ui:6.1.0         "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           8080/tcp                                                                                         malcolm-name-map-ui-1
add53463e7f8   malcolmnetsec/opensearch:6.1.0          "/usr/local/bin/dock…"   9 minutes ago   Up 1 second (health: starting)   9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp                                                           malcolm-opensearch-1
b84ee0a208a8   malcolmnetsec/freq:6.1.0                "/usr/local/bin/dock…"   9 minutes ago   Up 9 minutes (healthy)           10004/tcp                                                                                        malcolm-freq-1

Quite a number of ports have been exposed;

sudo ss -atlnp | grep -i docker
LISTEN 0      65535        0.0.0.0:488       0.0.0.0:*    users:(("docker-proxy",pid=8971,fd=4))   
LISTEN 0      65535      127.0.0.1:9200      0.0.0.0:*    users:(("docker-proxy",pid=8833,fd=4))   
LISTEN 0      65535      127.0.0.1:5044      0.0.0.0:*    users:(("docker-proxy",pid=8307,fd=4))   
LISTEN 0      65535      127.0.0.1:5045      0.0.0.0:*    users:(("docker-proxy",pid=6610,fd=4))   
LISTEN 0      65535      127.0.0.1:8022      0.0.0.0:*    users:(("docker-proxy",pid=6969,fd=4))   
LISTEN 0      65535        0.0.0.0:443       0.0.0.0:*    users:(("docker-proxy",pid=9030,fd=4))   
LISTEN 0      65535      127.0.0.1:5601      0.0.0.0:*    users:(("docker-proxy",pid=8922,fd=4))

You can therefore access various Malcolm services via browser (You can replace the localhost with your system IP address);

  • Arkime: https://localhost/
  • OpenSearch Dashboards: https://localhost/dashboards/
  • PCAP upload (web): https://localhost/upload/
  • PCAP upload (sftp): sftp://[email protected]:8022/files/
  • Host and subnet name mapping editor: https://localhost/name-map-ui/
  • Account management: https://localhost:488/

Malcolm Arkime Web Interface

install Malcolm network traffic analysis tool on Ubuntu

OpenSearch dashboard

install Malcolm network traffic analysis tool on Ubuntu

Malcolm Capture File and Log Archive Upload page;

install Malcolm network traffic analysis tool on Ubuntu

Host and Network Segment Name Mapping;

install Malcolm network traffic analysis tool on Ubuntu

Account Management;

install Malcolm network traffic analysis tool on Ubuntu

You can now start to analyse your network traffic using Malcolm.

For now, that is just what it takes to install Malcolm network traffic analysis tool on Ubuntu 22.04.

See how to analyze pcap file using Malcolm;

Analyze PCAP Files using Malcolm Network Traffic Analysis tool

Further reading.

Malcolm QuickStart

Other Tutorials

Install Arkime (Moloch) Full Packet Capture tool on Debian 11

Analyze Network Traffic using Zeek

Analyze Network Traffic Using Brim Security

LEAVE A REPLY

Please enter your comment!
Please enter your name here