Follow through this tutorial to learn how to install Malcolm network traffic analysis tool on Ubuntu 22.04. Malcolm is a network traffic analysis tool suite for full packet capture artifacts (PCAP files) and Zeek logs. The PCAP files or Zeek logs can be uploaded to Malcolm via browser, forwarded via the forwarders or can capture live traffic, parses and normalize the traffic for visualization via OpenSearch dashboards or Arkime.
Read more about Malcolm network traffic analysis tool and its features on their page.
Install Malcolm on Ubuntu 22.04
Malcolm provides scripts that can be used to easy deploy them on a Linux system.
There are two ways in which you can obtain the Malcolm installation script.
- Either by downloading the
install.pyand themalcolm_YYYYMMDD_HHNNSS_xxxxxxx.tar.gzfrom the Malcolm Github releases page. - Or by simply cloning Malcolm Github repository.
In this tutorial, we will use the second method to install Malcolm.
System Requirements
Check Malcolm deployment system requirements.
In our environment, these are the system resources we are using;
CPU cores;
sudo dmidecode -t 4 | grep -i 'core count' Core Count: 4RAM;
free -h total used free shared buff/cache available
Mem: 15Gi 14Gi 153Mi 7.0Mi 723Mi 372Mi
Swap: 0B 0B 0B
Disk Space;
df -hT -P /Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 150G 14G 131G 10% /Create Malcolm System Account
Create an account to run Malcolm with;
useradd -m -d /opt/malcolm -s /bin/bash -G sudo malcolmpasswd malcolmGet the ID of the user as this is required while configuring Malcolm below.
id malcolmSample output;
uid=1001(malcolm) gid=1001(malcolm) groups=1001(malcolm),27(sudo)Run system update;
Ensure system package cache is up-to-date.
sudo apt updateInstall Git package on Ubuntu 22.04;
You will need git to clone the Malcolm Github repository
sudo apt install gitClone Malcolm Github Repository
Run the command below to clone Malcolm Github repository;
su - malcolmgit clone https://github.com/idaholab/MalcolmInstall Required Malcolm Packages and Setup System Tunables
Next, change into cloned Malcolm Github repository directory;
cd MalcolmMalcolm installation scripts are located under scripts directory. Thus, to launch the installation, execute the command below.
sudo ./scripts/install.pyMalcolm runs atop Docker. Thus, when the script runs;
- You will be prompted to install required Docker packages. When prompted, accept to proceed with installation.
- Once the installation of Docker packages is done, you are prompted on whether to add a non-root user account to Docker group. And by all means, please add your standard user account into the group!
- Next, install Docker compose.
Installing required packages: ['apache2-utils', 'make', 'openssl', 'python3-dialog']
"docker info" failed, attempt to install Docker? (Y/n): y
Attempt to install Docker using official repositories? (Y/n): y
Installing required packages: ['apt-transport-https', 'ca-certificates', 'curl', 'gnupg-agent', 'software-properties-common']
Installing docker packages: ['docker-ce', 'docker-ce-cli', 'containerd.io']
Installation of docker packages apparently succeeded
Add a non-root user to the "docker" group?: y
Enter user account: kifarunix
Add another non-root user to the "docker" group?: no
"docker-compose version" failed, attempt to install docker-compose? (Y/n): y
Install docker-compose directly from docker github? (Y/n): y
Download and installation of docker-compose apparently succeeded
...
- Next, you will be prompted to update some kernel tunables. Accept for all prompts.
fs.file-max increases allowed maximum for file handles
fs.file-max= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y
fs.inotify.max_user_watches increases allowed maximum for monitored files
fs.inotify.max_user_watches= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y
fs.inotify.max_queued_events increases queue size for monitored files
fs.inotify.max_queued_events= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y
fs.inotify.max_user_instances increases allowed maximum monitor file watchers
fs.inotify.max_user_instances= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y
vm.max_map_count increases allowed maximum for memory segments
vm.max_map_count= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y
net.core.somaxconn increases allowed maximum for socket connections
net.core.somaxconn= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y
vm.swappiness adjusts the preference of the system to swap vs. drop runtime memory pages
vm.swappiness= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y
vm.dirty_background_ratio defines the percentage of system memory fillable with "dirty" pages before flushing
vm.dirty_background_ratio= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y
vm.dirty_ratio defines the maximum percentage of dirty system memory before committing everything
vm.dirty_ratio= appears to be missing from /etc/sysctl.conf, append it? (Y/n): y
/etc/security/limits.d/limits.conf increases the allowed maximums for file handles and memlocked segments
/etc/security/limits.d/limits.conf does not exist, create it? (Y/n): y
- Once that is done, the script will exit.
Run Malcolm System Configurations
Once the install script above completes, proceed to configure Malcolm;
sudo ./scripts/install.py --configureWhen configuration starts;
- Choose a user with which to run Malcolm as. We created malcolm user account above, whose UID and GID is 1001. Hence, say no to user with UID/GID 1000.
Malcolm processes will run as UID 1000 and GID 1000. Is this OK? (Y/n): n- Enter the user ID and group ID of malcolm system account.
Enter user ID (UID) for running non-root Malcolm processes; 1001Enter group ID (GID) for running non-root Malcolm processes; 1001Confirm;
Malcolm processes will run as UID 998 and GID 998. Is this OK? yes- OpenSearch and logstash settings;
Setting 10g for OpenSearch and 3g for Logstash. Is this OK? yesSetting 3 workers for Logstash pipelines. Is this OK? (Y/n): yes- Restart Malcolm when system or Docker restarts;
Restart Malcolm upon system or Docker daemon restart? YesChoose when to restart Malcolm. You have four options here; no, on-failure, always, unless-stopped. Choose the default option, unless-stopped.
- Choose whether to setup Malcolm with HTTPS. We select yes (SSL certificates will be generated at a later step)
- Choose whether Malcolm will run behind any proxy. You are given some options like Caddy, Traefik..We choose No.
- On Default networking, press enter to accept the defaults.
- If prompted whether to use LDAP, select no for now.
- Store OpenSearch index snapshosts locally in /opt/malcolm/Malcom/opensearch-backup? You can choose to accept the default or just enter the new path,
/opt/malcolm/. - Choose to Compress OpenSearch index snapshots (yes)
- When prompted to delete the oldest indices when the database exceeds a certain size choose no to avoid loosing data.
- Automatically analyze all PCAP files with Suricata: yes
- Automatically analyze all PCAP files with Zeek: yes
- Disable reverse DNS lookup locally for source and destination IP addresses in logs.
- Enable hardware vendor OUI lookups for MAC addresses.
- Perform string randomness scoring on some fields: yes
- Expose OpenSearch port to external hosts: no
- Expose Logstash port to external hosts: no
- Forward Logstash logs to external OpenSearch instance: no
- Expose Filebeat TCP port to external hosts: no
- Enable file extraction with Zeek: yes
- choose interesting as the extraction behavior.
- Select quarantine as the file preservation method.
- Scan extracted files/PE files with ClamAV: yes
- Scan extracted files/PE files with Yara: yes
- Scan extracted files/PE files with Capa: yes
- Lookup extracted file hashes with VirusTotal: no
- Download updated scanner signatures periodically: yes
- Should Malcolm capture network traffic to PCAP files: yes
- Specify capture interface(s) (comma-separated) on which Malcolm will use to network traffic: enp0s8
- Capture packets using netsniff-ng? (Y/n): yes
- Capture packets using tcpdump? (y/N): no
- Capture filter (tcpdump-like filter expression; leave blank to capture all traffic) (): You can disable traffic related to Elasticsearch (port 9200), Logstash (5044), Arkime(8005):
not port 9200 and not port 5044 and not port 8005. - Disable capture interface hardware offloading and adjust ring buffer sizes? (y/N): n
Malcolm has now been installed to /home/$USER/Malcolm.
Scripts for starting and stopping Malcolm and changing authentication-related settings can be found in /home/$USER/Malcolm/scripts.
Run system reboot to apply some of the system changes;
sudo systemctl reboot -iCreate Malcolm User Account
Navigate back to the Malcolm Github repository directory;
su - malcolmcd ~/MalcolmRun the authentication setup script;
./scripts/auth_setup- Store administrator username/password for local Malcolm access? yes
- Set administrator password and username.
- (Re)generate self-signed SSL certs for web traffic HTTPS: yes
- (Re)generate self-signed certificates for a remote log forwarder: yes
- Store username/password for forwarding Logstash events to a secondary, external OpenSearch instance: no.
- Store username/password for email alert sender account (see https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#authenticate-sender-account): no
Download Malcolm Docker Images
Next, run the command below to download Malcolm Docker images from the Docker hub.
cd ~/Malcolmdocker-compose pullDepending on your internet speed, this may take a few mins to complete pulling the images.
Once all the required images are downloaded, you can list them as follow;
docker imagesSample output;
REPOSITORY TAG IMAGE ID CREATED SIZE
malcolmnetsec/zeek 6.1.0 f866620ebd58 3 weeks ago 1.29GB
malcolmnetsec/dashboards 6.1.0 13775018c809 3 weeks ago 1.08GB
malcolmnetsec/arkime 6.1.0 5b7f6018db22 3 weeks ago 794MB
malcolmnetsec/logstash-oss 6.1.0 a8b93b43dbb3 3 weeks ago 1.46GB
malcolmnetsec/suricata 6.1.0 fa25601e29c4 3 weeks ago 285MB
malcolmnetsec/filebeat-oss 6.1.0 b4d1c83bf3de 3 weeks ago 648MB
malcolmnetsec/file-monitor 6.1.0 fa9438234b42 3 weeks ago 589MB
malcolmnetsec/nginx-proxy 6.1.0 bb9720d9d456 3 weeks ago 124MB
malcolmnetsec/file-upload 6.1.0 fdbd62ce5a92 3 weeks ago 259MB
malcolmnetsec/htadmin 6.1.0 623e193fd419 3 weeks ago 242MB
malcolmnetsec/opensearch 6.1.0 a1bbcaae7647 3 weeks ago 1.21GB
malcolmnetsec/pcap-monitor 6.1.0 0e0cb91f32a1 3 weeks ago 214MB
malcolmnetsec/api 6.1.0 54a99d983b37 3 weeks ago 173MB
malcolmnetsec/pcap-capture 6.1.0 9a1664dc488f 3 weeks ago 121MB
malcolmnetsec/freq 6.1.0 e4691604218f 3 weeks ago 131MB
malcolmnetsec/dashboards-helper 6.1.0 ef14087c003d 3 weeks ago 169MB
malcolmnetsec/name-map-ui 6.1.0 90f49b94bfe3 3 weeks ago 123MB
Running Malcolm Service
You can now start Malcolm services using the startup script, ./scripts/start.
The script can be used to start, stop, restart, wipe Malcolm data, etc.
./scripts/start --help
usage: start
Malcolm control script
options:
-v [DEBUG], --verbose [DEBUG]
Verbose output
-f , --file
docker-compose YML file
-s , --service
docker-compose service (only for status and logs operations)
-l [CMDLOGS], --logs [CMDLOGS]
Tail Malcolm logs
--start [CMDSTART] Start Malcolm
--restart [CMDRESTART]
Stop and restart Malcolm
--stop [CMDSTOP] Stop Malcolm
--wipe [CMDWIPE] Stop Malcolm and delete all data
--auth [CMDAUTHSETUP]
Configure Malcolm authentication
--status [CMDSTATUS] Display status of Malcolm components
You can simply start Malcolm by executing the start script with no arguments.
./scripts/startIt will take a few minutes to start all necessary Malcolm services.
The start script will launch Malcolm in the foreground and populate debug messages to the console.
Started Malcolm
In a few minutes, Malcolm services will be accessible via the following URLs:
------------------------------------------------------------------------------
- Arkime: https://localhost/
- OpenSearch Dashboards: https://localhost/dashboards/
- PCAP upload (web): https://localhost/upload/
- PCAP upload (sftp): sftp://[email protected]:8022/files/
- Host and subnet name mapping editor: https://localhost/name-map-ui/
- Account management: https://localhost:488/
NAME COMMAND SERVICE STATUS PORTS
malcolm-api-1 "/usr/local/bin/dock…" api running (starting) 5000/tcp
malcolm-arkime-1 "/usr/local/bin/dock…" arkime running (starting) 8081/tcp
malcolm-dashboards-1 "/usr/local/bin/dock…" dashboards running (starting) 5601/tcp
malcolm-dashboards-helper-1 "/usr/local/bin/dock…" dashboards-helper running (starting) 28991/tcp
malcolm-file-monitor-1 "/usr/local/bin/dock…" file-monitor running (starting) 8440/tcp
malcolm-filebeat-1 "/usr/local/bin/dock…" filebeat running (starting) 127.0.0.1:5045->5045/tcp
malcolm-freq-1 "/usr/local/bin/dock…" freq running (starting) 10004/tcp
malcolm-htadmin-1 "/usr/local/bin/dock…" htadmin running (starting) 80/tcp
malcolm-logstash-1 "/usr/local/bin/dock…" logstash running (starting) 127.0.0.1:5044->5044/tcp
malcolm-name-map-ui-1 "/usr/local/bin/dock…" name-map-ui running (starting) 8080/tcp
malcolm-nginx-proxy-1 "/usr/local/bin/dock…" nginx-proxy running (starting) 0.0.0.0:443->443/tcp, 0.0.0.0:488->488/tcp, 127.0.0.1:5601->5601/tcp, 127.0.0.1:9200->9200/tcp
malcolm-opensearch-1 "/usr/local/bin/dock…" opensearch running (starting) 9650/tcp
malcolm-pcap-capture-1 "/usr/local/bin/dock…" pcap-capture running
malcolm-pcap-monitor-1 "/usr/local/bin/dock…" pcap-monitor running (starting) 30441/tcp
malcolm-suricata-1 "/usr/local/bin/dock…" suricata running (starting) 8711/tcp
malcolm-upload-1 "/usr/local/bin/dock…" upload running (starting) 127.0.0.1:8022->22/tcp
malcolm-zeek-1 "/usr/local/bin/dock…" zeek running (starting)
....
Press Ctrl+z to run Malcolm in the background. You can always bring it to foreground by just typing fg.
Check the status of Malcolm containers;
docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1a164697dc9d malcolmnetsec/nginx-proxy:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 0.0.0.0:443->443/tcp, 127.0.0.1:5601->5601/tcp, 0.0.0.0:488->488/tcp, 127.0.0.1:9200->9200/tcp malcolm-nginx-proxy-1
d94fdbdc5edc malcolmnetsec/filebeat-oss:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 127.0.0.1:5045->5045/tcp malcolm-filebeat-1
41a1664e9863 malcolmnetsec/suricata:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 8711/tcp malcolm-suricata-1
18ddca5a001a malcolmnetsec/dashboards:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 5601/tcp malcolm-dashboards-1
6cc30e2dc8ea malcolmnetsec/file-upload:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 80/tcp, 127.0.0.1:8022->22/tcp malcolm-upload-1
21de5caca01e malcolmnetsec/zeek:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) malcolm-zeek-1
c7d9b3da48e8 malcolmnetsec/dashboards-helper:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 28991/tcp malcolm-dashboards-helper-1
d41224f2d9fc malcolmnetsec/logstash-oss:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 9001/tcp, 127.0.0.1:5044->5044/tcp, 9600/tcp malcolm-logstash-1
3de2293aac1b malcolmnetsec/arkime:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 8000/tcp, 8005/tcp, 8081/tcp malcolm-arkime-1
286a51eff770 malcolmnetsec/pcap-monitor:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 30441/tcp malcolm-pcap-monitor-1
9372ec08a576 malcolmnetsec/file-monitor:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 3310/tcp, 8440/tcp malcolm-file-monitor-1
13cb9b77965d malcolmnetsec/api:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 5000/tcp malcolm-api-1
dd079184f941 malcolmnetsec/pcap-capture:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes malcolm-pcap-capture-1
a00f79a8e1b9 malcolmnetsec/htadmin:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 80/tcp malcolm-htadmin-1
17750ddd1a72 malcolmnetsec/name-map-ui:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 8080/tcp malcolm-name-map-ui-1
add53463e7f8 malcolmnetsec/opensearch:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 1 second (health: starting) 9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp malcolm-opensearch-1
b84ee0a208a8 malcolmnetsec/freq:6.1.0 "/usr/local/bin/dock…" 9 minutes ago Up 9 minutes (healthy) 10004/tcp malcolm-freq-1
Quite a number of ports have been exposed;
sudo ss -atlnp | grep -i dockerLISTEN 0 65535 0.0.0.0:488 0.0.0.0:* users:(("docker-proxy",pid=8971,fd=4))
LISTEN 0 65535 127.0.0.1:9200 0.0.0.0:* users:(("docker-proxy",pid=8833,fd=4))
LISTEN 0 65535 127.0.0.1:5044 0.0.0.0:* users:(("docker-proxy",pid=8307,fd=4))
LISTEN 0 65535 127.0.0.1:5045 0.0.0.0:* users:(("docker-proxy",pid=6610,fd=4))
LISTEN 0 65535 127.0.0.1:8022 0.0.0.0:* users:(("docker-proxy",pid=6969,fd=4))
LISTEN 0 65535 0.0.0.0:443 0.0.0.0:* users:(("docker-proxy",pid=9030,fd=4))
LISTEN 0 65535 127.0.0.1:5601 0.0.0.0:* users:(("docker-proxy",pid=8922,fd=4))
You can therefore access various Malcolm services via browser (You can replace the localhost with your system IP address);
- Arkime: https://localhost/
- OpenSearch Dashboards: https://localhost/dashboards/
- PCAP upload (web): https://localhost/upload/
- PCAP upload (sftp): sftp://[email protected]:8022/files/
- Host and subnet name mapping editor: https://localhost/name-map-ui/
- Account management: https://localhost:488/
Malcolm Arkime Web Interface
OpenSearch dashboard
Malcolm Capture File and Log Archive Upload page;
Host and Network Segment Name Mapping;
Account Management;
You can now start to analyse your network traffic using Malcolm.
For now, that is just what it takes to install Malcolm network traffic analysis tool on Ubuntu 22.04.
See how to analyze pcap file using Malcolm;
Analyze PCAP Files using Malcolm Network Traffic Analysis tool
Further reading.
Other Tutorials
Install Arkime (Moloch) Full Packet Capture tool on Debian 11
