Advanced log collector management (Microsoft)

This article provides information about the following advanced configuration options for Defender for Cloud Apps Cloud Discovery log collectors:

• Modify the log collector FTP configuration
• Enable the log collector behind a proxy
• Move the log collector to a different data partition on Linux
• Inspect the log collector disk usage on Linux
• Move the log collector to an accessible host
• Define custom ports for Syslog and FTP receivers for log collectors on Linux
• Validate the traffic and log format received by log collector on Linux

Modify the log collector FTP configuration

Use these steps to modify the configuration for your Defender for Cloud Apps Cloud Discovery Docker.

Docker deployment

You might need to modify the configuration for the Defender for Cloud Apps Cloud Discovery Docker.

Changing the FTP password

1. Connect to the log collector host.
2. Run docker exec -it <collector name> pure-pw passwd <ftp user>
   1. Enter the new password.
   2. Enter the new password again for confirmation.
3. Run docker exec -it <collector name> pure-pw mkdb to apply the change.

   

Customize certificate files

Follow this procedure to customize the certificate files you use for secure connections to the Cloud Discovery Docker.

1. Open an FTP client and connect to the log collector.

   

2. Navigate to the ssl_update directory.
3. Upload new certificate files to the ssl_update directory (the names are mandatory).

   

   • For FTP: Only one file is required. The file has the key and certificate data, in that order, and is named pure-ftpd.pem.
   • For Syslog: Three files are required: ca.pem, **server-key.pem, and server-cert.pem. If any of the files are missing, the update won’t take place.
4. In a terminal window run: docker exec -t <collector name> update_certs. The command should produce a similar output to what’s seen in the following screenshot.

   

5. In a terminal window run: docker exec <collector name> chmod -R 700 /etc/ssl/private/.

Enable the log collector behind a proxy

After you configured the log collector, if you are running behind a proxy, the log collector might have trouble sending data to Defender for Cloud Apps. This may happen because the log collector doesn’t trust the proxy’s root certificate authority and is not able to connect to Microsoft Defender for Cloud Apps to retrieve its configuration or upload the received logs.

Use these steps to enable your log collector behind a proxy.

Set up the log collector behind a proxy

Make sure you performed the necessary steps run Docker on a Windows or Linux machine and successfully download the Defender for Cloud Apps Docker image on the machine. For more information, see Configure automatic log upload for continuous reports.

Validate Docker log collector container creation

In the shell, verify that the container was created and is running using the following command:

docker ps

Copy proxy root CA certificate to the container

From your virtual machine, copy the CA certificate to the Defender for Cloud Apps container. In the following example, the container is named Ubuntu-LogCollector and the CA certificate is named Proxy-CA.crt. Run the command on the Ubuntu host. It copies the certificate to a folder in the running container:

docker cp Proxy-CA.crt Ubuntu-LogCollector:/var/adallom/ftp/discovery

Set the configuration to work with the CA certificate

1. Go into the container, using the following command. It will open bash in the log collector container:
   BashCopy

   docker exec -it Ubuntu-LogCollector /bin/bash
   

2. From a bash window inside the container, go to the Java jre folder. To avoid a version-related path error, use this command:
   BashCopy

   cd "$(find /opt/jdk/*/jre -name "bin" -printf '%h' -quit)"
   cd bin
   

3. Import the root certificate that you copied earlier, from the discovery folder into the Java KeyStore and define a password. The default password is “changeit”. For information about changing the password, see How to change the Java KeyStore password.
   BashCopy

   ./keytool --import --noprompt --trustcacerts --alias SelfSignedCert --file /var/adallom/ftp/discovery/Proxy-CA.crt --keystore ../lib/security/cacerts --storepass <password>
   

4. Validate that the certificate was imported correctly into the CA keystore, by using the following command to search for the alias you provided during the import (SelfSignedCert):
   BashCopy

   ./keytool --list --keystore ../lib/security/cacerts | grep self
   

   

You should see your imported proxy CA certificate.

Set the log collector to run with the new configuration

The container is now ready.

Run the collector_config command using the API token that you used during the creation of your log collector:

When you run the command, specify your own API token:

collector_config abcd1234abcd1234abcd1234abcd1234 ${CONSOLE} ${COLLECTOR}

The log collector is now able to communicate with Defender for Cloud Apps. After sending data to it, the status will change from Healthy to Connected in the Defender for Cloud Apps portal.

How to change the Java KeyStore password

1. Stop the Java KeyStore server.

1. Open a bash shell inside the container and go to the appdata/conf folder.
2. Change the server KeyStore password by using this command:
   BashCopy

   keytool -storepasswd -new newStorePassword -keystore server.keystore
   -storepass changeit
   

    Note

   The default server password is changeit.

3. Change the certificate password by using this command:
   BashCopy

   keytool -keypasswd -alias server -keypass changeit -new newKeyPassword -keystore server.keystore -storepass newStorePassword
   

    Note

   The default server alias is server.

4. In a text editor, open the server-install\conf\server\secured-installed.properties file, and then add the following lines of code, and then save the changes:
   1. Specify the new Java KeyStore password for the server: server.keystore.password=newStorePassword
   2. Specify the new Certificate password for the server: server.key.password=newKeyPassword
5. Start the server.

Move the log collector to a different data partition on Linux

Many companies have the requirement to move data to a separate partition. Use these steps to move your Defender for Cloud Apps Docker log collector images to a data partition on your Linux host.

The following steps describe moving data to a partition called datastore and assumes you have already mounted the partition.

1. Stop the Docker service by using this command:
   BashCopy

   service docker stop
   

2. Move the log collector data to the new partition by using this command:
   BashCopy

   mv /var/lib/docker /datastore/docker
   

3. Remove the old Docker storage directory (/var/lib/docker) and create a symbolic link to the new directory (/datastore/docker).
   BashCopy

   rm -rf /var/lib/docker && ln -s /datastore/docker /var/lib/
   

4. Start the Docker service by using this command:
   BashCopy

   service docker start
   

5. Optionally verify the status of your log collector by using this command:
   BashCopy

   docker ps
   

Inspect the log collector disk usage on Linux

Use these steps to review your log collector disk usage and location.

1. Identify the path to the directory where the log collector data is stored by using this command:
   BashCopy

   docker inspect <collector_name> | grep WorkDir
   

   

2. Get the size on disk of the log collector using the identified path without the “/work” suffix:
   BashCopy

   du -sh /var/lib/docker/overlay2/<log_collector_id>/
   

   

    Note

   If you only need to know the size on disk, you can use this command: docker ps -s

Move the log collector to an accessible host

In regulated environments, access to Docker Hubs where the log collector image is hosted may be blocked. This prevents Defender for Cloud Apps from importing the data from the log collector and can be resolved my moving the log collector image to an accessible host.

Use these steps to download the log collector image using a computer that has access to Docker Hub and import it to your destination host.

Start the process by exporting the log collector image and then import the image to your destination host.

Export the log collector image from your Docker Hub

Use the steps relevant to the operating system of the Docker Hub where the log collector image is located.

Exporting the image on Linux

1. On a Linux computer that has access to the Docker Hub, run the following command. This will install Docker and download the log collector image.
   BashCopy

   curl -o /tmp/MCASInstallDocker.sh https://adaprodconsole.blob.core.windows.net/public-files/MCASInstallDocker.sh && chmod +x /tmp/MCASInstallDocker.sh; /tmp/MCASInstallDocker.sh
   

2. Export the log collector image.
   BashCopy

   docker save --output /tmp/mcasLC.targ mcr.microsoft.com/mcas/logcollector
   chmod +r /tmp/mcasLC.tar
   

    Note

   It’s important to use the output parameter to write to a file, instead of STDOUT.

3. Download the log collector image to your Windows computer under C:\mcasLogCollector\ using WinSCP.

   

Exporting the image on Windows

1. On a Windows 10 computer that has access to the Docker Hub, install Docker Desktop.
2. Download the log collector image.
   cmdCopy

   docker login -u caslogcollector -p C0llector3nthusiast
   docker pull mcr.microsoft/mcas/logcollector
   

3. Export the log collector image.
   cmdCopy

   docker save --output C:\mcasLogCollector\mcasLC.targ mcr.microsoft.com/mcas/logcollector
   

    Note

   It’s important to use the output parameter to write to a file, instead of STDOUT.

Import and load the log collector image to your destination host

Use these steps to transfer the exported image to your destination host.

1. Upload the log collector image to your destination host under /tmp/.

   

2. On the destination host, import the log collector image to the Docker images repository by using this command:
   BashCopy

   docker load --input /tmp/mcasLC.tar
   

   

3. Optionally, verify that the import completed successfully by using this command:
   BashCopy

   docker image ls
   

   

   You can now proceed to create your log collector using the image from the destination host.

Define custom ports for Syslog and FTP receivers for log collectors on Linux

Some organizations have a requirement to define custom ports for Syslog and FTP services. When adding a data source, Defender for Cloud Apps log collectors uses specific port numbers to listen for traffic logs from one or more data sources.

The following table lists of the default listening ports for receivers:

Use these steps to define custom ports.

1. In Defender for Cloud Apps, click the settings icon followed by Log collectors.
2. On the Log collectors tab, add or edit a log collector and after updating the data sources, copy the run command from the dialog.

   

    Note

   If used as provided, the following wizard provided command configures the log collector to use ports 514/udp and 515/udp.

   BashCopy

   (echo <credentials>) | docker run --name LogCollector1 -p 514:514/udp -p 515:515/udp -p 21:21 -p 20000-20099:20000-20099 -e "PUBLICIP='10.0.0.100'" -e "PROXY=" -e "SYSLOG=true" -e "CONSOLE=machine.us2.portal.cloudappsecurity.com" -e "COLLECTOR=LogCollector1" --security-opt apparmor:unconfined --cap-add=SYS_ADMIN --restart unless-stopped -a stdin -i mcr.microsoft.com/mcas/logcollector starter
   

   

3. Before using the command on your host machine, modify the command to use your custom ports. For example, to configure the log collector to use UDP ports 414 and 415, change the command as follows:
   BashCopy

   (echo <credentials>) | docker run --name LogCollector1 -p 414:514/udp -p 415:515/udp -p 21:21 -p 20000-20099:20000-20099 -e "PUBLICIP='10.0.0.100'" -e "PROXY=" -e "SYSLOG=true" -e "CONSOLE=machine.us2.portal.cloudappsecurity.com" -e "COLLECTOR=LogCollector1" --security-opt apparmor:unconfined --cap-add=SYS_ADMIN --restart unless-stopped -a stdin -i mcr.microsoft.com/mcas/logcollector starter
   

   

    Note

   Only the Docker mapping is modified. The internally assigned ports are not changed enabling you to choose any listening port on the host.

Validate the traffic and log format received by log collector on Linux

Occasionally, you may need to investigate issues such as the following:

• Log collectors are receiving data: Validate that log collectors are receiving Syslog messages from your appliances and are not blocked by firewalls.
• Received data is in the correct log format: Validate the log format to help you troubleshoot parsing errors by comparing the log format expected by Defender for Cloud Apps and the one sent by your appliance.

Use these steps to validate the traffic received by log collectors.

1. Sign in to your server hosting the Docker container.
2. Validate that the log collector is receiving Syslog messages using any of the following methods:
   • By using tcpdump, or similar command to analyze network traffic on port 514:
     BashCopy

     tcpdump -Als0 port 514
     

     If everything is correctly configured, you should see network traffic from your appliances.

     

   • By using netcat, or similar command to analyze network traffic on the host machine:
     1. Install netcat and wget.
     2. Download, and if required unzip, a sample log, as follows:
        1. In the Defender for Cloud Apps portal, click Discover, and then click Create snapshot report.
        2. Select the Data source from which you want to upload the log files.
        3. Click View and verify then right-click Download sample log and copy the URL address link.
        4. Click Close.
        5. Click Cancel.
     BashCopy

     wget <URL_address_to_sample_log>
     

     1. Run netcat to stream the data to the log-collector.
     BashCopy

     cat <path_to_downloaded_sample_log>.log | nc -w 0 localhost <datasource_port>
     

     If the collector is correctly configured, the log data will be present in the messages file and shortly after that it will be uploaded to the Defender for Cloud Apps portal.

   • By inspecting relevant files within the Defender for Cloud Apps Docker container:
     1. Log in to the container by using this command:
     BashCopy

     docker exec -it <Container Name> bash
     

     1. Determine if Syslog messages are being written to the messages file by using this command:
     BashCopy

     cat /var/adallom/syslog/<your_log_collector_port>/messages
     

     If everything is correctly configured, you should see network traffic from your appliances.

      Note

     This file will continue to be written to until it reaches 40 KB in size.

     

3. Review logs that have been uploaded to Defender for Cloud Apps in the /var/adallom/discoverylogsbackup directory.

   

4. Validate the log format received by the log collector by comparing the messages stored in /var/adallom/discoverylogsbackup to the sample log format provided in the Defender for Cloud Apps Create log collector wizard.

docker exec CustomerLogCollectorName tail -f -q /var/adallom/syslog/<datasource_port>/messages > /tmp/log.log

Source : Official Microsoft Brand
Editor by : BEST Antivirus KBS Team