FMP score

Future Misbehavior Probability (FMP) score

FMP score (formerly reputation score) is the estimated probability, that a given IP address (or other entity) will be reported as malicious in a given future time interval. The estimation is done using a machine learning model and should use as much data available about the entity as possible. The prediction interval is normally set to the next 24 hours.

The score may be computed as the probability of any alert or only alert of a specific type (e.g. FMP_scan as probability of scanning alert).

TODO: which FMP scores should NERD compute? (probably scan, access and "any", we don't have enough data for other categories)

Currently, FMP score is computed for IP addresses only.

Module computing the score

The score is computed by a plug-in module for NERD worker and stored as an attribute in the entity record.

The module computes and updates all the features needed by the ML model, and periodically uses the model (trained "offline" and stored in a file) to compute the current score.

For now, let's recompute the feature vector and the FMP score once a day, i.e. when !every1d event is received.

Features

The follwing features should be computed for each IP address from the data available in its record (based on the (still unpublished) paper). If necessary, new auxiliary attributes may be stored in the record (with a name prefixed by _, so they are not shown in the web interface).

Note: For now, let's implement only the ones in bold font.

• Number of alerts in the last day.

  • Ideally exactly from 24 hours before the current time, but NERD only stores number of alerts per day, so it won't work in the middle of the day. So, let's use the number of alerts in the current day + in the previous day.

• Total number of connection attempts (attack volume) in the last day.

  • For the time range, see the note above.
  • Number of connections should be present at least in all Recon.Scanning and Attempt.Login alerts, as ConnCount or FlowCount field. If it's not present, use 1.
  • Number of connections is currently not stored in entity records - this needs to be added. The best place is probably the events key, just add a new subkey (e.g. vol) next to n (see line 264 in warden_receiver.py; and see Attributes and Data model for documentation of the data format used)

• Number of detectors reporting the address in the last day

  • For the time range, see the note above.
  • This should be easy to get from the events tag.

• Number of alerts in the last week.

• Total number of connection attempts (attack volume) in the last week.

• Number of detectors reporting the address in the last week

  • All as above, just use data from the current day + 7 previous days.

• EWMA of number of alerts per day

  • Exponentially Weighted Moving Average, set alpha to 0.25.
  • There are at least two methods to compute this:
    • Every time the feature needs to be computed take number of alerts in each day (in e.g. last 30 days), multiply them by weights according to EWMA and sum them up.
    • There is a way to compute EWMA continuously - every time a new alert is added (hook a hanldler function to last_event_time), EWMA value is recomputed as EWMA_new = alpha*time*1 + (1 - alpha*time)*EWMA_old, where time is the time between this event and the previous one. The 1 means "1 alert", if we count something else (e.g. their volume), it can be replaced by a correspoding number .

• EWMA of total number of connection attempts per day

  • Same as above, but for attack volume.

• EWMA of a binary signal expressing presence of an alert (0 or 1) in each day

  • Same as above, but instead of number of alerts in a day, use 1 if there was at least one alert, 0 otherwise.

• Time from the last alert (in days)

• Average interval between alerts within the last week

  • In days.
  • Set to infinity if less then two alerts were reported.
  • There is currently no data to derive this from, we'll have to find out some efficinet way to compute this.

• Median interval between alerts within the last week

  • Same as above

• All the above, but for the whole /24 prefix

  • Count alerts of all IP addresses in the same /24 prefix as the current address
  • Currently impossible to get from data available in address record. Probably the best solution will be to introduce a new entity type, /24 prefix, and compute the features there. But this will be a significant amount of work.

• For the /24 prefixes, also compute the number of distinct IP addresses reported in last day and week

• Presence on blacklists

  • One binary (0/1) feature for each of the selected blacklists:
    • tor, blocklist_de_ssh, uceprotect, sorbs-dul, sorbs-noserver, sorbs-spam, spamcop, spamhaus-pbl, spamhaus-pbl-isp, spamhaus-xbl-cbl

• Binary features based on hostname tags

  • A feature with values 0/1 or -1/0/1 using following information derived from hostname (some of it is already resolved by tags.py module and stored in tags key in the records)
  • hostname_exists - whether hostname for the IP address is set or not (look at value of hostname tag in the IP record)
  • dynamic_static - 1 if dynamic tag is set, -1 if static tag, 0 otherwise
  • dsl - 1 if dsl tag is set, 0 otherwise
  • ip_in_hostname - 1 if IP address is encoded in hostname, 0 otherwise (this one is currently not available, it will need to update the tagging module)

ML model

The ML model is based on an ensemble of decision trees, implemented by the xgBoost library. The implementation allows to store and load a trained model to a file. The module therefore just loads the model from the file (path given in configuration), feeds in the computed feature vector and uses the output as FMP score.

TODO: For now, the model is going to be trained manually on offline data samples. In the future, we should find a way to automate this and periodically retrain the model on new data.