You don't know what you don't know - Inventory

With all the Log4J madness happening over the last few weeks it got me thinking ..... (https://www.cisa.gov/uscert/apache-log4j-vulnerability-guidance)  Here are a few questions you should be kicking around with your OPS/Security teams when the dust settles.  But don’t wait too long after an incident, you want the wound to be fresh so you get the details.   People forget over time, especially things they do not want to remember. 

• 
  
• 
  
• How do we know if we are affected?
  • Do we even run the product/library/application that is affected by the vulnerability?   Where do you start?  It all begins with INVENTORY.  Over the many years of the cyber industry one thing still remains: we are horrible at inventory and tracking of assets.  If you do not have a centralized asset list or inventory, start today!  Open a spreadsheet and start filling in rows with asset details;   name, ip, OS version, Patch info for OS, Applications running on Asset with Version Numbers.  This is a small list and can be made as complicated/detailed as you want.  I know, most folks are saying a spreadsheet?  Why a spreadsheet?  The point isn’t the spreadsheet, the point is you have a list of assets.  Most of the time it isn’t the medium you are capturing the data, it is the process around the inventory that collapses.  After creating the list, share the list in a centralized location where multiple ‘approved’ folks can update the list.  Bake it into the company's psyche.  Onboarding/offboarding of assets, update the inventory list.   Patches to inventory, update the asset list.   Most companies have that ‘running’ inventory file that when you look at it you find multiple assets that haven’t been living on the network for years..   
  • This is where a centralized CMDB or Asset Database might be helpful using a vendor tool but without the basics you will still need to have a basic list of assets.  When the vendor comes in to plan the implementation one of the first questions they will ask is ‘ do you have a list of assets/CIs?
• How do we know ‘what’ is affected?
  • Most vulnerabilities relate to a version or library within a product/application/service it is important to capture the version numbers of applications running on your assets so you can narrow down what is affected.  The building blocks to your inventory start with an asset/device from there start filling in the application list and version numbers.  It does feel like a daunting task when starting from nothing but remember, every little piece of information you add to your inventory list will save you hours of pain later or during an actual incident when you are scrambling for the information.   
  • This is where a centralized CMDB with the version details is helpful or an Asset Inventory/Patching system.  Most Vulnerability scanners today now create a list of Assets and associated application versions as well but might not be your central inventory list and you will need to integrate that platform with your inventory of record.  Most vendor tools are providing integrations between these types of systems and updating/adding to your central inventory has gotten a lot easier then yesteryear.   When doing a  review of your current systems make sure to lay out/document the details around each platform, the use, data sets/information that it provides, and how this could be useful within other platforms when trying to answer a question.  If all the data is collected and integrated together it becomes a lot easier to narrow down what asset might be affected by a vulnerability or in answering other questions like: 
    • What system is not patched?
    • What system is running unapproved software/applications? 
• How do we remediate what is affected?
  • Answering this question needs to happen far before an incident is happening.  Your organization needs to have a process/SoP/Plan documented on the incident response and patching procedures with different use cases/scenarios and actions different teams in the organization will take to remediate the issue and how communications will happen. How will you remediate?  Patch, apply  workaround, network configuration change, pull device from the network.  This scenario can play out like a bad ‘pick your own adventure’ book if you haven’t thought through some of these scenarios and understand what is needed for each.  Who is involved?  How will we track changes during the incident?  How will we communicate during the incident?  Does the severity of the vulnerability affected roll out?  How does that affect the Change Management Process?   What does an emergency Change Request encompass when dealing with the situation?  When  do we go ‘back’ to using “normal” changes for an asset?   How will the company handle the post incident review and incorporate lessons learned?  Evolution of process/SoPs is important and it all starts with having the basic block in place to add to it.   Most of the time the block is a first cut at a document to capture the process.  It doesn’t have to be perfect, it won’t be as long as you refer to it and improve upon it. 
• How do we protect against the vulnerability if we cannot remediate?
  • Patch, work around, network config change, pull from network?  Really depends on a lot of factors:     
    • What is affected?
    • What systems?  Are they mission critical?   Are they internet facing?  Do they have security controls around them that already loosen the vulnerability criticality 
• How do we check for this being exploited prior to knowledge of vulnerability? 
  • Need to have that historical look into your network and assets.  What has happened to them? Who has talked/communicated with them?  Did they take actions not normal to their behavior? 
  • Logging of assets will go a long way here in helping you understand if you were impacted by a vulnerability. Yes, logging sucks, setting up the data pipeline, collecting logs, setting up monitoring of logs (alerts/triggers), etc.   It’s a lot of work but if you build a repeatable process that is implemented into other processes (onboarding/offboarding) the burden gets easier every time you do it.  One major area that isn’t done enough is log analysis and learning the logs.  There are TONS of log types out there and most products don’t stick to a universal log format so you’ll need to roll up your sleeves and learn the specifics of the log types, event types, fields associated with log/event types, what types of events are logged by the product/app/service/etc…  Take some time here do high level queries against the data to start from the 50k foot view and drill into the data.  Start with high level count queries against the log type(event type) fields.  
    • COUNT, ACTION FIELD
      • (24987, Deny;   300009, Accept;  23421, Reject)
    • COUNT, DISINCT(CNT((SrcIP), ACTION FIELD, order by COUNT
    • Look for patterns in event type events.  Are there event types that should never appear in logging?  If so, might be a good alert to setup in case they do start showing up in the logs. 
• How do we shorten the above?
  • Automation…  Looking at each of the above, the process around it and breaking out a game plan on how you can take a manual process and automated it with a tool/script/platform.
    • Any piece of a manual process that can be shortened with the help of a computer should be looked at.  Does it make sense to automate it?  Where do you start?
    • Do you integrate this workflow into a SIEM, Ticketing platform, Automation Platform? Ideally some of the tools you are running in your environment already handle some of these tasks and you will just need to work on enabling plugins/add-ons to create the connections and communications between systems.   Lean on your vendor for assistance in this area, they should have ample documentation to get you started and if there is a custom plugin you need to discuss it with them to get support in place.  Last thing you want to be doing is managing a plugin and trying to keep up with all the API updates/changes done by a vendor. 
• How do we onboard devices/services/assets with auto tracking/inventory built in?
  • This question builds off the Automation response above.   Look at your basic SoPs that are done for a majority of the employees in your organization.  From onboarding of the employee to hardware assignment to MFA and security controls.  Anything you can automate within this process will save you time/effort 3 fold.  
  • There should be a repeatable process when it comes to server build out as well so identify where you can start automating.    Gold image server build, call to inventory system to add asset to CMDB, update inventory with applications, etc. 
• How do we query/dashboard using the above data?   
  • If a new vulnerability is disclosed can you build reports or ‘close to’ real time dashboards showing your current exposure?   
    • Do you have the data to do this?
    • Do the tools you use have the ability to do this?
    • How easy is it to create these reports/dashboards?  Do you need a trained person to whip out these reports or is the tool easy enough to use that in a time of need someone could go in and create a report? 

Once you start getting a handle on the above..  You will need to ask the same questions around your vendors, supply chain, and partners.   It never ends…

Supply chain, vendors, partners: 

• How do we check vendors for vulnerabilities? 
• How do we check our supply chain for vulnerabilities? 
• How do we get the above details without the wait?
• How do these parties notify you of the vulnerabilities?  How do they communicate remediation/patching/etc? 

That’s all for now. 
Bartlett


​

---