Had an unfortunate event at work today.  One of my coworkers deleted the Deployment plan for one of our projects.  Not just one of the deployment plans, but the WHOLE projects deployment plans. We are an Atlassian shop utilizing everything from Confluence(for requirements and documentation), Jira (for issue and project management), Bitbucket (for code repo, peer review, branch management), to Bamboo (for builds/automated testing/deployments).  Bamboo can be a pain at times but it gets the job done and like any CI/CD server it has its quirks and you need to provide it proper care and feeding. 

Well, half way through the morning I receive a message basically saying "O SH**" and that the deployment plans had been deleted on accident.  After a few minutes of fuming it was time to get this thing back up and running.  I contacted our IT team to find out this VM has not been backed up since Sept of last year.  This day kept getting better and better.  I took a little time to run through the deployment plans, build plans, and configurations on the server and document anything and everything I thought would or could be useful after the revert. I was surprised by the lack of options in Bamboo to recover from changes, when it's gone IT'S GONE!

Long story short.  We reverted.   Luckily a lot of the deployment plan which was deleted was in that snapshot but it got me thinking about what we could have done to avoid this problem all together. 

• Backups:
  
  • Verify there is a CONSISTENT backup plan with your IT team.  Consistent being a FULL BACKUP at least every few days and an incremental backup done nightly.
    • Backups can come at many levels.  Understand the difference between a FULL System backup and a backup the application provides.  In our case  Bamboo does have a backup 'option' but if you read the fine print it is not meant to be used in production.  So we are leaning to a full system backup, thankfully VMs make this a VERY easy process, not like yesteryear. 
    • At least every 6 months do a FULL SYSTEM restore from the backup to triple check things will work as expected when the SH** hits the fan.  Include the team on this recovery test, don't allow one member of the team to become the 'backup/recovery' person. 
    • Define which team is responsible for what part of the 'machine'.  If the IT team is responsible for the underlying HOST but the development team is on the hook for the application have that documented and agreed upon.  You can't do these things in times of crisis!
      
    • DOCUMENT THIS PLAN and share it with all related parties!!
• User Permissions
  • Who on your team should have DELETE permissions on anything? Is it necessary?
  • Should DELETE only be given to the Manager or LEAD Developer?  But who watches the 'watchers'
  • Stick to LEAST PRIVILEGE for users no matter who they are and make them follow the process when additional permissions are needed. 
  • Have some level of skills/knowledge 'check' for up and coming developers or team members so you provide necessary training before handing over the keys to the kingdom.  A Junior level developer might not have any idea what the CI/CD server does, so don't allow them to go in as a member of the ADMIN group and make changes. 
• Changes: 
  • Changes are changes whether they are code, tests, build plans, or deployment plans.  They should go through some level of Peer Review.  
  • Added bonus of Peer Review is you get another set of eyes on the change which boosts cross training and relieves some of the SPF (single point of failure) within the team.  
• Training and Documentation: 
  • The more folks know the better the organization will be, train the team(s) on the technology and PROCESS around build/deployments.
  • Keep a high level diagram showing the network, hosts, and communications so explaining the overall process is easy to understand and management of the IPs/hosts is clear. 
  • Keep in mind that your document repo (Confluence) is just another system which can go down.  So if you are relying on that server because it holds all your backup/emergency SOPs you better have a plan B!

Today was rough, but with any issue comes opportunity.  Use the 'lesson' and learn from it so it doesn't happen again.  EVERYONE should walk away from the experience with more knowledge of the tool/process, better skills around the tool, and the confidence that this type of issue will NOT happen again because the team is taking the right steps in the future. 
 
What are some of your worst backup/recovery experiences? 

Bartlett

I updated my All Aboard presentation.  It's my thoughts on project management, vision, goals, and impediments. 

Some thoughts around team management and project management. 

Using small incremental steps to build up to the big project. 

​Happy Fall everyone!

I attended my 5th graders field trip to The EDGE challenge course a few weeks ago.  It was  great experience to see their young minds trying to solve/complete challenges in a group environment. 

• Chaperones were told to NOT HELP the teams
• Young minds are a reflection of their experiences
• Team/Group dynamics
• Communication
• Frustration 
• Lessons Learned/Continuous Improvement

My thoughts on micro-management and other areas of project/product and team management. 

I attend a lunch and learn for Tenable yesterday in Tysons Corner.   The talk covered their TENABLE.io product mostly with small mentions of their other tools/products: Security Center and Nessus. 
Tenable.IO is there newest product.  It has a faster scanning engine and more integration capabilities (web app scanning).
Here is the high level architecture of the tool.  The product provides all of the functionality of the old nessus scanner and has added multiple things: 

• Active Scanner - Like old Nessus product
• Passive Scanner -  Captures network information across the wire to inventory and alert for changes in the environment (between active scans)
• Agent - Install agent on end points for on demand/scheduled VM scans
• API and SDK to start creating integrations with other systems in your environment
• Third-party sources - System is building 'connectors' to allow other systems to send information into the tenable.io product.  (example they used was AWS.  You enter your creds for AWS environment it will pull and load the CloudTrial logs and provide vulnerability information off of that data) 
• Container Security - Plugin for scanner to detect Containers running on servers/assets

How you can utilize this tool: 

• OPS: Start utilizing scanner during on-boarding process of new devices and changes to understand your environments/assets
• OPS/IT:   Utilize the active scanner to keep a baseline of your network/asset.  Use the passive scanner as a way to keep an eye on things in that environment 'between' active scans. 
• DEVELOPMENT:  If you using Docker or any other Container type technology and want to add another layer of security into your Security Lifecycle. Part of the tenable.io suite allows for scanning of Docker Containers, this could be added to your CI/CD server to run scans against containers every time there is a build and add some additional security to your SDLC process. 
• Research/Intel :  Pull output from tenable.io tool and match it with current threat details in your Threat Management Tool (i use Recorded Future to help with this piece).  If done right it will take the vulnerability information and match it against the 'threats/risks' and provide a good guide on where to start with patching and the timeline needed . 

Did a presentation on Bright Talk about Ransomware.  If you want to hear me drop my knowledge you can check it out: 

https://summits.brighttalk.com/webinar/ransomware-prevention-is-the-best-medicine

 RFUN is Recorded Futures yearly conference.  It's a 2 day event.  First day covering training of the Tool and second day was multiple key notes and breakout sessions.  This year it was held at the Newseum in Washington, DC. 

Day 1: 
Multiple training courses on how to use the Tool and it's robust API for integrations/automation and to help with Event Enrichment and Correlation.  Recorded Future has multiple integrations with other tools to allow Analysts to quickly get 'backing/context' information(enrichment) for an IP, Domain, Malware, Threat Actor, and Vulnerability.   Other integrations allow your SIEM, log, EDR, and endpoint solutions to automatically alert based off of Recorded Futures Risk Lists.  Recorded future also has information provided by their Omni Partners allowing users to see more details related to the event and insights these vendors provided all in one place. 

After a full day of training there was an opening reception at the International Spy Museum.   Myke Cole talked with the group about the need for Cyber Leaders to get "in the weeds" when discussing and understanding the current threats.  Executives need to understand what these threats mean and how to protect their environments from them. 


Day 2: 
The morning of Day 2 was all keynotes and 'Fireside chats'​. 

The Grugq discussed the need to detect and remediate intrusions in a quick and efficient manner. He highlighted how hard it is to know if your defense is effective and how the offense has the positive signs of an attack (launch attack did it work = yes/no). He discussed dwell time and how attackers work to stay under the radar.  Attackers who successful elude detection can live on the network for up to 5 years (5 years being when companies usually revamp their hardware/services/solutions).  But one key take away from the talk was how attackers behave after they compromise a machine.  When hitting the machine, moving to the next stage of attack (data exfil), and then cleaning up after themselves(remove root kit, scrub logs, etc) the detection rates were fairly low.  But if the attacker kept repeating the date exfil steps(and actions on the machine) the chances of detection started to go up quickly.  One other aspect of the talk was that Attackers don't have time on their side but the defenders do.  Once the attacker launches that attack time is against them but the defender has all the time to detect/notice and take action.

The next talk was a fireside chat with Errol Weiss and Christopher Ahlberg (CEO of Recorded Future).  Errol is the SVP of Cyber Threat Intelligence at BoA.  This talk went over everything from building a good Threat intel team (starting small and building based on customer requirements) to providing daily briefs to your company which discuss current threats, events associated to it, and the recommendations to protect yourself/company. 

Robert M. Lee discussed Crashoverride the first ever malware framework designed and deployed to attack electric grids..  Robert discussed the threats to our power grid and future issues we will need to address. 

Next up was discussion on AI with Chris Poulin and Staffan Truve.  They covered multiple areas in the world of AI and machine learning.  

Last morning discussion was the Future of Recorded Future by Matt Kodama the VP of Product.  There are a lot of interesting areas RF is moving into not just security related. 

Here are the afternoon breakout sessions i attended: 

"Trafficker" Analysis:  Human Trafficking Threat Intelligence: 
Presentation by Jason Wonn from the Global Emancipation Network which focuses on human trafficking. Jason discussed how they use the Recorded Future tool to research different trends to alert police.  This organization is always looking for volunteers especially technology smart folks with time to spare.  If you want to help go here.
​
Case Study:  Threat Intelligence at Fannie Mae
Brian Scavotto from Fannie Mae discussed some of the challenges they faced around Threat Intelligence.   He discussed how he is using his team to educate the user base there for the latest threats and how they can protect themselves.   They have started an internal Threat Blog and Cyber Situational Awareness Report  which is emailed out to the company weekly. He also discussed the importance of cross team collaboration and seeing how the Threat Intel team can help other groups in the company. 

Technical Sources: The Science and Art of File Reputation
Igor Lasic discussed the Reversing Labs products and the benefit they provide to Recorded Future.  Reversing Labs is now a partner of Recorded Future and some of the File Reputation information will now be displayed in the Intel Cards. If you are looking for a STATIC analysis tool it might be worth taking a look. 

Case Study:  Threat Intelligence at TIAA and Vanguard
The last talk of the day and my favorite of the conference.  These presenters provided multiple use cases on how they are using Recorded Future in their environments to help manage security and RISK.   Couple of examples: 

•  Situational Awareness around Entities they have built in RF.  Any time there are changes around these entities they can see trends/changes and can act accordingly.   These changes can be related to anything from security news, to company announcements, to their partner/vendor news and issues. 
• Daily Threat Brief:  There team provides a threat brief to the company based off of the events in RF.  It is all automated and provides everyone with a good idea of what is happening 'now' in the Cyber arena. 
• Alerts:  They have setup alerts in RF to notify them when any of the following is seen on the internet; BID #s, Supplier ID#, Corporate Credentials (email, user names, etc), CIDR Block information, Domain information, Typo squatting.
• Vulnerability Research:  They have RF showing them the latest threats against their supported product list so they can make better decisions around which patch or workaround needs to be applied and when. 
• Dashboard Creation:  They have created multiple Threat Views in RF to show them any specific news around some of their Office locations. 
  

They also talked about the need to track Use cases.  Just like requirements in development work the use cases need to be documented, discussed, reviewed(periodically) , and the outcomes need to be noted. 

All in all this was a great conference and recommend to everyone to check out the Recorded Future product. 

When in doubt about your job ask yourself the following questions:

1.  Do you like the work you do? 
2.  Do you like the folks you work with? 
​3.  Are you learning/growing? 

If you answer NO to any of these then ask WHY NOT?   

No one can answer these questions but you!  Other folks can help you GET to your answer but at the end of the day it's up to you!  If you answer NO, ask yourself how can i change that to a YES?  Can I get involved with other projects at work?  Can i spend more time with my colleagues to get to know them?  Can i signup for an on-line learning site?  Or ask other team members how they do their work?  

If there is no good answer or action plan you can put together it might be time to move on our look for other opportunities.