Springsmith

Bits and pieces from around the conference.

Conferences are a great opportunity to meet people with diverse interests and to pick up links to websites with all sorts of things to explore further.

OpenMicroscopy (aside)

I had an interesting chat with a chap from OpenMicroscopy an open source, open service for sharing raw scientific data from microscopes. A very cool project attempting to do something very academic.

There were also some Records of Scotland people there. I wonder if they are still doing stuff with MongoDB I haven’t see a job ad about that of late.

Instant Gratification Monkey

I can’t even remember which talk he came up in – but Google him if you like funny self-help books. My 18 year old son knew all about it – it’s a thing, apparently.

I suspect the IGM of eating people’s cheese when they can’t find it because it has been moved.

Active and Passive Eh?

I made a note which seemed very important and profound – but a bit like writing down a dream it seems a bit sparse now… can any body help? (it might be to do with update & failover)

“Active active app… Active passive db” is what it says – what does it mean?

Mousache seemed to be a popular templating language

Moustache templating managed a couple of passing quick mentions – (personally Handlebars has more going for it, or even EJS(embedded Javascript) as these have as simple a syntax as Moustache for doing the easy things but can do so much more). I like EJS because in a NodeJS and web-app world it keeps the number of languages to a minimum. I’d rather have one language to be an expert in than four to be mediocre at.

Additional resources…

http://www.oreilly.com/security/free/http://www.oreilly.com/webops/free/

https://www.mulesoft.com/lp/whitepaper/api/top-microservices-patterns?utm_source=twitter&utm_medium=cpc

http://www1.memsql.com/oreilly-predictive-analytics.html?utm_source=twitter&utm_medium=cpc&utm_content=oreillyml&utm_campaign=oreillyml

https://www.nginx.com/resources/library/complete-nginx-cookbook/?utm_campaign=core&utm_medium=ebook&utm_source=twitter-cpc&utm_content=nginx

http://www.javascriptsource.com/tutorials/javascript-building-a-fitness-app-data-and-models.html?utm_medium=social&utm_source=twitter&utm_campaign=book_getprogrammingwithjavascript&utm_content=article_buildingafitnessapp_mar1416

A Classic Quote

For everyone that likes Microsoft Windows: “Those who do not understand Unix are condemned to reinvent it, poorly.” Henry Spencer

How to Run a Security Workshop for Developers. 

GDS (i.e. government digital services) gave an interesting talk about running a security workshop. Provided as a reaction to a very poor and sexist externally source workspace.

People tend not to learn from history so this is a means of giving them experience.

This was another example of being empowered to cause a change in culture.

Security errors using actual source from the organisation… Including the development environment generally used, and past examples from the code base.

Collaboration based exercises, rather than competition.

15 people

Instructions and help for tech set up sent out well in advance

Setup local network set up on a Raspberry Pi – to provide servers and a sandbox while hacking.

Short retrospective at time for feedback

Several people had run similar workshops since – which has to be a ringing endorsement. I’d happily run or attend one of these. There is a lot of work required though.

The Handling Secrets Breakout…

This is an area of weakness for many organisations. Wow! you should hear some of the stories. Even the confident and competent people live in fear of being caught out by bad actors within their organisation. As usual, the assumption of your network being penetrated and limiting damage is the order of the day.

Watch Out for the Weakest Links with the Biggest Payoff

• Credentials for cloud services
• Data access – database connection credentials an infamous problem– “In SQL select is the most dangerous command”
• Vulnerable build chains – a way to breach the perimeter of 3rd parties

Handling Secrets

• 2 factor auth for login, and the importance of protecting logins.
• SSL keys great for ease of use and protecting the perimeter.
• SSH should be unavailable on production micro-service servers. The aim is a blackbox with low attack surface.
• No root login anywhere – for many reasons
• Selinux – argh! I’m not a big fan – a lot to learn for esoteric benefit. Maybe there are some wizards/bots out there these days.

Role playing

Role based access – the start-up “we have three engineers and they are all devops” people are in a quite different place from the enterprise “over my dead body” school of thought. I think we can all guess which is more “role-based”.

Having said that, many start-up people did have policies and those that didn’t were very interested in how to implement them, because much like “version control be vital in groups, and improve productivity for an indiviual dev” so it is that role based data access is the right thing to do in order to prevent massive mess-ups. GDPR reared it’s ugly head.

Several tools were mentioned…

*The Unix “pass” command seemed to have good support for retaining passwords locally – and has a Mac port and a Windows equivalent. https://www.passwordstore.org/

Some multi user / enterpise options were:

• Vault
• Lastpass
• Secretserver….this was considered by users to be horrible

Regardless, in addition to controlling passwords 2-factor authentication was considered vital. yubikey was the tool of choice for this. Other options include Google.

There was some debate about trusting keys to third parties but this was countered by the observation that email pretty much universally being purchased from 3rd parties e.g. Google and Office 365.

The question as usual led back to the adage “ask not whether you will be compromised; ask what you will do when you are.”

Data Protection – The Gathering Storm

New data protection legislation GDPR – big problems for small operators and a big issue for monolithic systems without traceability already built in.

Logging

Datadog and Forest were brought up as monitoring tools for microservices.

Online coding puzzles during learning hour, was suggested as a good way to develop coding skills.

Through out the conference there was a whole lot of discussion about how Devops interacts in the live environment. There were many ideas discussed and there was a considerable amount of time dedicated to the processes and tools to support them.

Testing in Live

Eeek! that sounds “exotic” Parallel live systems are used in various ways I’m not going to type it all up http://blog.christianposta.com/deploy/blue-green-deployments-a-b-testing-and-canary-releases/

Most of these techniques deliver a lot of information and help deliver what the customer wants more quickly and reliably BUT it does not do this in a “throw it over the fence to operations never see it again environment”. The separation of the live data from the data about the live system should not be so challenging if the operational logging is suitable, but that needs development and operations co-operation, and time invested. That is a total cost of ownership sell, not an isolated development and separate support sell.

Bitmovin is a video transcoding company that uses permanent canary i.e. they always have two versions on the go and switch users over gradually. back and forth. Another variation is feature flags for A/B testing – same version of code configured differently Shadowing….same traffic user sees old… Verifies new… Able to monitor relative performance – (upgrades/changes to protocols are an issue here) A major difference can be whether there are separate databases or not (obviously data migration is an issue with this – although document bases NoSQL DBs would handle this).

I would say the thing to do with this is to be in a position to replicate the infrastructure for testing and pivot between approaches. To choose between approaches is a false dicotomy, both on a product, project and sprint-by-sprint basis. Is there a data change? Is there a protocol change? Is there a user interaction to be measured? Is performance the key measure? With Docker and Kubernetes the replicated infrastructure need not be up for longer than required and can be readily reconfigured if done right.

Docker in live

Do you know where your Docker images come from?

Images should all be signed! You must run a private Docker image registry for security, repeatability.

Specific versions of container images from controlled sources – none of the “latest” version from a hooky repo, please!

Don’t patch live containers. Containers should be rebuilt tested and installed. Otherwise here lies the madness of manually patched systems. Service discovery is therefore important because containers have to configure themselves. NB this can also be handle by kubernetes secrets. Waiting for other services is a thing too. Automated start-up is an issue if it all has to be in order.

Images should be impregnable and use service discovery to acquire third party tokens. Only the deployment environment user should have access to the master discovery keys by proclamation.

An Ops Tool

One thing I’ve colaborated on was getting a web front end to read A+ on the RSALabs SSL testing tool. I’ve always wondered how to Pen-test my own services without spending silly money on experts.

Tools to Support Testing

OpenSCAP looks like a really great tool for examining the attack surface (out of date auth protocols, open ports and the like) You can specify the security policy you wish to apply… It generates an Ansible script to fix your issues. I’m not sure if that is applicable if you don’t use Ansible or whether it is human readable to use it for manual patching.

Chaos Monkey – tool for testing your infrastructure for machines and networks going up and down.

Site Reliability

Google Site Reliability Engineering… https://www.amazon.co.uk/Site-Reliability-Engineering-Production-Systems/dp/149192912X Sigterm not sigkill – the responsibiliy of microservices to “tidy up quickly and quietly” (as my primary school teacher used to say). Expecially true as Docker sigterms and then sigkills automatically. Likewise, “FFS don’t use latest as a config tag” was oft repeated advice over the two days.

Some interesting stuff about how to run parallel systems for users switching over groups and proportions from one to the other. I’ve done this myself on a distributed EFTPOS systems (to provide “exponential rollout” to limit risk).

When ‘it Hits the Fan

There was an interesting talk about how to analyse incidents. Three Steps: Observation – Response – Remediation

“Mean Time To Recover” – slicing and dicing

Accountability = responsible for reporting, not to take the blame

Disable a service the when micro-services on which it depends are unavailable

A Status page for stake holders running on a different system was considered to be essential. Certainly my experience is that this can be reassuring for users and is helpful for keeping the people fixing the system focussed on fixing rather than reporting. Automatic functionality is useful for proactive notification, .

There is a free book which was recommended. Bit.ly/PIR_book

Tool for the Ops side of the Devops

Kubemonkey is a tool providing circuit breakers graceful degradation

Inside the Container. 

What OS should go into a microservices container?

Alpine is a tiny footprint Unix like OS. Running it in a read-only docker image with no SSH is the basis of a syatem with a very limited attack surface. This is very much the tiny no-attack surface approach – rather than the RedHat with its “another layer to administer approach”. Great for Node.JS! but Java is just plain massive and risky ridden, so loses a lot of the advantages. As a dev

Many people declared they used Java as the langauge for microservices – but there were significant NodeJS and Ruby contingents plus a couple of C / bare metal enthusiasts.

So RedHat based for Java and alpine for NodeJS. A bit like whether you have lots of stained glass windows and lots of gold in a church or not, this was an indication of a bit of a schism. Are the extra bells and whistles essential for security or do they just complicate things?

Securing Containers

Some steps to securing a container…

• No root login – this is a security risk on Docker.
• limit ssh – if possible none.
• namespaces – chroot etc.
• logging – a separate service
• resource restriction – so a runaway container doesn’t consume everything (you need to leave headroom for the loggin service if nothing else).
• Don’t run anything as root. (aside – people have been known to build rpms as root – so the whole install goes in as owned by root).

Kubernetes to Configure Containers

Kubernetes configmaps & secrets https://kubernetes.io/docs/concepts/configuration/secret/#overview-of-secrets

OpenShift does the same thing as k8s but provides multitenancy support. For people who still like one big mess of servers.

Use Kubernetes for configuring persistent storage – many options due to the flexibility of Unix filesystems.

RedHat – very Enterprise

The RedHat guy was very much in favour of seLinux. As a general rule the RedHat presentation still had a siloed approach, with a high cognitive load “add another layer and use an enterprise stack”. Having said that OpenStack has a place in multi-tenancy. But, to go all metaphorical, if you’ve ever lived with a family in a tenemental stair you don’t wonder why people favour detatched houses. The simplicity of responsibility and

Running micro-services – much agreement on services, and less agreement on tools. 12 Factor App https://12factor.net/

Playing with Micro-Services

Kubernetes for the Faint Hearted

“Kubernetes nice but don’t I need a cluster before I can develop on it…?” Minikube is the answer. It has a cousin for OpenShift https://www.openshift.org/minishift/

Since the conferences this has turned up too… https://www.telepresence.io/ (if you have SSH access to the proxy server it sets up a VPN – see notes on not having SSH access on live micro-services) – this definitely works with Minikube but what about MiniShift?

When Micro-Services Go bad!

Sprawling… Coupling… Too many services not enough people… Multilayered rather than flat… Poor API documentation…

A MicroServices Top-tip:
Log the caller stack. Who called who in response to what web request.

Failure cases and dependencies

A Massive Selection of Tools. 

Another theme, not surprisingly, was tooling. Again a very generous exchange of tools evaluated, used and developed. Some of the notes below are in the expectation that people will follow the links and read the summaries. It’s not my job to type in yet another summary!

Overall the principal tools were Kubernetes with Docker. Really this grows from mind share and availability of expertise perspectives, rather than technical capabilities. Other tools are available.

Git also had a big following. As the version control jam holding the process cake together. It’s plethora of hooks aiding its use in integrating other tools.

The command line was very much the interface of choice. Scripts made a frequent appearance (bash not Powersh**).

Kubernetes has rolling UPDATE out the box, great. Role defined security – not so much 🙁

Devops seems to mean a quite different things to different people. “Devs doing ops”, “automation of ops”, “co-location of dev and ops”, “you build it you run it”, “automation of the build and deployment process”, “containers”

Routing and Service Discovery

OpenShift provides routing, Kubernetes presumes NGINX will handle that.

istio An open platform to connect, manage, and secure micro-services. https://github.com/istio

linkerd Resilient service mesh for cloud native apps – linkerd is a transparent proxy that adds service discovery, routing, failure handling, and visibility to modern software applications https://linkerd.io/

There was talk by one presenter of “Container Difference Exchange” Adaptive Action: Leveraging Uncertainty in Your Organization By Glenda H. Eoyang, Royce J. Holladay

https://books.google.co.uk/books?id=zg_QkuY_QPoC&pg=PA89&lpg=PA89&dq=Container+Difference+Exchange&source=bl&ots=6n8zxdgCMa&sig=n0KHG0ecJ98NE4o4i2sIBYDYkvE&hl=en&sa=X&ved=0ahUKEwjahJ3V0Y7XAhWF1xoKHXW9AOQQ6AEIVDAI#v=onepage&q=Container%20Difference%20Exchange&f=false

Habitat – new tool

Habitat – easy Chef from the people who brought you Chef. New product. I picked up a sticker from the silver sponsor’s stand, of a cat hiding in a box.. whatever. It sounded interesting at first glance it looks like a bunch of generic config files, install/config scripts and wizards to use them.

Amazon Web Services – a bit intimidating – Open Guide to AWS is a good intro https://github.com/open-guides/og-aws

The Phoenix Project – recommended by several speakers as an accessible/fun into to the dynamic world of devops.

ElasticSearch

The Elastic man gave one of the more sales-y talks. Mostly concentrating on an “out of the box” look at databases. Some titbits though… From what I’ve seen ElasticSearch is very much a leader in Big Data Analytics. They have their own DB. As with Oracle and Microsoft the question is whether you end up making money for them. ElasticSearch provides its own official Docker images – the official ones on Docker hub are just “Docker official”. docker.elastic.co has the “Elasticsearch offical” images.https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html He mentioned X-pack is a free (or supported for money) Docker image for all of the ElasticSearch products pre-configured – for a generic scenario.

Weaveworks – a process case study

A great look at automating a process around git. After all everyone wants their build scripts, config files and test evidence under control too!

Gitops weaveworks https://www.weave.works/blog/gitops-operations-by-pull-request

Our provisioning of AWS resources and deployment of k8s is declarative Our entire system state is under version control and described in a single Git repository Operational changes are made by pull request (plus build & release pipelines) Diff tools detect any divergence and notify us via Slack alerts; and sync tools enable convergence Rollback and audit logs are also provided via Git Sweet… and with a https://www.weave.works/blog/provisioning-lifecycle-production-ready-kubernetes-cluster/

A Warning for History

As is very common with Docker meetups there was some intense technical detail… “Don’t mutate bind mounted directories.” was an assertion that was made and attracted nods from bearded blokes in T-Shirts around the room. “Hmm, I thought.” https://unix.stackexchange.com/questions/198590/what-is-a-bind-mount

I think this came from Adrian’s sizzling Docker ignite – too fast to make notes it’s mostly available here https://blog.docker.com/2017/11/tips-tricks-docker-captains/

Consortium for service innovation

Diversity

This was all before the recent celebrity revelations hit the news, so hats off to the organisers.

Beyond the fact that I don’t like and cannot abide the bro-grammer mentality, nasty internet trolling and any form of intimidation in the workplace I don’t have any particular axe to grind. Experience suggests that without positive efforts these things creep in.

The organisers made a real effort when it came to gender balance/diversity. There was a far higher proportion of female speakers than I was expecting. The organisation team also saw a good mixed representation. Which is leading from the front. Certainly in the breakout sessions I was in it was more often than not women compering. There were “diversity tickets” available.

Monoculture is never healthy, and experience should not be a weapon of intimidation but rather it is something to be shared, and ignored, then shared again, and finally to be demonstrated as right or wrong.

Despite the fact that it might sound a bit condescending the talk by Jenny Duckett was particularly robust in its coverage of both hard and soft skills, and the conference would have been lessened without her contribution. More about her talk on running security workshops in the “Security is a big thing” post.

The Edinburgh tech scene is, I like to think, a pretty liberal crowd. No we are not perfect and we are pretty homogenous.

Soft Skills

Work by Lencioni was mentioned (by more than one person) http://www.fivebehaviors.com/About.aspx a very “soft skills” book, with talk of things such as healthy vs toxic conflict.

Chatops

Chatops was mentioned many times. There is a long intro here https://www.atlassian.com/blog/software-teams/what-is-chatops-adoption-guide

Basically, using Slack, and bots; all integrated with development and ops. Very “open kimono” – even to the extent of reporting failed developer builds so assistance can be offered. It is beloved of remote working scenarios and although I still prefer to overhear people swearing at their computers, to be honest from what I see of my 14 year old son, social interaction online is something they are very comfortable with.

Soft Skills for Remote Working

One of the breakout sessions was on remote working. The idea that if you have any remote workers then the local workers should communicate with someone sitting at the next desk by dialling into a chatroom seemed to have traction. It still seems weird to me.

Very comprehensive Chatops seemed to be very important for passing around general advice, updates, allowing people to overhear topics, and reducing isolation.

What I will say is my 14 year old son socialises with his mates online using Discord – “we just hang out, play games together, listen to music” so the younger generation is more at ease with this. He has this 21st Century soft skill – I see a lot of people chatting to kids using Facetime, so the young have these skills, too – I imagine my great-grandparents generation much preferring visiting to using the telephone.

I’ve used regular meetings over high quality video conferencing to work with cross site teams to avoid too much travel… much better than audio only.

Cross-skilling Doesn’t Happen by Accident

If you want devs to run the systems they build then you have to give devs training in recovering broken systems.

I would also add that if you want Ops people to check code/scripts into source control then you need to give them training too.

Even if you have automated things people have to understand what it is doing (not necessarily how it is doing it). Automation is never perfect. Instructions are never complete. If you don’t want behaviour resembling pulling the handle on the fruit machine hoping to win this time.

Why Did I Attend? 

I have a several fold interest – from living the devops dream in a start-up and a hankering to return to them one day; from the perspective of working in a public sector enterprise and trying to work out how to bring some more of the magic into the organisation; GDPR coming over the horizon and trying to work out what that is going to mean; and finally having a home “sandbox” project where I want to “do things right” and not spend an inordinate amount of time doing them.

As an engineer I’m big into repeatability and make it easier to do the right thing rather than the wrong thing. I also see that “the only defect free production line is a stopped production” and that “keeping down costs and boosting turnover are what makes profit”. Devops is very much addressing these issues. I took two days holiday and paid for it myself, so I’m keen.

Devops “all things to all people”?

Having just asserted what devops means to me. I will now list out a bunch of other things that devops means to different people. Many of the overlap or are alternative ways of looking a the same thing, and some are not.

• Automating operations
• Giving operations control to devs
• Getting operational requirements fed into the development
• Getting operations support into the development environment
• Providing representative systems to devs
• Making devs eat their own dog food
• Building dynamic teams

Event Summary

“First the housekeeping” as they always say at these things.https://www.devopsdays.org/events/2017-edinburgh/welcome/

I’ve included the schedule at the bottom of this post – in case the site above disappears and so you can Google the people or the topics. If you search on Youtube you often find the speaker giving the talk at one conference or another. The talks really exist to give the delegates something in common to discuss and to give jumping off points. Therefore YouTube is a poor substitute for being there. A good talk can be quite low grade ore in terms of hard facts.

Format

I attended all the main sessions, I will include pics of the Open Space ideas boards and mark what I attended. Ignites are 5 minute talks with the slides progressing automatically at a rate of one every 15 seconds – a bit like TED talks, they are considered to be a “thing”. https://en.wikipedia.org/wiki/Ignite_(event)

Rather than a sequential blow-by-blow account I’m going to split take-aways into more functional groupings. I will also be attempting to keep the elephants at the top of the page and the details at the bottom.

One of the main themes of the conference was “culture”. Diversity, Ops vs Dev vs Dev Ops, Startup vs Enterprise, New vs Established, Public vs Private sector. It was actually all very positive in this regard. People seemed genuinely interesed in sharing experiences, and learning from one another.

October 23, 2017

Registration and Breakfast

09:00 – 09:15 Opening Welcome

09:15 – 09:45 Opening Keynote: Maria Gutierrez

09:45 – 10:15 Adrian Mouat – Microservice Deployment Techniques

10:15 – 10:45 Peter Varhol – Using Machine Learning to Optimize DevOps Practices

10:45 – 11:00 Break

11:00 – 11:30 Gerie Owen – DevOps and Groupthink: An Oxymoron?

11:30 – 12:00 Abhishek Chanda – Helm and the zen of managing complex Kubernetes apps

12:00 – 12:30 Cookie Lanfear – From Software Development Bootcamp to Junior DevOps Engineer

12:30 – 13:00 Philipp Krenn – Providing and Supporting Docker Images

13:00 – 13:10 Sponsors

13:10 – 14:00 Lunch Break

14:00 – 14:30 Ignites

Jon Hall – Aligning DevOps and IT Support in the Enterprise, Through Intelligent Swarming

Philipp Krenn – NoSQL Means No Security?

Murdo Aird – Using ChatOps in an open and conversational workflow

14:30 – 15:00 Open Spaces Planning

15:00 – 15:30 Break

15:30 – 16:00 Open Space 1

16:00 – 16:30 Open Space 2

16:30 – 17:00 Open Space 3

17:00 – 17:20 Day Close

17:30 – 19:30 Drinks Reception

October 24, 2017

08:00 – 09:00 Registration and Breakfast

09:00 – 09:15 Opening Welcome

09:15 – 09:45 Paul Gillespie – The Perfect DevOps Storm

09:45 – 10:15 Charlotte Godley – Engineering a Continuous Delivery Pipeline

10:15 – 10:45 Josh Atwell – Work + Family + Self + Fast Paced Industry = _()/

10:45 – 11:00 Break

11:00 – 11.30 Neil Crawford – Repository driven development

11:30 – 12.00 Chris Van Tuin – A DevOps State of Mind: Continuous Security with Kubernetes

12:00 – 12.30 Jason Hand – Scrutinizing the Scrutiny

12:30 – 13:00 Jenny Duckett – Encouraging a culture of learning across your organisation

13:00 – 13:10 Sponsors

13:10 – 14:00 Lunch Break

14:00 – 14:30 Ignites

Ken Mugrage – The answer to the “where do we start” question

Gabriel Nepomuceno – Intelligent Diagnostics and debug

Adrian Mouat – Docker Tips and Tricks

14:30 – 15:00 Open Spaces Planning

15:00 – 15:30 Break

15:30 – 16:00 Open Space 1

16:00 – 16:30 Open Space 2

16:30 – 17:00 Open Space 3

17:00 – 17:20 Day Close

As a conference DevopsDays Edinburgh 2017 was spot on. My thanks to the organisers, speakers, sponsors and other attendees.

Mostly I make notes during talks because I don’t have time to sit through them all again on YouTube/Vimeo. A few people have requested my notes, so I’ve typed them up to a modest standard (there is beer in the pub that is needing drunk). Conferences are interesting because they are a curated, and much like stand-up comedy, and music concerts hearing the ‘standards’ live can be refreshing. Also as the talks and Open Spaces layer up some consensus and unique character comes into being. Hell, you had to be there.

All errors and opinions (and erroneous opinions) should be assumed to be mine – unless you hear it from the person in question. Factual errors and arguments are welcomed. I have augmented various places with my “stream of consciousness” while listening to the talks (so if you think you are mentioned there might be some things you don’t remember saying and if you were in the audience there might be things you don’t remember hearing). Some things might be repeated, like a riff. Some opinions expressed might be stronger than (or even contradict) my beliefs – the Devil’s advocate is abroad. Besides knuckling under and knuckling down and doing your best is a largely what the world of work is about.

Part of the joy of attending conferences is the chance to meet interesting people and bore them silly. Never having been a fan of “socially novel” situations it took me a few conferences to get into the swing of them. It’s an interesting challenge trying to find common ground with strangers, and fascinating to find out their working methods, processes and challenges. Always bubbling away at the back of your mind are questions like “Could that work on the projects I am working on?” “Would I do it like that?” “Is this more trouble than it’s worth?” “I wonder if it is that easy?” “That is so useful, are they selling the shop?”

The right to modify the texts to improve it (mostly) is reserved. Comments can be addressed to david at springsmith dot com