How Just Eat runs devops at scale

UK-based food delivery company Just Eat runs a hugely complex devops culture across 35 software development teams in five geographies, working together to maintain 450 microservices.

At peak times Just Eat is processing 2,700 orders per minute, normally on a Saturday evening, which requires them to spin up in excess of 1,500 AWS instances. The engineering team ships up to 500 releases a week and generates 1.5TBs of logs per day.

Building a central SRE team

Site reliability engineering is of paramount importance for Just Eat, as it is to rivals like UberEats and Deliveroo, as outages cause an oversized reaction from hungry customers.

Take Just Eat’s reported outages on New Year’s Day, its second busiest day of the year after Valentine’s Day, where a tweet from the company reporting technical issues across its website and app received more than 200 responses, some verging on the dramatic side. “You’re ruining my life!” one user tweeted.

Speaking at the London leg of the AppDynamics World Tour last week, Richard Haigh, director of technology at Just Eat, said: “Site reliability engineering all starts with this first premise: we believe truly that dev teams own their product, full stop.

“So the devs create the code, they ship it, they look after it in the evenings, they take the pages home and we call them at 3 o’clock in the morning if their code fails and they help fix it. They own that code cradle to grave.”

However, those dev teams aren’t alone, they are supported by dedicated SRE teams, who run on what Haigh calls ‘five pillars’. These are:

• Relentlessly protect site availability
• Enable change to be delivered fast but baking quality in using automated delivery pipelines
• Optimise the use of infrastructure and resources – using scalable cloud solutions and ensuring the right spend there
• Innovate to stay ahead, especially when it comes to tooling – wether it is open source, commercial or built in house – to have the best possible solution
• Fostering a blameless culture and enabling autonomous teams

This starts with a central SRE team of 50-60 people who run a 24/7 service operations centre. “Their job is to boss the first ten minutes of any problem we’ve got,” Haigh said.

This team also looks after hosting across various cloud platforms, delivery automation (CI/CD pipelines), all monitoring, logging and alerting sitting under a team they call ‘observability’ internally, and service management run by a set of country managers.

Tooling

In terms of aligning these teams, Just Eat runs a whole range of activities to keep lines of communication open.

 Read more ANZ expands DevOps process overhaul after ‘fantastic’ early results

This starts with a daily standup to review any issues they had seen in the past 24 hours and prioritise next steps. Teams also run weekly risk meetings ahead of the busy weekend period. Lastly there is a monthly technology all-hands to discuss lessons learned and new tooling.

When it comes to tooling, Just Eat wants to fall in the sweet spot between giving development teams complete autonomy and a central team dictating which tools to use.

“We try to support that diversity and decentralisation but at the same time control some of the things that are larger in scale and more important to the business,” Haigh said.

“So we have central support for a range of tooling like the monitoring stack, but we give dev teams the opportunity to interact with that in an open source way, so they can take a version, fork it, manipulate it and give it back to us and we say that’s great and incorporate it back into the code base for everyone.”

However dev teams adopting new tools “go into that knowing the risks and taking that on, but there is also survival of the fittest,” Haigh said. “So if you have chosen that tool and it turns out to be absolutely awesome – we’ve seen this with something like Slack recently – the next team takes it and we get to the point where it’s best in breed and we take that onboard to try and find some of those economies of scale, enterprise support and make it easier for others to adopt it.”

Bennie Johnston, head of site reliability engineering at Just Eat also explained from the stage how the company has had to scale devops over the past four years due to some pretty serious growth at the publicly listed company, from just one release a day to nearer 500 per week today.

For example, four years ago the Just Eat dev teams were falling back on written run books to help resolve issues.

“Run books weren’t the way we were going to scale devops,” Johnston said. “Just Eat kept being successful so we got more and more dev teams. What we realised was our platform-as-a-service team needed scaling and that is where the deployment of monitoring came in.”

So the team went out to market to look for the right tools to help automate some of these processes. “That’s where AppDynamics came in, as well as our internal SRE team building tools to give the operations teams buttons and levers to affect the flow of traffic and protect that core user experience,” he said.

Speaking about AppDynamics in particular, Johnston outlined how it allowed Just Eat to centralise monitoring.

“We had monitoring-as-a-service from very early on, it was one of the things that allowed us to do devops, if you are operating something you need to be able to see it,” he explained.

However monitoring was developed by individual teams for their own services, so it was fractured.

For example: “For the Order API we had some great metrics, but from an operations point of view we had very little visibility into what was actually going on. We didn’t even really have real time order rate, this was something that came out of our database three days later.

“AppDynamics gave us the ability to look across all of our teams and services to correlate cause across and gave us that visibility into our systems,” he said.

AI

So, what next for JustEat? “We want incident resolution AI,” Haigh said.

At the moment managers will arrive to work in the morning to a raft of HipChat chat logs to pick through any incidents that occurred overnight.

“My dream is to come in one morning and there has been an incident but we won’t have had to wake any engineers up,” Haigh added. “In fact the bots and systems we are looking at will have identified where something has gone wrong and find what they need to do to recover those systems.

“If we get really good those dependent systems will talk to each other and will understand the implications of making that change. Those systems will work together to resolve that and then we will run sanity checking and it is logged in those same HipChat channel so that when we, the humans, come in after a good night’s sleep we can see exactly where the system was hurting and how to alleviate that moving forwards.”

Source : http://www.comeneat.com/blog/just-eat-runs-devops-scale/

Just Eat wants to use AR to drive delivery orders

For Just Eat, augmented reality isn’t just about people playing with trippy camera effects, it’s a way to drive sales on social networks.

The fast-food delivery service is testing how effective AR ads are at convincing people to order on social networks with an AR camera effect, developed by agency Byte London, that will run on Facebook over the next month as part of Just Eat’s campaign for the football frenzy around the World Cup. Just Eat is awarding certain people who download its app with collectible plates featuring illustrations of food, which when held up in front of the camera effect in Facebook Messenger, turns into the head of ex-footballer John Barnes asking to be licked.

“We don’t do AR for AR’s sake,” said Ben Carter, Just Eat’s marketing director in the U.K. “Everything we do from a social perspective is about driving activity through the funnel. AR isn’t the answer to all marketing, but it does give us a way of engaging with our audiences in a way that you can’t get on other channels.”

The company is considering more tests, now that Facebook has started sharing the performance of the effects with brands. Until recently, advertisers with effects on the social network were only able to see how many times they were used and how many views they generated. In March, however, certain brands were given access to performance data. That’s how Just Eat discovered the first effect it ran in December sparked a surprising amount of orders. The effect had around 102,000 unique impressions, and 10 percent of those people went on to order from the service within seven days during the six-week campaign.

The effect ran at a time when there weren’t many others on the social network, which made it easier for people to discover through their friends sharing it. Around 75 percent of the effect’s reach was not paid for, according to Just Eat. (The effects are free for brands to use, but they can pay to amplify their reach.) Part of that reach came from celebrity chef Ainsley Harriott’s post of himself using it, so the food delivery service is now weighing whether to use influencers in future campaigns.

Other advertisers like StudioCanal pinned effects to their pages and saw shares rise with minimal paid support because it was so easy to be naturally discovered. While they won’t fully offset Facebook’s declining reach for brands over the years, the AR effects are a reprieve for marketers who had resigned themselves to having to spend money to get noticed on the social network.

Unlike other brands, Just Eat piloted AR ads during Facebook’s test of the technology rather than buy Snapchat’s more established formats. The food delivery service was reluctant to pay for a lens on a platform that some media buyers have said costs as much as $1 million per day to use. In recent months, however, Snapchat has lowered its fees.

“The media cost to having a lens on Snapchat had been pretty prohibitive, but that’s changing,” one media exec said. “We saw Snapchat recently, and there’s a lower barrier to entry in terms of media, which can be as low as $40,000 now.”

Adidas and Coty are among a number of advertisers to have been encouraged by Snapchat’s revamped AR pitch. Both brands recently let fans buy directly from their lenses. Ad campaigns with lenses, on average, drive a 15 percent lift in purchase intent and a 9 percent sales lift in the U.S., according to Snapchat numbers verified by Nielsen.

Just Eat has done little on Snapchat, but the success of its Facebook effect combined with Snapchat’s easy performance tracking could tempt it to buy a lens to see which platform’s ads are more effective.

While AR on social networks can drive results, AR still needs a bigger audience to interest performance marketers, said Kieran Bass, managing partner at digital creative agency Kitty. Until this changes, AR is likely to be used for branding campaigns, and any performance metrics that come from the campaigns will likely be treated as a bonus, he said.

Source : http://www.comeneat.com/blog/just-eat-wants-use-ar-drive-delivery-orders/