Last night I met our newest batch of Social Superheros for our pre-hackathon Changemaker 101 session. Unfortunately I won't be able to attend this years hack. I'll be in Cape Town, South Africa attending the .Astronomy9 conference, running a tutorial session, and delivering an invited talk. But our RHoK lead-up events are fantastic and I wanted to stay involved at some level. Intrepid Travel very kindly hosted us for this event and it was great to have a few of their staff participate and see first hand how we RHoLL.
I've talked about Changemaker 101 in previous posts, so I won't go into the details here. Essentially Mandi was her awesome business analyst self and walked our changemakers through the basics of software development and introduced them to Agile and Lean methodologies. The goal of the evening was to help them identify their main roadblock and to prepare problems statements for the upcoming RHoK Info Night.
This year we have nine changemakers, including two return changemakers (Jen & Roberto);
- Martina Clark from Carers Couch (@carerscouch)
- Roberto Pietrobon from Care To Compare
- Jennifer McConachy from Berry Street (@berrystreet)
- Julie Gibson from Hitnet (@HITnet_au)
- Cathy and Beccah Bartlett from mADAPT – Shifra (@mAdapt_App)
- Andrew Whelan from Start Broadband (@startbroadband)
- Shari Davies from Domestic Violence Resource Centre (@DVRCInc)
- Pollyanna from PollyAnnaR / The BIG Picture Space Inc.
- Josie Lauristen Lee from Go Jane Give (@gojanegive)
While I won't be here to witness the RHoK Summer Hack magic – did I mention I'll be living it up in South Africa? ;) it was so great to meet Martina Clark from Carers Couch. This is such a great social enterprise I really want her to get the most out of RHoK. She also lost her best friend to cancer and because of our not-quite-shared experience we kind of hit it off. You can read about her story here. Funnily enough we also happen to live in the same street! You gotta love Elwood folk :)
Carers Couch provides advice, resources and attempts to build a supportive community around those who are caring for someone with cancer; be it a husband or wife, a mother or daughter, a father or son, the neighbour who cooks frozen dinners, the friend and fellow school mum who ferries the kids around, the close friend, the confidant, the not-so-close but still very important friend, to the local barista who brings morning coffee and cheer.
Martina came to us with a really clear, albeit challenging problem:
Carers Couch already exists, and has been up and running for well over a year. It sounds like she's done an amazing job so far with setting up a website, building up the community, giving talks, getting key stakeholders in place (e.g., Peter Mac), and participating in the Melbourne Health Accelerator Program. Martina's problem now is that she's reached a point where her business can't scale easily, which is where RHoK can really help. A little business analysis and development work, and tech mentors to assist in the move from manual and clunky to streamlined and automatic, can make a massive impact. So over the next few week RHoK will be helping Martina figure out how to deliver carer support in a really effective way. I'm super excited about this project and I'm hoping that we continue working with Martina post-RHoK to make sure she ends up with a whizz-bang tech solution with everything Carers Couch needs. After all, Rome wasn't built in a day (or two days...)
My fellow RHoKstar (and veteran) Shaun Wilde will be her RHoK Buddy and Tech Lead this year.
She's in such good hands.
Melbourne certainly doesn’t disappoint when it comes to tech networking events, and last night’s GitHub Constellation event was no exception. A fantastic event and an opportunity to meet experts, leaders and passionate advocates and builders of open source software. As expected the food and wine was excellent and there was an abundance of Octocat stickers… all the Octocat stickers.... and as much as I would have like to, I didn't wear my GitHub hoodie. ;)
Let’s start with some fun facts. There are over 95,000 GitHub users in Australia with 31% from Victoria and the majority of those based in Melbourne.
Daniel Figucio (@dfigucio), Director Solutions Engineering at GitHub began the night by giving a GitHub Project update. Most of these were announced recently at GitHub Universe. It seems like they are turning their focus towards projects and GitHub as a data platform. I’m not entirely sure what that means but it sounds like a good move. They are also focussing more on data-driven security and they’ve implemented separate team discussions as part of issues. There is now a snazzy graph that tracks dependencies within a repository and soon you will be able to track security vulnerabilities. Security alerts are an important step towards keeping code safe.
They are also ramping up their own machine learning projects in order to understand customer behaviour. I wasn’t surprised to hear this. It seems more and more companies are using machine learning to predict user behaviour and create user-driven products. The GitHub News Feed now includes a Discover Repositories tab. Recommendations are based on the users you follow, your starred repositories, and what is trending. Admittedly I’ve never really followed or starred things in the past but I will be now that I’m using GitHub more. My favourite “new” thing is GitHub Collections.
I’d never really taken the time to explore GitHub. I tend to just go directly to the repositories of people I know. But the project collections look really good. I’ve got to say I was pretty pleased to see the Made in Africa collection on the front page. I don’t know if GitHub knows I’m about South Africa and that I help run Random Hacks of Kindness. I’m guessing not… but you never know.
And then there were the speakers…
Julie Mission (Make it APPen) is a nurse by trade and self-confessed nerd by nature. She builds apps to enhance patient care, mainly for pain management and carers, and assists hospitals and healthcare professionals to create their own. She’s a pretty fantastic woman who started building apps in her 50s, although she’s been programming in DOS since the 80s. She now uses Xamarin to build her apps and last year she wrote a book called Planning and Designing an App to Enhance Patient care. I had a really good chat with her afterwards and was pleased to find out that she’s based in Bendigo. Earlier this year RHoK Australia expanded and RHoK Bendigo is now up and running. I’m hoping she will get involved in the community.
"Zendesk's Answer Bot uses deep learning to understand customer queries, responding with relevant knowledge base articles that allow customers to self-serve. Research and development behind the ML models underpinning Answer Bot has been rewarding but punctuated with pivotal deviations from our charted course"
I recently had the opportunity to hear Zendesk Data Scientists, Arwen Griffioen and Chris Hausler talk about their journey from product ideation to launch, starting with a traditional customer-base d machine learning approach, and ending with a single global deep learning model that serves tens of thousands of accounts.
This was a fantastic talk that gave a really good insight into how the Zendesk Machine Learning team works and what they value. Both Arwen and Chris have research backgrounds, which is always great to see. Arwen has a computer science background and finished her PhD on ecological modelling (using MLA) in 2015. Chris has a computational neuroscience background and finished his PhD in 2014.
Apparently the data team at Zendesk is like a team sport a real mix of talent: engineers, software developers and data scientists all working together towards a common goal. I love that any additional trainging (i.e., deep learning) is done as a team and includes everyone, regardless of their specific role. I’ve heard of data science teams where only the most senior are allowed to up skill at work and then pass on the knowledge — the rest have to do it in their own time — which is ludicrous. High performance teams work when people are encouraged grow and learn and develop new skills.
The talk began with the anatomy of a data product. I loved their iceberg analogy. While things may appear to be advancing smoothly (at least in press releases, conference talks, shareholder letters), the bulk of the time is really spent researching new methods, trying things that ultimately fail — life would be terribly boring if we had all the answers º — designing, testing, and re-engineering.
To build the Answer Bot, the team started out with a fairly simple machine learning model. By simple I mean supervised learning using NLP on a software ticket, and using a logistic classifier to predict the most relevant help document or article. The assumption was that this would be fairly accurate, performant, familiar and explainable. Because the industry and therefore context around tickets varies broadly for all of Zendesk’s clients, labelled data would need to be provided for each client, and because the Answer Bot learns on the job and improves with more data, you can’t really switch it on from the get go. For this to work well the team needed to spend a lot of time preprocessing data.
The team explored unsupervised classification, using both tickets and articles as inputs which worked well, except that it would require ~100,000 different models (for each client) and it takes a really long time to train. Part of the reason is that the same words can has a very different meaning depending on the user, and different industries have different sets of words. For example "ticket" may mean an issued ticket, given so that someone can join a queue, or it could be something that is purchased, for example a movie ticket. Answering a question such as "what do I do if I lose my ticket" requires a good understanding of context. If you try to build a single model will all the words the dictionary, you're going to run out of parameters pretty quickly.
Pivoting to Deep Learning:
This happened quite a few months down the track and came partly out of their journal club. They essentially started from scratch, and this required loads of reading and retraining the whole team. A lot of uncertainty and not really knowing what they were doing, but with some knowledge that NLP problems work well with deep learning and the more data you can throw at it the better. Zendesk has no shortage of data. After the talk I asked Arwen how much she and the team knew about deep learning before coming to Zendesk and her answer was “basically nothing” (I love this company!)
The team split into two groups and tackled various aspects of the problem. I wasn’t surprised to hear they use TensorFlow. I was really pleased to hear Chris say that problem solving is a creative process — the mark of a great researcher, and not something you can learn easily.
The initial perceptions of deep learning were that you could develop one robust model, that it would work well, and that the more data you threw it at the better it would work. This is one of my big worries about machine learning and deep learning. Weights are determined as if by magic, loss functions are calculated and "accurate" results are taken as gospel. From my experience with astronomy data, I can tell you right now that if you start with ALL THE CRAPPY DATA you can still get a good fit, after all you just need to keep adding parameters — seven dimensional string theory anyone? BUT.... The result will inevitably meaningless. Chris summed this up eloquently; "if you put shit in, you're going to get shit out"... or something to that effect. So this is where things get really exciting. This is where you have to go back and figure out each step of the miracle that is deep learning and exploring everything that’s going on and what could be implemented, whether the data introduces unintended biases — turns out datasets with large numbers tickets were artificially skewing things, and whether there are overfitting problems (Hint: unless you have an underlying physical model there will almost always be overfitting problems).
Of course the hard work paid off and it sounds like they’ve come up with a bloody good solution. The entire process took six months and it was a good year before the product was considered reliable enough for deployment. They spent quite a lot of time validating the model, developing reliable performance metrics, ensuring consistency, and taking the time to do proper human user testing. I was both surprised and pleased that Zendesk allowed the team spend so much time researching. Since I’ve never worked at a tech company I’m not sure what would be considered normal, but my impression is that many data science teams are expected to data analysis results out on pretty short timescales, regardless of data quality.
Lessons from the team:
- ML products are really hard work.
- “Vanilla” ML works really well. Logistic regression and Random Forrest work really well.
- Always start with the simplest model.
- Deep learning isn’t magic
- When it finally works, it’s great.
Thanks to Cal's hard work, Zendesk recently came on board as a National Sponsor.
Needless to say we are thrilled.